Does it matter in which direction I travel in relativity theory?

I'm not a physicist, but I'm still very interested in the relativity theory, especially in how the twin paradox is explained. Actually, it does not make sense to me and I hope you can answer my following question to help me understand how it works:

Does it matter in which direction i travel in the relativity theory? When I first read about the theory, I thought, that it does not matter if A moves away from B, or B moves away from A, because every party will see the effect of "time going slower" for the other pary, meaning A sees B aging slower and B sees A aging slower.

But regarding the twin paradox, when A leaves the earth with insane speed, and returns years later, A has aged less than the people on earth. If this is actually true, then my previous assumption is wrong and it does matter if A moves away from B or the other way round, because the effect could be turned around, so that the earth ages less than A, if the earth moved away from A instead.

If this is true, then this brings up a next question: How can we know if B moves away from A or A moves away from B? Since the earth, our solar system, and even our galaxy is already traveling in the universe at some speed in a specific direction, we could actually stop moving by flying in the exact opposite direction at the same speed. but would this be considered as the earth moving away or as we moving away?

I really hope you can help me with that :)

Update for @Alfred Centauri's answer

the stay-at-home twin does not change direction while the travelling twin does.

When I look at it from only the travelers perspective, then yes. But when I look at it from the earth's perspective, then the earth indeed changes its direction, while the traveler does not. Why is that changing direction thing that important?

As I could see in Will's answer, the change of speed in universe seems to be the critical difference. In the time where the traveler is still on the earth, both the traveler and the earth move at the same speed in the universe, which means they age exactly the same. As soon as the traveler starts to accelerate or decelerate (move faster or slower than the earth) he slows his own time, no matter if he accelerates or decelerates.

If this is true, this means that not only the relative speed of the objects influences the time. Also the change of speed in the universe does.

But this opens another question: What if 2 travelers move to the exact same position in the universe, but traveler A goes there at twice the speed? Are the two travelers the same age when they meet at the destination? Or is one traveler older because he moved faster?

Answer

If this is actually true, then my previous assumption is wrong and it does matter if A moves away from B or the other way round,

In the context of the Twin "Paradox", it doesn't matter. It isn't the direction that makes the difference, it is the path through spacetime.

The key to understanding the twin paradox without the frightening mathematics of GR is this: from any IRF (inertial reference frame), the stay-at-home twin does not change direction while the travelling twin does.

For example, chose as your reference frame, instead of the one above, the frame of the travelling twin on the outbound leg (we've switched which twin is "moving away").

In this frame, it is the stay-at-home twin that ages slowly, during the outbound phase, compared to the travelling twin. However, and crucially, the travelling twin must change direction in order to return to the other twin.

So, at the halfway point, the travelling twin changes direction and is now travelling faster than the stay-at-home twin.

Now, it is the travelling twin that is ageing more slowly than the stay-at-home twin. Moreover, due to the non-linear time dilation factor, the travelling twin is ageing slowly enough on the inbound leg such that the total ageing along the outbound and inbound paths is less than the total ageing along the straight path of the stay-at-home twin.

The bottom line is that the situation isn't symmetric. The stay at home twin never changes direction while the travelling twin does.

But when I look at it from the earth's perspective, then the earth indeed changes its direction

No, from any inertial (non-accelerated) reference frame, the stay-at-home twin's path through spacetime is straight.

You're not taking into account that the travelling twin changes from one IRF (the outbound leg) to another (the inbound leg), i.e., the travelling twin boosts from one frame to another while the stay-at-home twin does not. The travelling twin's frame of reference is non-inertial during the turnaround.

This is the crucial difference between the two paths through spacetime; one has a "kink" (the turnaround), and the other does not.

Why is that changing direction thing that important?

Because a straight spacetime path connecting two events is different from a curved or kinked spacetime path connecting the same two events.

Intuitively, in Euclidean geometry, the shortest path between two points is a straight line.

Counter-intuitively, in the Minkowski geometry of spacetime, the straight path between two events is the "longest" in the sense that the elapsed time (proper time) along a straight path is larger than for any other path.

Since the stay-at-home twin's path is straight, the ageing along that path is greater than for any other path.