electromagnetism - Which mathematical operation does the right hand rule for current come from?

I am currently wondering about the famous right-hand rule for magnetic fields around currents.

Where does it come from mathematically that when you point with your thumb in the direction of the current, your curved fingers will point in the direction of the $B$-field? In other words, which mathematical operation does it come from? Probably it has to do with some vector product or Stokes theorem application, but I am not quite sure about it.

Answer

It comes from the cross product. Every situation in which you have to use the right-hand rule corresponds to some mathematical equation that involves a cross product.

In this case, the relevant equation is the Biot-Savart law,

$$\vec{B} = \frac{\mu_0}{4\pi}\int \frac{I\,\mathrm{d}\vec{l}\times\vec{r}}{r^3}$$

If you use the right-hand rule - the version you know to use for cross products - to compute the cross product of $\mathrm{d}\vec{l}$ and $\vec{r}$, like this (apologies for the crude drawing):

you will get the same result as from the curling-fingers form of the right-hand rule that you've shown in your question. That's by design, and in fact the curling-fingers form of the rule is just a shortcut for an infinite number of applications of the cross-product form.

In case you're curious, there is a deeper reason the right hand rule is needed for cross products. When you take the cross product of a vector and another vector, you get a slightly different mathematical object called a pseudovector or axial vector. The magnetic field at a point is the best-known example of a pseudovector. Despite looking just like a vector, pseudovectors actually represent a magnitude and an oriented plane, whereas an ordinary vector represents a magnitude and direction. Now, if you have a plane, and you want to represent it with an arrow, in a sense you can do that by picking the arrow to be perpendicular to the plane, and then your convention is that an arrow represents the plane that is perpendicular to it. But there are two arrows perpendicular to the plane; how do you choose which one to use? That's where the right-hand rule comes in. It plays a role in mapping a pseudovector (the thing that e.g. magnetic field really is) to a vector (the thing we use to represent e.g. magnetic field).