forces - Torque direction meaning

I apologize if this question is dumb, but I've looked all over for a straightforward answer and either I can't find one or the terms are too complex for me to understand. I have only a rudimentary knowledge of Mechanics, but I do understand basic Linear Algebra.

So torque, mathematically, is the cross product of the radial distance vector and a force vector. This cross product gives another vector that is orthogonal to both vectors and it points either outside or towards the "page" (in the context of a two dimensional diagram).

Assuming this is correct, I do not understand what it pointing in or out means. Does it even have a phyisical, intuitive meaning?

The best answer I've been able to come up with is that it's just a mathematical convention with no actual phyisical meaning, meant to provide a framework within which operations between torque vectors, such as addition and substraction, make sense.

Am I correct or way off the mark here?

Answer

As in the comments, there's certainly something of a convention at work here and it's to do with the "co-incidence" that we live in three spatial dimensions.

As in Greg's answer, torque is intimately linked with angular momentum through Euler's second law. That is, torque and angular momentum are about rotational motion. And rotations, in general, are characterized by the planes that they rotate together with the angles of rotation for each of these planes. In three dimensions, the plane of rotation can be defined by a single vector - namely the vector orthogonal to the plane. So we have the concept of the "axis" of rotation, but this is not general, its simply that a line happens to be the subspace of a three dimensional vector space that is orthogonal to the plane of rotation. In four and higher $N$ spatial dimensions, the concept of an axis is meaningless: not only does an axis not specify a plane (the space orthogonal to a plane is of dimension $N-2$), but also a general rotation rotates several planes (up to and including the biggest whole number less than or equal to $N/2$).

So the "true" information specifying a three dimensional rotation is the "bivector" $A\wedge B$, where $A, B$ are linearly independent vectors defining the plane, and a bivector is an abstract directed "plane" just like a "vector" is an abstract directed "line". Cross products in three dimensions are actually bivectors, not vectors, but we can get away with thinking of them as such in three dimensions.

Some further reading to help you out: the Wikipedia pages Plane Of Rotation, Rotation Matrix and Orthogonal Group (rotation matrices form the group $SO(N)$, the group of orthogonal matrices with unit determinant).