general relativity - What does a frame of reference mean in terms of manifolds?

Because of my mathematical background, I've been finding it hard to relate the physics-talk I've been reading, with mathematical objects.

In (say special) relativity, we have a Lorentzian manifold, $M$. This manifold has an atlas with local coordinates.

In differential geometry, when people talk about a $\phi_1: U_1 \rightarrow V$ where $V$ is an open set of $M$, and $U$ is an open set of $\mathbb{R}^4$; and if another is $\phi_2: U_2 \rightarrow W$ is another ($U_2$ and open in $\mathbb{R}^4$, and $W$ an open in $M$), then $\phi_1^{-1} |_{V\cap W}\circ \phi_2|_{\phi_2^{-1}(V\cap W)}$ is a coordinate change.

However, in physics it seems that the meaning is different. Indeed if $p \in M$ then you can have a reference frame at $p$, but you can also have a reference frame that is accelerated at $p$. I'm not sure how to interpret this mathematically! What is the mathematical analogue of having an accelerated frame of reference at a point, as opposed to having an inertial frame of reference at a point?