general relativity - Does spacetime position not form a four-vector?

When one starts learning about physics, vectors are presented as mathematical quantities in space which have a direction and a magnitude. This geometric point of view has encoded in it the idea that under a change of basis the components of the vector must change contravariantly such that the magnitude and direction remain constant. This restricts what physical ideas may be the components of a vector (something much better explained in Feynman's Lectures), so that three arbitrary functions de not form an honest vector $\vec{A}=A_x\hat{x}+A_y\hat{y}+A_z\hat{z}$ in some basis. So, in relativity a vector is defined "geometrically" as directional derivative operators on functions on the manifold $M$ and this implies, if $A^{\mu}$ are the components of a vector in the coordinate system $x^\mu$, then the components of the vector in the coordinate system $x^{\mu'}$ are

$$A^{\mu'}=\frac{\partial x^{\mu'}}{\partial x^\mu}A^\mu$$ (this all comes from the fact that the operators $\frac{\partial}{\partial x^\mu}=\partial_\mu$ form a basis for the directional derivative operators, see Sean Carrol's Spacetime and Geometry)
My problem is the fact that too many people use the coordinates $x^\mu$ as an example of a vector, when, on an arbitrary transformation, $$x^{\mu'}\neq\frac{\partial x^{\mu'}}{\partial x^\mu}x^\mu$$ I understand that this equation is true if the transformation beween the two coordinates is linear (as is the case of a lorentz transformation between cartesian coordinate systems) but I think it can´t be true in general. Am I correct in that the position does not form a four-vector? If not, can you tell me why my reasoning is flawed?