What justifies dimensional analysis?

Dimensional analysis, and the notion that quantities with different units cannot be equal, is often used to justify very specific arguments, for example, you might use it to argue that a particular formula cannot possibly be the correct expression for a particular quantity. The usual approach to teaching this is to go "well kids, you can't add apples and oranges!" and then assume that the student will just find it obvious that you can't add meters and seconds.

I'm sorry, but... I don't. I'm not convinced. $5$ meters plus $10$ seconds is $15$! Screw your rules! What are the units? I don't know, I actually don't understand what that question means.

I'm specifically not convinced when this sort of thing is used to prove that certain formulae can't possibly be right. Maybe the speed of a comet is given by its period multiplied by its mass. Why not? It's a perfectly meaningful operation - just measure the quantities, multiply them, and I claim that the number you get will always equal the current speed of the comet. I don't see how "but it doesn't make sense to say mass times time is equal to distance divided by time" can be a valid counterargument, particularly because I don't really know what "mass times time" is, but that's a different issue.

If it's relevant, I'm a math student and know extremely little about physics.

Answer

Physics is independent of our choice of units

And for something like a length plus a time, there is no way to uniquely specify a result that does not depend on the units you choose for the length or for the time.

Any measurable quantity belongs to some set $\mathcal{M}$. Often, this measurable quantity comes with some notion of "addition" or "concatenation". For example, the length of a rod $L \in \mathcal{L}$ is a measurable quantity. You can define an addition operation $+$ on $\mathcal{L}$ by saying that $L_1 + L_2$ is the length of the rod formed by sticking rods 1 and 2 end-to-end.

The fact that we attach a real number to it means that we have an isomorphism

$$u_{\mathcal{M}} \colon \mathcal{M} \to \mathbb{R},$$ in which $$u_{\mathcal{M}}(L_1 + L_2) = u_{\mathcal{M}}(L_1) + u_{\mathcal{M}}(L_2).$$ A choice of units is essentially a choice of this isomorphism. Recall that an isomorphism is invertible, so for any real number $x$ you have a possible measurement $u_{\mathcal{M}}^{-1}(x)$. I'm being fuzzy about whether $\mathbb{R}$ is the set of real numbers or just the positive numbers; i.e. whether these are groups, monoids, or something else. I don't think it matters a lot for this post and, more importantly, I haven't figured it all out.

Now, since physics should be independent of our choice of units, it should be independent of the particular isomorphisms $u_Q$, $u_R$, $u_S$, etc. that we use for our measurables $Q$, $R$, $S$, etc. A change of units is an automorphism of the real numbers; given two units $u_Q$ and $u'_Q$, the change of units is

$$\omega_{u,u'} \equiv u'_Q \circ u_Q^{-1}$$ or, equivalently, $$\omega_{u,u'} \colon \mathbb{R} \to \mathbb{R} \ni \omega(x) = u'_Q(u_Q^{-1}(x)).$$ Therefore, $$\begin{align} \omega(x+y) &= u'_Q(u_Q^{-1}(x+y)) \\ &= u'_Q(u_Q^{-1}(x)+u_Q^{-1}(y)) \\ &= u'_Q(u_Q^{-1}(x)) + u'_Q(u_Q^{-1}(y)) \\ &= \omega(x) + \omega(y). \end{align}$$

So, since $\omega$ is an automorphism of the reals, it must be a rescaling $\omega(x) = \lambda x$ with some relative scale $\lambda$ (As pointed out by @WetSavannaAnimalakaRodVance, this requires the weak assumption that $\omega$ is a continuous function -- there are everywhere discontinuous solutions as well. Obviously units aren't useful if they are everywhere discontinuous; in particular, so that instrumental measurement error maps an allowed space of $u_\mathcal{M}$ into an interval of $\mathbb{R}$. If we allow the existence of an order operation on $\mathcal{M}$, or perhaps a unit-independent topology, this could be made more precise.).

Consider a typical physical formula, e.g.,

$$F \colon Q \times R \to S \ni F(q,r) = s,$$ where $Q$, $R$, and $S$ are all additive measurable in the sense defined above. Give all three of these measurables units. Then there is a function $$f \colon \mathbb{R} \times \mathbb{R} \to \mathbb{R}$$ defined by $$f(x,y) = u_S(F(u_Q^{-1}(x),u_R^{-1}(y)).$$

The requirement that physics must be independent of units means that if the units for $Q$ and $R$ are scaled by some amounts $\lambda_Q$ and $\lambda_R$, then there must be a rescaling of $S$, $\lambda_S$, such that

$$f(\lambda_Q x, \lambda_R y) = \lambda_S f(x,y).$$

For example, imagine the momentum function taking a mass $m \in M$ and a velocity $v \in V$ to give a momentum $p \in P$. Choosing $\text{kg}$ for mass, $\text{m/s}$ for velocity, and $\text{kg}\,\text{m/s}$ for momentum, this equation is

$$p(m,v) = m*v.$$ Now, if the mass unit is changed to $\text{g}$, it is scaled by $1000$, and if the velocity is changed to $\text{cm/s}$, it is scaled by $100$. Unit dependence requires that there be a rescaling of momentum such that $$p(1000m,100v) = \lambda p(m,v).$$ This is simple -- $10^5 mv = \lambda mv$ and so $\lambda = 10^5$. In other words, $$p[\text{g} \, \text{cm/s}] = 10^5 p[\text{kg} \, \text{m/s}].$$

Now, let's consider a hypothetical situation where we have a quantity called "length plus time", defined that when length is measured in meters and time in seconds, and "length plus time" in some hypothetical unit called "meter+second", the equation for "length plus time" is

$$f(l,t) = l + t.$$ This is what you've said - $10 \text{ m} + 5 \text{ s} = 15 \text{ “m+s”}$. Now, is this equation invariant under a change of units? Change the length scale by $\lambda_L$ and the time scale by $\lambda_T$. Is there a number $\Lambda$ such that $$f(\lambda_L l, \lambda_T t) = \lambda_L l + \lambda_T t$$ is equal to $$\Lambda f(l,t) = \Lambda(l+t)$$ for all lengths and times $l$ and $t$? No! Therefore, this equation $f = l + t$ cannot be a valid representation in real numbers of a physical formula.