lagrangian formalism - Why is Fermat's principle not formulated as principle of least action?

I noticed from the units of $S$ that despite the notational similarity Fermat's principle $$\delta S= \delta\int_{\mathbf{A}}^{\mathbf{B}} n \, ds =0$$ is not a principle of least action but a principle of least length. Confusingly one often writes this principle even using a so called "optical Lagrangian" $L= n \frac{ds}{dx_3}$ as $$\delta S= \delta\int_{\mathbf{A}}^{\mathbf{B}} L \, dx_3 =0$$ However this Lagrangian doesn't have units of energy as the usual Lagrangian $L= T-V$ has. Instead the optical Lagrangian has no units. So I wondered what is missing to make Fermat's principle into a principle of least action and figured from the units that one has to multiply the principle with some "optical momentum" $p_O$ so $S$ becomes an action: $$\delta S= p_O\ \delta \int_{\mathbf{A}}^{\mathbf{B}} n \, ds =0$$ But what should this $p_O$ be? Since we are talking about light we could set $p_O=\hbar k = \frac{h}{\lambda}$. But Fermat's principle is the basis of geometrical optics, which can be derived as the limit of zero wavelength $\lambda \rightarrow 0$ from Maxwells wave equations of light. And a zero wavelength means light of infinite momentum according to $p_O= \frac{h}{\lambda}$. So it doesn't seem to make sense to multiply Fermat's principle with a momentum, because that would yield a infinite value for the action.

So is the reason why we cannot express Fermat's principle as principle of least action the fact that geometrical optics implies an infinite momentum for light?