photons - $U(1)$ local gauge invariance in QED

While constructing Lagrangian of QED, we don't add the mass term for photon $\dfrac{1}{2} m^{2}A_{\mu}A^{\mu}$ because gauge invariance does not allow. I want to ask, whether "$\bf{Theoretically}$", is this the only reason we don't have mass term. I know why we need this term to vanish by using the experimental facts, but I feel like I am missing something while constructing the lagrangian for QED. Please stay within the domain of Quantum field theory and Lagrangian formulation while answering (if possible).