hilbert space - Gauge invariance and the unitarity

I want to discuss the relation between the unitarity and the gauge invariance. Suppose we have for simplicity an abelian gauge theory (say, EM theory). We want to quantize it in terms of 4-potential $A_{\mu}$.

1) It turns out that if we impose the covariant gauge (like $\partial_{\mu}A^{\mu} = 0$) on the physical state $|\Psi\rangle$, namely $$ \langle\Psi_{1}|\partial_{\mu}A^{\mu}|\Psi_{2}\rangle = 0, $$ then it turns out that the physical state contains the equal number of time-like and longitudinal photons; they are associated with creation-destruction operators attached to $A_{0}, \mathbf{A}(p)\cdot\mathbf p$ respectively, and the former (time-like) has negative norm. But the latter (longitudinal) provides that the contribution of these photons in the norm of the physical state is zero, which is due to equal numbers of longitudinal and time-like photons. In the literature this procedure of the quantization is called the Gupta-Bleuler quantization.

In this gauge the relation between unitarity and gauge invariance is thus obvious: by breaking the gauge invariance we thus break the unitarity, since the negative norm states enter the game.

2) Suppose now we want to quantize the theory in a gauge $A_{0}= 0$. There is a remaining set of time independent gauge transformations related to longitudinal degrees of freedom, for which the Gauss constraint $$ G(x) = \nabla \cdot \mathbf{E} - \rho $$ is a Noether current. Precisely, the hamiltonian $H$ of the theory commutes with $G$. To eliminate such degrees of freedom from the physical state, we have to require $$ \tag 1 G(x)|\Psi\rangle = 0, \ \ \ \ \text{ or, equivalently, } \ \ \ \ [G(\mathbf {x}), H(\mathbf{y})] = 0, $$ where $[,]$ denotes the commutator and $H$ is the hamiltonian.

3) Next, suppose the breaking of the gauge invariance by the gauge anomaly. Let's choose the gauge $A_{0} = 0$. Then anomalous breaking of the gauge symmetry provides violation of $(1)$: $$ [G(\mathbf {x}), H(\mathbf{y})] \sim \mathbf A \cdot [\nabla \times \mathbf A]\delta(\mathbf x - \mathbf y) $$
Thus the anomalous breaking implies the presence of the longitudinal photons.

Next, people often say that the gauge anomaly breaks the unitarity. But I don't see how the presence of longitudinal photons in the physical state causes the violation of the unitarity. It looks out that with the anomaly the photon mass is generated.

My question: Could you clarify where the unitarity violation is hidden in the $A_{0}=0$ gauge after anomalous breaking of the gauge invariance? I.e., how the longitudihal photons break the unitarity?

P.S. There is the fact that the unitary representation of the Poincare group for massless state with helicity 1 (our ''photons'') can't be realized by the 4-vector $A_{\mu}$. Precisely, if we treat $A_{\mu}(x)$ as the field whose Fourier image produces the one-particle photon state, then in the momentum space we're faced with lorentz non-covariance of the polarization vectors $\epsilon_{\mu}(p)$: under Lorentz transformations given by the matrix $\Lambda_{\mu}^{\ \nu}$ we have $$ \epsilon_{\mu}(p) \to \Lambda_{\mu}^{\ \nu}\epsilon_{\nu}(p) + c p_{\mu} $$ Then the gauge invariance (given in terms of the Ward identities) is required in order to maintain the unitarity and the Poincare covariance. This observation was made by Weinberg.

Although it is true and seems to answer on my question (why the gauge invariance is required for the unitarity independently on the gauge fixing), people who may be not familiar with his argument anyway state that the gauge anomalies ruins the gauge invariance and hence the unitatity. So I want to know whether the another explanation of the relation between the gauge invariance and the unitarity exists.