Why can't compact symplectic groups $Sp(n)equiv USp(2n)equiv U(2n)cap Sp(2n,mathbb{C})$ be gauge groups in Yang-Mills theory?

The gauge groups in Yang-Mills theory can be things like $O(10)$ or $SU(5)$ but continuing the pattern from real to complex, the next obvious thing would be quaternion matrices. A group like $U(4,H)$ where $H$ is the quaternions. This is another name for $Sp(4)$ (according to Wikipedia!).

A group like $U(4,H)$ I always thought would be interesting since it would be split $U(1,H)\times U(3,H)$ and $U(1,H)=SU(2)$ and $U(3,H)$ would have subgroup $SU(3)$.

But I have never seen a Yang-Mills theory with a compact symplectic gauge group so apparently there must be a good reason for that.

Do you know the reason? Is there a theoretical reason or an experimental reason?

Answer

The structure of standard model $SU(3)\times SU(2)\times U(1)$ is chiral which basically tells you the necessity of chiral fermions. If left-handed fermions transform under a representation $R$ of the symmetry group then due to charge-conjugation relating left-handed and right-handed fermions as $$\psi_{Right}=C(\bar{\psi^C})^T_{Left}$$ and so, right handed fermions should transform under the complex conjugate representation $R^*$. If $R$ is real or pseudoreal, then left-handed and right-handed fermions transform in same representation of the group and the theory is known to be a vector like theory (QCD). To have chiral structure of fermions, one has to have $R \ne R^*$ which demands $R$ to be complex.

Even though QCD is vector like and $2=2^*$, the whole standard model is chiral as can be seen by writing $R$ for left-handed fermions as, $$R=(3,2)_{\frac{1}{6}}+(3^*,1)_{\frac{-2}{3}}+(3^*,1)_{\frac{1}{3}}+(1,2)_{\frac{-1}{2}}+(1,1)_{1}$$ the complex conjugate to which is not same as $R$.

It is known that $USp(2n)$ for $n>2$ admits real and pseudo-real representations (Weinberg Vol. 2, chapter 22) and $USp(4)$ is not big enough to contain standard model.

Moreover using a $Sp(n)$ like gauge group demands even number of fermion multiplets otherwise the gauge theory will show a non-perturbative anomaly$^1$ involving fourth homotopy group of $Sp(n)$.

1- Ed Witten, nucl. phys. B223 (1983),433-444.