quantum field theory - Are group representations possible when the solution space is not a vector space?

As far as I understand, the motivation for using representation theory in high energy physics is as follows. Assume that a theory has some (internal or external) symmetry group which acts on a vector space. Then fields satisfying the theory will have to transform under some representation of that symmetry group, by construction.

What happens if we have some internal or external symmetry structure that is no longer acting on a vector space? The gauge group diffeomorphisms of general relativity spring to mind. Is there some more general 'representation' type theory which comes to our aid? And are there any examples of internal symmetries where this viewpoint is needed?

Apologies if this question is imprecise or flawed - I'm just starting to get my head around the foundations of the subject! Many thanks in advance!

Answer

Let $G$ be a group, e.g. a finite group or a Lie group.

Then there exists the notion of a group action $G\times X\to X$, where $X$ is a set. The set $X$ does not necessarily have to be a vector space. It could e.g. be a manifold. And even if $X$ has vector-space structure, the group action could be non-linearly realized, i.e., a group element $g\in G$ is represented by a non-linear operator $T_g:X\to X$.

Non-linear realizations pop up all over the place in modern physics. For instance, in nonlinear realization of supersymmetry, or in nonlinear realization of the conformal group.

Example: Let the Lie group $G=GL(2,\mathbb{C})$ of invertible $2\times2$ matrices

$$\tag{1} A~=~\begin{pmatrix}a & b\\c & d \end{pmatrix}, \qquad \det(A)\neq 0, $$

act on the complex plane $\mathbb{C}$ (which, by the way, is a vector space) as

$$\tag{2} A.z ~:=~\frac{az+b}{cz+d}, \qquad (AB).z ~=~ A.(B.z)~. $$

In this way, matrices get non-linearly represented as meromorphic functions. The subgroup $SL(2,C)$ is the global conformal group in two space-time dimensions, which e.g. plays a fundamental role in the world-sheet description of string theory.

Finally, let us mention that in mathematics there exists a generalization of the notion of a $\mathbb{F}$-vector space, where the field $\mathbb{F}$ is replaced by a ring $R$. It is known as an $R$-module.