quantum mechanics - Unitary representations of the diffeomorphism group in curved spacetime

In (special) relativistic quantum mechanics there is a standard argument that says that the (rigged) Hilbert space of states $H$ should be equipped with a projective unitary representation $U$ of the Poincaré group $P$ that goes something like this:

Suppose we have two observers $O_1$ and $O_2$ and $x\in P$ is the Poincaré transformation that maps $O_1$s coordinate system into $O_2$s coordinate system. Then, if $O_1$ and $O_2$ attempt to measure the same thing, they will actual find different states, $\left| \psi _1\right>$ and $\left| \psi _2\right>$ respectively, in $H$ (for example, if you first find the vector $(1,0,0)$, rotate your axes by $\pi/2$ and measure the same vector, the numbers you now record are going to be $(0,1,0)$). In fact, this gives us a map of states: $\left| \psi _1\right> \mapsto \left| \psi _2\right>$ (it's only going to be well-defined up to phase). Let us call this map $U(x)$, so that $\left| \psi _2\right> =U(x)\left| \psi _1\right>$.

However, if we accept the principle of relativity that physics should be the same in two reference frames related by a Poincaré transformation, then we better have that

$$\left| \left< \phi _1|\psi _1\right> \right| =\left| \left< \phi _2|\psi _2\right> \right| =\left| \left< U(x)\phi _1|U(x)\psi _1\right> \right|$$ for all (normalized) $\left| \psi _1\right>$ and $\left| \phi _1\right>$ because this represents a probability (of course, $\left| \psi _2\right> :=U(x)\left| \psi _1\right>$ and $\left| \phi _2\right> =U(x)\left| \phi _1\right>$).

Wigner's Theorem then tells us that this gives a projective unitary representation of $P$ on $H$. (Note: I am allowing some elements of $P$ to be represented by antiunitaries.)

Now, if you try to do the same argument in curved spacetime, you run into the obvious problem that you will not in general have an analogue of the Poincaré group (I believe there exist spacetimes which possess no Killing fields); however, it naively seems as if the principle of general covariance suggests that we should 'upgrade' the isometry group of spacetime to the entire diffeomorphism group of the spacetime in the above argument. (In particular, I don't see how this argument makes crucial use of the fact that both observes coordinate bases are orthonormal.) That then, would imply, that the Hilbert space of states in a quantum theory of curved spacetime should possess a projective unitary representation of the diffeomorphism group of that spacetime. This, however, at first glance, seems like it would be false.

So, where does the argument break down if you replace the Poincaré group in special relativity with the diffeomorphism group in general relativity? Or, is it in fact the case that we should obtain a unitary representation of the entire diffeomorphism group.