operators - Why does non-commutativity in quantum mechanics require us to use Hilbert spaces?

I am reading Why we do quantum mechanics on Hilbert spaces by Armin Scrinzi. He says on page 13:

What is new in quantum mechanics is non-commutativity. For handling this, the Hilbert space representation turned out to be a convenient — by many considered the best — mathmatical environment. For classical mechanics, working in the Hilbert space would be an overkill: we just need functions on the phase space.

I understand the definition of a Hilbert space, but I do not understand why non-commutativity compels us to use Hilbert spaces.

Can someone explain?

Answer

I understand the definition of a Hilbert space. But I do not understand why non-commutativity compels us to use Hilbert spaces.

It doesn't, but that's not what Scrinzi is saying.

The reason is doesn't is because we could work, for example, in Wigner quasiprobability representation: $$\rho\mapsto W(x,p) = \frac{1}{\pi\hbar}\int_{-\infty}^\infty\langle x+x'|\rho|x-x'\rangle e^{-2ipx'/\hbar}\,\mathrm{d}x'\text{,}$$ where for pure states $\rho = |\psi\rangle\langle \psi|$ as usual, and Hermitian operators correspond to functions through the inverse Weyl transformation.

Note that $W(x,p)$ is a real function that's just like a joint probability distribution over the phase space, except that it's allowed to be negative. The uncertainty principle requires us to give up something, but it doesn't actually force Hilbert spaces on us.

However, what Scrinzi is saying is two things: (a) Hilbert spaces are very convenient to us in quantum mechanics, and (b) Hilbert spaces could be used in classical mechanics, but because non-commutativity doesn't exist in classical mechanics, it's "overkill" there, whereas it's "just the right amount of kill" in quantum mechanics. Both claims are correct.

The reason we can have used Hilbert spaces in classical mechanics is because they can represent very general algebras of observables, while the classical algebra of observables, being commutative, is actually simpler. (Cf. the Gel'fand–Naimark theorem for $C^*$-algebras in particular.)

The Hilbert space formulation of classical mechanics was done by Koopman and von Neumann in 1931-1932. But what their formulation actually does is generalize classical mechanics unless one imposes an artificial restriction that you're only ever allowed to measure observables in some mutually commutative set; only then is the classical (in the 19th century sense) mechanics is recovered exactly.

It is that artificial restriction that quantum mechanics lifts. Physically, non-commutativity observables corresponds to an uncertainty principle between them.