cosmology - What are quantum fluctuations, really?

So, I have read a lot about quantum fluctuations, and how they are responsible for:

• Lamb shift


• Spontaneous emission of photons from excited atomic states


• Casimir effect


• ...

and the explanation always mentions either or both of these two things:

• $E$ being the energy, $\langle E\rangle = 0$ but $\langle E^2\rangle \neq 0$ because of the $\Delta E \Delta t > \hbar$ uncertainty principle, so actually there is some temporal appearance of energy


• The vacuum is full of pairs of virtual particles and antiparticles annihilating each other, and for a time $\Delta t$ dictated by the above uncertainty principle, they can interat with real particles.

---

Now, I fall in the line of though that:

• $\Delta E \Delta t > \hbar$ is not a real uncertainty principle, as t is not an operator. This expression is derived from Ehrenfest theorem and quantifies the maximum change that the energy eigenvalue can undergo in an interval $\Delta t$.


• Virtual particles do not exist, they are just an artefact of perturbative expansions.

---

Is there a physical, QFT-free, qualitative way to understand quantum fluctuations? And in order to make sense of the very real phenomena associated with them, like the ones I listed at the beginning?

Is there a relation to the ground state energy problem leading to the cosmological constant problem?

Answer

You asked for a qualitative picture, so here goes.

Consider a simplified example: the quantum harmonic oscillator.

Its ground state is given by

$$ \Psi(x) = \text{const} \cdot \exp \left( - m \omega_0 x^2 / 2 \hbar \right). $$

Now suppose that we are measuring the position of this oscillator in the ground state. We could get any real value, with probability density $|\Psi|^2$. In reality, because of the exponential decay, most of the values are distributed within the window of width

$$ \Delta x \sim \sqrt{\frac{2 \hbar}{m \omega_0}}, $$

with the mean concentrated at $x = 0$.

Because measuring an individual oscillator is a complicated process which results in it getting entangled with the measurement device, let's simplify the problem – say we have an ensemble of non-interacting oscillators all in ground states, and we measure them all independently. The distribution of values $\{x_i\}$ is expected to mostly lie within the mentioned above window, but the actual values are unknown. We usually say that those are due to quantum fluctuations of the position operator.

The same thing happens with the quantum field, which upon inspection is nothing more than a collection of weakly interacting harmonic oscillators. If we take an ensemble of vacuum quantum field configurations (say, independent experiments at a particle accelerator), and we measure a value of the field at a point, we will see that it is not equal to zero (as it would be in the classical theory), but instead the values are distributed within an error window and are otherwise random. This are quantum fluctuations of the QFT vacuum.

These fluctuations are sometimes attributed to "virtual particles", or "virtual pairs", which are said to be "born from the vacuum". Sometimes it is also said that they can "borrow energy from vacuum for a short period of time". AFAIK these are just analogies, appealing to the consequence of Erenfest's theorem (the so-called time-energy uncertainty relation).

But the fluctuations undisputably have very real, measurable effects. Qualitatively, those effects come from a difference between the physical picture of the same thing painted by classical fields and quantum fields. You can say that quantum fields reproduce classical fields on certain scales (measured in the field value), which are much greater than the size of the error window. But once the precision with which you measure field values becomes comparable to the size of the error window, quantum effects kick in. Those who like painting intuitive pictures in their heads say that this is caused by quantum fluctuations, or virtual particles.

UPDATE

Belief that observed Casimir effect has something to do with vacuum fluctuations of the fundamental QFT is misguided. In fact, in the calculation of the Casimir force we use an effective field theory – free electromagnetism in the 1D box, bounded by the two plates. Then we look at the effective vacuum state of this effective QFT, and we interpret the Casimir force as a consequence of the dependence of its properties on the displacement between the plates, $d$.

From the point of view of the fundamental QFT however (Standard Model, etc.) there is no external conducting plates in the first place. If there were, it would violate Lorentz invariance. Real plates used in real experiments are made of the same matter described by the fundamental QFT, thus the state of interest is extremely complicated. What we observe as Casimir force is really just a complicated interaction of the fundamental QFT, which describes the time evolution of the complicated initial state (which describes the plates + electromagnetic field in between).

It is hopeless to try to calculate this in the fundamental QFT, just like it is hopeless to calculate the properties of the tennis ball by studying directly electromagnetic interactions holding its atoms together. Instead, we turn to the effective description, which captures all the interesting properties of our setup. In this case it is free electromagnetic effective QFT in the 1D box.

So to summarize: we are looking at the vacuum state of the effective QFT and the dependence of its properties on $d$. Alternatively, we are observing an extremely complicated fundamental system in a state which we can't hope to describe.