cosmology - Why is acceleration of expansion of space unexplained?

I didn't know how to better phrase the question so here we go.

It says that the farther we see from our galaxy the faster the other object seems to be going away (accelerating).

My common sense tells me to expect this due to following reason.

If space is expanding at every point equally then farther objects would naturally move away faster as there are more points of space between you and a far object than points of space between you and near object.

So as we move farther from an object the amount of space (say no of space points of length dx) increases hence there is more space expanding away from each other than for a near object and that is why far away object would appear to be accelerating.

But obviously its not that simple as i said or i may be completely wrong as scientists seem to consider this acceleration to non understood phenomenon like dark energy

So What am i missing here?

Answer

The FLRW energy equation for the motion of test masses in the universe is

$$\left(\frac{\dot a}{a}\right)^2 = \frac{8\pi G\rho}{3}.$$ the scale factor for space is $a$ and its time derivative is $\dot a$. I derived this from Newtonian dynamics. The density of mass $\rho$ for the case of a quantum vacuum energy level is constant. I now replace this with energy density with $\rho\rightarrow~\rho/c^2$. This leads to the dynamical equation $$a(t) = a_0e^{t\sqrt{\frac{8\pi G\rho}{3c^2}}}$$ which is an exponential expansion. The cosmological constant is then $\Lambda~=\frac{8\pi G\rho}{3c^2}$, which is determined by the quantum vacuum energy.

The question is then what is the density of mass-energy, or more exactly what is the nature of the quantum vacuum. The vacuum is filled with virtual quanta. A pendulum sitting vertically will be a motionless plumb in classical mechanics. However, quantum mechanics informs us there is an uncertainty in its position and momentum $\Delta x\Delta p~\simeq~\hbar$. This means it can fluctuate about its vertical plumb position. The Hamiltonian for the harmonic oscillator transitions from the classical to quantum form as

$$H = \frac{1}{2m}p^2 + \frac{k}{2}x^2~\rightarrow~\frac{\omega}{2}(a^\dagger a + aa^\dagger)$$ This can by put in the more standard form with the quantum commutator $[a,~a^\dagger]~=~1$, and so $aa^\dagger = a^\dagger a + \frac{1}{2}$. The quantum Hamiltonian is then $$H = \omega a^\dagger a + \frac{1}{2}\omega,$$ where this last term is a zero point energy for the fluctuation of the harmonic oscillator. For a quantum field there is a big summation over Hamiltonians for each frequency $$H = \sum_n\omega_n \left(a_n^\dagger a_n + \frac{1}{2}\right)$$ The sum over this residual energy or zero point energy is summed up to the Planck energy. This leads to a huge vacuum energy. For most quantum work this term is eliminated, often by something called normal ordering. However this leads to a huge energy that can't be ignored in cosmology. The cosmological constant is $\Lambda \simeq 10^{-53}cm^{-2}$. The sum over these zero point energy leads to and expected $\Lambda \simeq 10^{67}cm^{-2}$. The difference is $120$ orders of magnitude off.

There is a lot of work on this subject. Much of it focuses on gauge fluxes through wrapped D-branes. Some progress has been made in reducing the expected vacuum energy. However, so far it has not been possible to show the very small vacuum energy we know from cosmology exists. It also can't be zero! It is my thinking that this reflects our lack of understanding in quantum gravity. We know some things about quantum gravity, but we really do not have a complete theory of it. This vacuum energy that propels cosmological expansion, called dark energy, is then not fully understood.