Adiabatic Quantum Computing: why not just set the system in its problem Hamiltonian $H_{P}$ immediately?

Background: In any adiabatic quantum computer (AQC) algorithm, we solve problems in the following manner: We have an initial Hamiltonian, $H_{0}$, whose ground state is easy to find, and a problem Hamiltonian $H_{P}$, whose ground state encodes the solution to our problem. If we then evolve our AQC for a time $T$ so that its energy is described by the Hamiltonian $$H(t) = (1-t/T)H_{0} + (t/T)H_{P}$$ then provided a couple of conditions apply, the system will be in the ground state of $H_{P}$ at time $T$ (and voila, we would have a solution to our problem)

Question: If we just set up the AQC so its energy is initially described by the Hamiltonian $H_{P}$, why wouldn't the system just 'fall' into its ground state (encoding a solution to our problem immediately)? Why do we need to evolve the AQC from the initial Hamiltonian $H_{0}$ into $H_{P}$?

Answer

I once asked the exact same question during a course on quantum computation. Systems only "fall into" their ground states when they are in thermal equilibrium at zero temperature. Both of these pieces are problematic: (a) many systems that have been proposed for quantum computation have energy scales low enough that getting them down to sufficiently low temperatures is extremely challenging, and (b) as Peter Shor pointed out, you have no idea how long it will take for the system to actually reach thermal equilibrium - you could have a physical equivalent of a Monte-Carlo sign problem, where it takes exponentially long in system size for local perturbations to get you into thermal equilibrium.

But if you can control the initial Hamiltonian $H_0$, you can "force" the system into its ground state much more quickly - in principle by measurement filtration, but more realistically by making $H_0$ unfrustrated and with a very large characteristic energy scale. For example, if you have a system of quantum spins and you apply a huge uniform field to the whole system ("huge" meaning much larger than the temperature and the relevant spin interaction scale), then the whole system will align with the field very quickly and you can be confident that you're in the ground state.