quantum field theory - How do derivative couplings affect canonical quantization?

Consider a Lagrangian for a scalar field $\phi$ with an interaction term

$$\mathcal{L}_{int} = (\partial^2 \phi)^2 \phi.$$ Here I'm suppressing all indices for brevity. Now, this is just a three-point interaction with some extra momentum factors, so there seems to be no problem writing down the Feynman rule.

However, actually trying to carry out the procedure with canonical quantization looks tricky. The interaction term can't be written as a function of $\phi$ and $\partial \phi$, even if you use integration by parts, so we must let the Lagrangian have $\partial^2 \phi$ dependence. But then I don't know how to produce the Hamiltonian, because that's based on a Legendre transformation from $(\phi, \partial \phi)$ to the fields and momenta $(\phi, \pi)$. The new dependence on $\partial^2 \phi$ seems to totally mess this up; I don't know how to handle it in classical mechanics, let alone QFT!

How do you actually obtain the Feynman rule for this interaction, in canonical quantization?

Answer

Comments to the question (v2):

1. 
   

   OP is considering the higher-derivative Lagrangian density

   $${\cal L}_1~=~ \frac{1}{2}(\partial\phi)^2 +\frac{g\phi}{2} (\partial^2\phi)^2,\tag{1}$$ where $g$ is a coupling constant. We use Minkowski sign convention $(+,-,-,-)$.
   
   


2. 
   

   Quantum mechanically, the model is not unitary and therefore ill-defined, cf. the Ostrogradsky instability. However, classically (and as a formal perturbative expansion in Feynman diagrams), it makes sense.

   
   


3. 
   

   One can in principle apply the Ostrogradsky procedure for higher-derivative theories. However, here we will just follow our nose: It seems natural to try to lower the number of derivatives by imposing a constraint $\Phi\approx \partial^2\phi$ via a Lagrange multiplier $B$. In other words, consider the Lagrangian density

   $${\cal L}_2~=~\frac{1}{2}(\partial\phi)^2 +B\Phi+\partial_\mu B~\partial^{\mu}\phi+\frac{g\phi}{2} \Phi^2 ~\sim~ \frac{1}{2}(\partial\phi)^2 +B(\Phi-\partial^2\phi)+\frac{g\phi}{2} \Phi^2 ,\tag{2}$$ where the $\sim$ symbol means equality modulo total spacetime divergence terms.
   
   



4. 
   

   It is tempting to integrate out the $\Phi$ variable again. Then we arrive at the interesting Lagrangian density

   $${\cal L}_3~=~\frac{1}{2}(\partial\phi)^2 +\partial_\mu B~\partial^{\mu}\phi-\frac{B^2}{2g\phi} .\tag{3}$$
   
   


5. 
   

   In other words, the Lagrangian densities (2) and (3) reduce to the original Lagrangian density (1) when we integrate out the new variables
   

   $${\cal L}_2\quad\stackrel{\Phi}{\longrightarrow}\quad{\cal L}_3\quad\stackrel{B}{\longrightarrow}\quad{\cal L}_1 .\tag{4}$$ The Lagrangian densities (2) and (3) do not have higher-order derivatives. Therefore the usual techniques can be applied to find the Hamiltonian formulation and Feynman rules. (In the ${\cal L}_3$ case, the $\phi$ field should apparently be expanded around a non-zero classical solution. Perhaps a field redefinition $\phi=e^{\varphi}$ would be useful?) We leave that as an exercise to the reader.