quantum electrodynamics - How does a photon "know" that it's left one charge and that it's going to another one?

How does it know the same charge it left will be the same charge it will return to? My understanding is photons are neutral and have no charge.

i.e.

Like charges repel, unlike attract. All charged particles emit photons which are uncharged. So how does the photon "know" that it's leaving one kind of charge and "lands" on another?

A neutral "particle" shouldn't be affected by charge.