quantum mechanics - Delayed choice experiment clarification (concrete setup)

I know that clarifications about delayed choice experiment was asked million times, and I understand the idea, but I was not able to find the discussion of this particular situation anywhere, though I tried hard. This setup is described in Brian Greene's Fabric of the Cosmos book (note my question is about modification of this setup, not about it exactly as described in book). Here is a picture of setup from the book

So photon goes from source to the beam splitter, then there are two down-converters that split photon into two, each of which goes in different directions: one follows the previous path (that one is named "signal") and the other goes into different direction to the detector ("idler" photon). Two signal photons then go to detector. Without down-converters we see interference pattern in detector - that seems clear. With down-converters it is claimed that we don't see interference pattern because detecting idler photons reveals information about which path photon has "chosen": if photon detected in top detector - photon has chosen top path, if not - then bottom one. So interference pattern disappears (so is claimed in the book).

Then book follows to description of modification of this setup to erase which path information - this part is clear for me and it's not what I'm interested about. What is not clear is:

What if we remove idle photon detectors from the setup, but leave down-converters in place? Then it seems "which path" information is not revealed and interference pattern should appear again? But that does not make much sense, because we can decide to put or not a detector at much later time (after corresponding singal photon hit the screen). So depending on if we see pattern or not we should know what happened in the future, which of course cannot happen. Is that assumption about removing idler detectors leads to interference pattern incorrect? If so, why? What will happen if we switch idler detectors on and off by our choice? I suspect this has something to do with the fact we now have two photos instead of one (two photons from down-converters). Maybe those photons have different phases and combined they do not form any pattern?