It has worked!
In a bit more detail:
I didn't add entries for P3 and P4 on retroarch.cfg because, what would I set them to if I only have two controllers? 4 and 5, assuming that I could be plugging in additional ones?
Anyway, I tried to re-add the two physical Logitech controllers to EmulationStation; it detect 4 controllers, but it would keep detecting the button presses as coming from the emulated Xbox controllers.
I then reset the controller settings for EmulationStation on the Retropie menu and again, it would detect 4 controllers, but it also would keep detecting the button presses as coming from the emulated Xbox controllers.
I had to disable xboxdrv and then it would only detect 2 controllers and the button presses a coming from the physical Logitech controllers. The conclusion here is for EmulationStation the physical and emulated controllers don't clash; the emulated ones take priority and the button presses are understood to be coming from them.
I then checked that on the Retroarch menu the emulated controllers were assigned to P1 and P2 and removed the physical controllers that were assigned to P3 and P4. And then everything was fine. The crux here appears to be that I had a clash, as both the emulated and physical controllers were assigned to players and probably registering button presses. Something that support this hypothesis is that on games where you have a grid to select your character (e.g. Mario Kart) it was fairly obvious that when I was pressing once to move one cell to the right, for instance, the selection box was moving two cells.
Thanks again for all your time helping me figure this out.