So to update.. Yes it the retroarch-joyconfig idea I had above works :)
I finally got my head around the joystick stuff
Global style:
input_a_btn
Individual Player style
input _playerx _a_btn
So if you only want a certain brand of gamepad to take over for player 3 and 4, you'd edit the retroarch-joypad/mygamepad.cfg file and add the "_player3" bit there. You can add a _player4 copy as well. This will leave player1 and player2 alone and force the gamepads to 3 and 4.
Alternatively, if you don't want it to be limited to certain gamepads, you can remove the retroarch-joypad/ files and just put those same lines directly into the main retroarch.cfg file (or per system if you like) and have any gamepad of the same mapping style work. This is good for those generic chinese ps2 style gamepads with the usb dongle.
As herb_fargus shows above, follow the usb index and set the indexing in the retroarch.cfg with
input_player3_joypad_index = "2"
input_player4_joypad_index = "3"
Which doesn't necessarily follow the usb ports one for one... but those usb ports assign the default indexing on startup. For example, I have a dongle that supports 2 ps2 gamepads in one, but it still indexes them as 2 and 3 because they are loading up in the system as those indexes, regardless of actual physical usb port.
Thx all