@jakejm79 Did you ever figure this out? I was thinking it would be awesome to use the same idea to unlock the console with key inputs to be able to use a coin dispensor. But am not finding anyway to do what you were requesting either.
The 'Auto Config Buttons' dialog allows selection of single keys for each of the control functions, then, right at the end, it asks for the 'hot key' button. Without a 'hot key' assignment the control buttons work on their own. If you hold a key down for more than 2 seconds while at any selection (including for 'hot key') the dialog will assign 'nul' as the button.
If you don't have very many buttons/inputs then the use of a 'hot key' + 'game button' is the only way to control Retropie. This also is the mode you will have to use if you use genuine console pads which do not have any control buttons, just game action buttons.
If you do have spare buttons/inputs then assign the control functions to unique buttons and set the 'hot key' to nul. This can be done by running the config buttons dialog manually with - 'Settings -> Inputs -> User 1 Binds -> User 1 Bind Default All' or by manually editing the file it creates at '/opt/retropie/configs/all/retroarch.cfg'
I'm also new to this and awaiting the delivery of extra hardware to experiment with how to add an extra Zero Config board that is used exclusively for the control buttons. (Maybe configure it as Player 5? does anyone know how to do this?)
@stevepax Hello and belated welcome to this forum. I'm close to finishing my own arcade cabinet myself and I plan to do something similar to you but with two bluetooth controllers for players 3 + 4. I didn't dive into the matter deeply yet, however, but I'll try to answer your questions to the best of my current knowledge.
If the emulator and the game rom support four players, you should be able to configure the controls for players 3+4 just like those for players 1+2. Mind that some games come in different versions (roms). TMNT has 2 and 4 players versions, for example. Its 2 players versions have "2p" in their rom names.
At least Retroarch should recognize your controllers upon reconnection. A common advice is to connect them every time to the same usb ports in the same order. There's also a script by @meleu called joystick-selection that simplifies the mapping of controllers to players, and if the controllers should be recognized by their names or by the ports they're plugged into.
Alas, I don't know.
You're rightly terrified to lose your configuration if you didn't backup it recently. Do so now, because bad things happen all the time to "valuable" data, e.g. storage failure. (I put "valuable" in quotes because there is a saying that data with no backups isn't important by definition; if it were, it would have backups.)
Default behaviour is to assign controller number in the order of connection.
You could use the @meleu 's joystick-selection script (available from experimental packages) to map your controllers by core (ie snes pad for snes games).