Ok. So it seems that order would get screwed up whenever I add the 2 gamepads controllers back in. I had to go set the controller order for each system I use to correct it. I had thought that leaving the systems unset would use the global settings, but....
Now I need to understand how to swap controllers around for specific games in N64 and megadrive, or if I just limit those to only the gamepads, since most need the analog sticks