I would also add that you can usually get the order you want by the order of the ports you plug them into (so long as you don't use a USB hub). With the 4 ports on my Pi3b, I plug them in this order:
So, my pandora's box takes up 0 and 1, keyboard in 2, mouse in 3. if I add another controller, I replace the mouse or keyboard. My pandora's box always stays player 1 and 2 that way.
I use a usb hub for mouse/keyboard/sony and xbox controller.
Pandora is direct to Pi4.
could that be the issue ?