I am using 2 iBuffalo SNES controllers plus an arcade stick too, which are connected all the time through usb. And I don't have this problem, just saying it. What I don't understand, why do you use xboxdrv? These gamepads are working out of the box.
I also use RetroPie joystick-selection script, which lets you set the gamepads as player 1, 2, 3, 4 in what order you want. After restart, this selection remains. Now, when I think back, I installed this because I had the problem after restart the order wasn't fixed without this script. It was randomly changed. This script gives fixed order, either system wide or even different settings for each emulator.
Not sure if I did understand your problem and if you already use this script.