@tim-s-up I broke down and re-wired all the controls. It turns out the kit I got (from Microcenter) had some bad advice about not worrying about the order they are wired, it said remapping in emulationstation would be fine.
But it DID matter that A button for joystick1 was wired the same as the A button for joystick2, etc. I had several things wired differently on each side. That meant when I mapped them independently in ES they would work, but ES would apply that mapping to the other side too.