Thanks for the detailed steps @meleu. I followed them and the controllers started working!
So, as a recap, here are the issues I faced and how they were resolved:
(outside of this forum post) I installed the latest retropie and found that it did not correctly generate entries in retroarch-joypads; that resulted in in-game keypads not working, and somehow the keyboard esc button being mapped to the left dpad. I resolved that by reimaging the retropie.
I installed the latest retropie and found that my 8bitdo controller wasn't doing anything in the games. I was able to get the dpad to work by restarting the rpi.
I found that the 8bitdo controller's dpad worked in games, and the Start button pressed A. Nothing else worked. I resolved this by disabling the 8bitdo hack menu option in retropie's settings.
This issue is now resolved thanks to @meleu's help.