So I finally figured this out. Turns out I needed to add 'evdev' as a module in /etc/modules, and then everything is hunky dory.
Here's where I finally found my solution: https://github.com/adafruit/Adafruit-Retrogame/issues/21
Hopefully someone else will find this useful.