@burgoyne Did you ever get any resolution to this issue? I had the same thing happen to me and I am also looking for a solution to use the same gamepad.

I found that I've written a response on the same subject about four months ago, so it's probably easiest if I just refer back to that entry here on the forum:

https://retropie.org.uk/forum/topic/126/n64-controller-analogue-stick-issues/6

[edit]

You may wish to use jstest to aid obtaining and setting the calibration values. If it's not already present on the system:
sudo apt-get install joystick

For more information, please see:
http://mpolaczyk.pl/raspberry-pi-series-reverse-engineering-of-joystick-usb-communication-protocol/

jstest returns the full description of the device; this matches that displayed by emulators in RetroPie (in text at the bottom of the screen as an emulator loads; within the RetroArch Input menu; within Hatari's Joystick menu, etc)

for example:

jstest /dev/input/js0

Joystick (WiseGroup.,Ltd PS/SS/DC Joypad to USB BOX)

jstest /dev/input/js1

Joystick (HuiJia PS/SS/N64 Joypad to USB BOX)