You definitely need to configure the controls in the advmame.rc file for trackball/spinner controls to work correctly in AvanceMAME. Look for these entries and correct them as follows:
device_mouse raw
device_raw_mousedev[0] /dev/input/mouse0
device_raw_mousedev[1] /dev/input/mouse1
device_raw_mousedev[2] /dev/input/mouse2
device_raw_mousetype[0] ps2
device_raw_mousetype[1] ps2
device_raw_mousetype[2] ps2
You want "raw" input capture for the mouse, and you want them set to "ps2". Depending on how your setup detects the devices, they may come in as Mouse0, 1, 2 etc. So, just set all of them this way.
Then, further down into the .rc file, you specify which input is used by MAME like this:
input_map[p1_trackballx] mouse[0,x] mouse[1,x] mouse[2,x]
input_map[p1_trackbally] mouse[0,y] mouse[1,y] mouse[2,y]
input_map[p1_dialx] mouse[0,x]
input_map[p2_dialx] -mouse[0,y]
You see how I have mouse 0, 1, and 2 all mapped in the trackball x and y? That's because, if I had an external mouse plugged in, my trackball might shift over to 1 instead of 0. Later, if I unplug the external mouse, it shifts back to 0. By specifying all three possibilities (I also have spinners), it doesn't matter--all of the mouse inputs map as a trackball.
The dial inputs are what advancemame calls the spinners. For my setup, mouse 0 was consistently my spinners. So, Player 1 would get the x-axis and player 2 would get the y axis, although I noticed that the y axis was spinning backwards, so I added the minus in front to reverse it.
Your setup will be different I am sure, but you should be able to get the inputs working this way. Once configured in the .rc file, you can play with sensitivity settings using the in-game MAME GUI.
One final note--the polling rate isn't fast enough by default. The trackball and spinners have way more resolution than was being captured causing a good fast spin to register as a backspin until it slows down. That can be very frustrating. You can increase the polling rate to fix this. Add: usbhid.mousepoll=2 to your /boot/cmdline.txt.
Thank you for this! I have spent 3 days trying to get this spinner to work. It's rocking now!