I know this is SUPER old, but after parsing through possible ways/threads to get a trackball working on MAME2003, I was finally successful (after 3 attempts where I gave up and just went with advancemame)
A) in retroarch-core-options.cfg, add:
mame2003-dialsharexy = "enabled"
mame2003-mame_current_mouse_enable = "enabled"
I didnt verify if you actually need mame2003-dialsharexy = "enabled" but Im glad it is working
B) in /opt/retropie/configs/arcade/retroarch.cfg, add:
input_player1_mouse_index = "MOUSEINDEX"
Where:
MOUSEINDEX is the index for your mouse /dev/input/mouse*. I needed to test through a couple as what I needed to enter above, wasnt the same as the device's index. Unclear how these correlate
Your retroarch config (in this case arcade) may be different for your emulator. Easiest way I know to determine the config is when its running
ps -aux | grep retro and look for the program and its referenced config file