this is a (sort of) known issue.
specifically, MAME (not 2003 or any of the libretro cores, but MAME standalone) appears to be faster run via the desktop, than via the framebuffer. some info here: https://retropie.org.uk/forum/topic/28362/enhanced-performance-of-mamedev-mame-in-x11
i wonder if this could be fixed by configuring the launch script to launch MAME within X (on pi only)?
it has been on my list to investigate for years now, but yeah...