@rc_cola
Once when I experienced problems with cd tracks on psx games, I experimented with different audio drivers. If you dig more and search the web, you will find some different opinions. I think one of the audio driver will synchronize with video and if you get frame drops or anything else can't keep it up, then audio starts to stutter. Changing the driver did resolve this problem, if i remember right. The audio wasn't no longer synchronized, but it sounded without problems and I couldn't tell anything wrong with it. It was only happening with cd tracks. So, in short, this could be one of the tradeoffs. I have no more details about this, sorry.
Btw, you may find more information if you search about these topics in generell, like sdl vs alsa. These are common in Linux. And just in case you missed this page:
https://retropie.org.uk/docs/Sound-Issues/ Note, they speak about older Raspberry Pi versions on this page.