try mupen64plus-gliden64. it should be the best performance and compatibility in general, but be warned that it's quite normal for n64 to exhibit poor performance on pi4, but several games can perform pretty well.
I tried uncommenting and changing the hdmi_group and hdmi_mode options in the config.txt, but no audio improvement.....just poor video!
from what to what? i suggest a 1080p60 mode in your config.txt (please provide copies of all configs you edit.). any further lowering of resolution you need to do can be done via the runcommand, although personally i leave everything at 1080p.
please also provide /opt/retropie/configs/n64/mupen64plus.cfg