Hi, this is an old issue, but I just configured mine RetroPie few days ago.
I don't want overclock my device, this can means a risk for me. So, I was playing Tony Hawks Pro Skater, and I think the same issue was happening.
The solution for me was reduce the video quality (also the video clock) and skip the sound of music while playing (the music I needed to change on Game configuration trying to skip the Audio system calls).
As software developer I know some error happens on the game and emulator prints a lot's of messages behind the game. I think if I know the shell commend to make the N64 Retro Pie Emulator skip the log messages the game can run even better.
If anyone knows a command like this please tell me.
Thank you