would you be able to provide the info in the following: https://retropie.org.uk/forum/topic/3/read-this-first
I think some of it is in your first post, but I'm wondering what emulator is being used for N64 in particular.
lr-Mupen64plus uses retroarch configs and should work okay, but it isn't particularly the default emulator
Mupen64plus used it's own configs. It is supposed to do an auto-config, but that doesn't always work. You can go into the configs and set it manually.
I tend to use lr-Mupen64plus on systems that don't auto-config right, though it may not necessarily be the best performance. N64 is buggy enough that the extra performance difference isn't worth it to me.