Assuming your controller has enough buttons to support all of the N64 buttons, all of those buttons are available by default if you configured all of the buttons on your controller during the initial gamepad detection. If you need to reconfigure it I recommend resetting your controller configuration in the Retropie system menu->Retropie Setup so that it prompts you again.
I have a question,
could you manage to get a GPIO output to toggle from high to low or vice versa when something happens in the game? are there any methods to detect if gameover or highscore happens?
I ask because I have RGB mood lighting on my cabinet, and would love to make it flash and strobe etc if you die or get a highscore etc.
I understand that detecting an emulator starting and detecting what happens inside the emulator are two different things, but you dont get if you dont ask :)
Yeah good point, the FBA emulators also run recent MAME romsets, those are worth a try too.
You can also run the very latest lr-mame on RetroPie.
However, even the latest MAME doesn't have the 2D games right. If you truly want to play these games, fix their emulation in mainline MAME then consider backporting them to earlier stable MAME libretro cores.