@darom
Please say which controller, emulator and game you are using.
I had a similar problem. Try other emulators (Atari, NES, etc) and see if those work. If your controller/s also works in EmulationStation (the menu that pops up when u first activate retropie), then the problem is that the controller isn't configured right in-game or for the specific n64 emulator.
If your n64 emulator starts with the letters lr, you can access the retroarch menu to change the controller settings in-game. You can do this by plugging in a keyboard to ur RPI4B and using Hotkey + X (in-game) to open the retroarch menu and change your controls. Play around with changing them till something works.
Here's the link about retorarch on the docs: https://retropie.org.uk/docs/RetroArch-Core-Options/
Video I also used:
Also make sure to check this out and include all the info in it: https://retropie.org.uk/forum/topic/3/read-this-first
Hope you have fun. I really recommend trying Rogue Squadron on n64 and Star Wars racer (for 2 persons). Relatively hard games, but worth it.