@Spartacus as the fellows told you before me, the main problem on loading the N64 games could be the extension. If the files are zipped, unzip them and throw those files into the proper folder. The N64 extension should .n64 .v64 or .z64. Most of the games should work properly depending on the configuration and your raspberry model. I always had problems with Killer Instinct since I'm not able to find a good file for that game. But for the others, I had no problem at all.
Concerning the MAME incident, as @herb_fargus stated, the problem should reside on the romset you have. MAME has many different versions and a single game could work fine in one oh those versions but could not work in other. I don't remember but I guess most of the working games are in the romset 1.38
You can check it.
