@deltax5 you can build RetroPie on top of Linux. Check out the official documents for more info on doing that. I've got a little laptop running Ubuntu and it's pretty good. If you put RetroPie on top of that I think you'll be onto a winner. Pi3 is still not great for N64 emulation.
Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first