SUCCESS - it looks like I've fixed it. The problem was that I hadn't commented out the unused buttons in the "Input device" buttons section of retroarch.cfg. Before I updated RetroArch didn't care, now it obviously does.
So I rolled back, updated and then only made that change to the retroarch.cfg. Everything else still works as before, because I didn't need to reset EmulationStation or MAME cfg files.
In the end it was a simple fix, thanks to everyone for all their feedback which helped me find the solution. Hopefully this thread helps someone else down the track.
Sorry for major bump, but just wanted to say that despite several months between posts, I finally managed to update my install completely to version 4.6.9 and I would say yes it does seem to have improved a bit. I reduced my N64 stuff and completely removed Dreamcast though, as lets be honest you aren't going to get much joy on that front on a Pi3B+. But the couple of dozen N64 games I do have seem to run a lot better than they did on previous installs with only a small margin of tweaking and my usual slight GPU overclock, and I notice the standalone GlideN64 is definitely improved. Very pleased, and worth it for pi3B+ diehards imho.
For what it's worth I had actually stumbled on this setting when messing around but I set the non-windowed value (because it's full screen... I assumed that meant it wasn't windowed) and when that didn't fix it I moved on.