It appears that in /opt/retropie/configs/n64/InputsAutoCfg.ini that under the relevant controller there was no entry for B button, and A button was incorrect. I edited this manually to correct for this, but I still get the same problem when i run the emulator - e.g., my attempted fix makes no difference to how it actually performs!