Self-answering: the problem with D64 files turned out to be a dodgy disk. So what you need to do to play C16/Plus 4 games in MESS is add the following lines to EMULATORS.CFG:
advmessc16prg = "/opt/retropie/emulators/advmame/bin/advmess c16 -log -cfg /opt/retropie/configs/c16/advmess.rc -quickload %ROM%"
advmessc16disk = "/opt/retropie/emulators/advmame/bin/advmess c16 -log -cfg /opt/retropie/configs/c16/advmess.rc -floppydisk %ROM%"
advmessplus4prg = "/opt/retropie/emulators/advmame/bin/advmess plus4 -log -cfg /opt/retropie/configs/c16/advmess.rc -quickload %ROM%"
advmessplus4disk = "/opt/retropie/emulators/advmame/bin/advmess plus4 -log -cfg /opt/retropie/configs/c16/advmess.rc -floppydisk %ROM%"
Then when you boot, choose the version you want from the RUNCOMMAND menu, eg if it's a C16 PRG file choose "advmessc16prg".
Once a PRG file is loaded, just type RUN. (I find if you do it too soon you have to do it twice, so wait a couple of seconds.)
If it's a D64 file, type LOAD"",8,1, then RUN when it's done.
AdvMESS runs the games much faster and smoother than VICE, but its compatibility is quite poor, so horses for courses.