as far as i can tell, this behavior can be different depending on the BIOS. load up a neogeo rom for instance, and you will see a retroarch quickmenu >> options entry you can change to a few different button combinations (Holding Start being the default). then take 1942 for example in where holding start does nothing, and you must activate the service mode from retroarch quickmenu > options then restart the core.
also, there is a line in /opt/retropie/configs/all/retroarch-core-options.cfg
fba-diagnostic-input = "Hold Start"
that might be what you're after.
i'm not 100% certain on all this but i hope it helps point you in the right direction.