hey @dask, i spent a little time with this and managed to figure out a few things..
expanding on your idea of using the overlay/bezel system, it's possible to (using a hotkey combo) swap a transparent overlay with a control image overlay. this can be done on a per-system basis using the retroarch.cfg files located in /opt/retropie/configs/<system>
retroarch.cfg:
overlay_directory = "/your/path/to/overlays"
input_overlay_enable = true
input_overlay_hide_in_menu = true
input_overlay = "/your/path/to/overlays/mame-controls.cfg"
input_overlay_opacity = 0.9
input_overlay_scale = 1.0
input_overlay_next_btn = "6"
i think you can substitute the last line with input_overlay_next = "<keyboard key>" and input_overlay_next_axis = "<joystick axis>" to adapt this to your hardware scheme.
next, create the two images needed: transparent.png, and mame-controls.png
and here are the contents of mame-controls.cfg:
overlays = 2
overlay0_overlay = transparent.png
overlay1_overlay = mame-controls.png
overlay1_name = "mame-controls"
overlay0_full_screen = true
overlay1_full_screen = true
overlay0_descs = 1
overlay0_desc0 = "overlay_next,400,460,rect,40,20"
overlay0_desc0_next_target = "mame-controls"
overlay1_descs = 0
there probably already exists a much better way of doing it, so this is just a proof of concept.
either way, hope it's enough to help you get started.