Displaying button actions

Texacate

Hello, I'm hoping someone has done this before in RetroPie...

GOAL: For every game, I want to dynamically display what action each button on my control panel does.

EXAMPLE: Let's assume my button layout is
[ Y ][ X ][ L ]
[ B ][ A][ R ]

For Paman, the buttons do nothing, display something like this:
[ ][ ][ ]
[ ][ ][ ]
(but prettier of course)

For Galaga, Fire is mapped the the B button, display:
[ ][ ][ ]
[ Fire ][ ][ ]

For StreetFighter:
[ LightPunch ][ MedPunch ][ FiercePunch ]
[ LightKick ][ MedKick ][ FierceKick ]

This is mostly for arcade games, where controls the every game could be widely different. On consoles, there are a LOT fewer buttons. Not so worried about NES, SNES, GBA, etc. So for augment's sake, let limit this to arcade games.

I have spent a considerable amount of time with Google. I found many pieces of a solution (which I will list below for reference) but nothing that ties it all together.

I guess I should mention that, currently, I don't have any pre-conceived requirements on the hardware/software that does the displaying. OpenSource and less expensive is always better, but I'm open to ideas.

My control panel a two-player. 1 joystick and 6 buttons per player plus a track ball in-between.

I do have light-up buttons, and purchased a 16 channel LED driver from Ultimark. With that hardware, along with a software utility called Poor-man's LEDBlinky with RetroPie and Pac-Drive. I can at least illuminate only the buttons that are active, leaving the inactive buttons off. This was an awesome addition to the user-friendliness of my cabinet. Getting that configured the way I wanted, I learned a lot about Emulation Station's runcommand_onstart, and runcommand_onend scripting, and a learned bit of RUBY scripting too.

Now I want to take it to the next level. Here is what I've already found:

I found a database that will associate a rom name, with a list of button actions, like "Fire", "Jump", "Hard Punch", etc. This info is a critical component to an eventual solution. Controls.dat restructured JSON file. https://github.com/yo1dog/controls-dat-json.
Learned the driving two displays from a RPi is not a trivial thing to do. It can be done on a couple very small LCDs, by driving SPI protocol over the GPIO. But if we are talking anything much bigger than a 3" display, having separate Pi to drive the other display would be easier. That is "if" I can figgure out how to slave one RPi off the other.
Found cool image display utility, an if you give it the right arguments, you can be be connected via an SSH an make the remote RPi display an image with outer complexity of the X desktop environment. The utility is called FBI.
Found project where someone managed to display arcade marquee artwork on a second display. The blog was quite impressive. They did describe the hardware setup, but little was mentioned on the software drivers and communication.

Other thoughts:

I believe Retroarch has limitations on layering. Although editing the overlay image is a possibility, I can't just layer a second image on top of an overlay image file.
If the image that gets displayed is a bitmap (like .JPG or .PNG), I will need to find some sort of programmable image editor. And I will need so figure out how to script it, such that the text uses the info in my controls.json file.

mitu

@Texacate Someone had a similar idea recently - https://retropie.org.uk/forum/topic/21464/ - for showing the button layout on the start up image.

Texacate

Thank you so much for the pointer. I never considered using the startup image. Excellent idea. But the downside is: once the game starts, the map/legend is gone... Ideally I like it to be persistent. I need to read that thread carefully to see if they figured out how to automate the creation of the control map image.

mitu

@Texacate The topic has one suggestion for how to automate things - using imagemagickfrom the command line to programatically compose the image. Here's an idea:

have a master image with the layout
[ Y ][ X ][ L ]
[ B ][ A][ R ]

and a separate text inset that would explain the buttons -
Y = Fire
X = Turbo
etc.
But the inset would be empty (in this template image).

have one (transparent) image with each button selected/lit - i.e. for instance let's say `A' is used, have the same image as above, but just with

[ A ]
When generating the final image, combine the layout with the images containing the buttons and populate the text in the inset with the button names/actions.

BJRetro

@mitu seems like maybe for a typical 6 button arcade setup this page’s explanation for text positioning with gravity might work on an image because it would always place the text the same relative place on an image of the buttons: https://imagemagick.org/Usage/annotating/ . Very over my head on the coding aspect though.

hiulit

I could help with imagemagick.

But @Texacate has a point:

But the downside is: once the game starts, the map/legend is gone... Ideally I like it to be persistent

@mitu do you know if it's possible to alter EmulationStation or RetroArch to map button to show/hide the controller image?

mitu

@hiulit Emulationstation is pushed in the background once the game starts, so there's little use to modify it.
RetroArch can use some overlays (bezels) to display the button mappings and I see there's some settings like input_overlay_show_physical_inputs (dunno what it means) or input_overlay_next_btn. Ideally you could have an additional overlay (bezel) to show the button mapping and toggle between the normal (or empty) overlay to the button layout overlay using input_overlay_next_btn. I'm thinking at something similar discussed in this topic which handles shaders.

mitu

Aha, I found out what input_overlay_show_physical_inputs does. You can create an overlay (bezel) to show the controller buttons and you can map regions of the overlay to be toggled as you press the buttons on your controller. Here's an example for a NES overlay which can show buttons being pressed while you play.

hiulit

@mitu WOW, that's even cooler that showing a plane image! O_O

Now, how could the process of creating those bezels be automated?

hiulit

@mitu There's an arcade bezel https://github.com/libretro/common-overlays/blob/master/gamepads/arcade/arcade.cfg that could be used

mitu

@hiulit They don't look so hot

hiulit

@mitu No. they don't.... :( But then again, we could create a new bezel, right?

BJRetro

@hiulit I wonder why the name one is on top of the game whereas NES was outside of the frame.

Furluge

These would be amazing if they were being shown outside the screen in all that unused blanks space as opposed to on top of the gameplay.

hiulit

@BJRetro I don't understand. I see both bezel on top of the game.

hiulit

@Furluge What" unused blank space" are you referring to?

Furluge

@hiulit The sides of the screen when using a 16:9 monitor to display games meant for 4:3 aspect ratio. I'm saying I don't understand why the overlays are covering up the screen when they could be filling empty space you would usually see art / bezels fill up for a 16:9 monitor showing a 4:3 game. Displaying controller inputs seems like a pretty nifty usage of that space but all these overlays I'm seeing, even the metroid one, are putting everything over the display area, which seems silly to me. It looks like it was meant cell phone on screen controls to be honest. Actually that's probably exactly what these overlays were made for.

mitu

@Furluge The screenshots were done by me, by just loading the bezels/overlays without any parameters (i.e. screen size, viewports size). They can be configured to show outside of the game viewport, I just loaded them to test the button actions reflecting on the bezel changes.

Furluge

@mitu Ooohh, ok, I'll have to take a look at them then. Though they still do look suspiciously like on screen phone controls. :D

mitu

@Furluge I suppose they look like RA's controls when run on a mobile device - but they're overlays. I posted the link to them in the first screenshot post.