Create a custom ES system able to launch games for many systems

meleu

@namyzark I suggest you to use the @TMNTturtlguy 's trick here: https://retropie.org.uk/forum/topic/11499/change-es_systems-cfg-script

meleu

@hansolo77 and all RetroAchievements users, if you to create a custom system for RetroAchievements, the hascheevos tool can help you in this task. Here are the steps:

Create an achievements custom system as described in the OP.
Go to the hascheevos repo and follow the instructions to install it.
Use the trick to create a list of all ROMs that have cheevos in a directory. Here is an example for Mega Drive games:

./hascheevos.sh -u USER -p PASSWORD ~/RetroPie/roms/megadrive/* > ~/megadrive-roms-with-cheevos.txt

Get the add-game-to-custom-system.sh script (I talked about it in a post above):

wget https://raw.githubusercontent.com/meleu/share/master/add-game-to-custom-system.sh
chmod a+x add-game-to-custom-system.sh

Give the created list's content as arguments for the add-game-to-custom-system.sh:

cat ~/megadrive-cheevos.txt | xargs -d '\n' ./add-game-to-custom-system.sh -d ~/RetroPie/roms/achievements

[optional] Repeat the steps 3 and 4 for other supported systems.
Done!

Meanwhile we keep trying to make it less laborious (thanks to the big contributions from @pjft ).

hansolo77

EEEEEE!!!!!! Something to test! I'm now officially on vacation (from work, but staying home), so I have all week to play around with this.

hansolo77

Ugh migraines. Trying to figure out the PowerBlock problem I'm having and then to tackle this is just too much. I'm on vacation for crying out loud. Why'd I have to go and have a migraine? So if I understand correctly, I have to first create the "folder" achievements, then run the scrape tool to get a text list of my games. then use ANOTHER tool to convert those lists into symbolic links, and then go back and run yet another tool to add in the missing metadata? Seems like a lot of manual labor. Can't wait for some automation. :) Might have to try this tomorrow hen my head feels better.

namyzark

@meleu said in Create a custom ES system able to launch games for many systems:

https://retropie.org.uk/forum/topic/11499/change-es_systems-cfg-script

This was a great starting point @meleu. Thank you so much. Now to create custom folders for each user with only their games!!!

mattrixk

@hansolo77 said in Create a custom ES system able to launch games for many systems:

I'm now officially on vacation (from work, but staying home)

The best kind of vacation.

hansolo77

@meleu - I'm getting hung up at step 8 in the OP:

create the /home/pi/bin/runcustom.sh script (the code is pretty small):

The problem comes when I try to save the file because the /bin part of the path doesn't exist. Does this need to be in that path, or can I put it somewhere else and just make sure the es_systems knows where it is? It looks like it should reside with the runcommand.sh file, or maybe with the other ones (runcommand-onstart.sh and runcommand-onend.sh) located at /opt/retropie/configs/all.

Also, are you sure that ~/.emulationstation/es_systems.cfg is the correct location for that file? When I edited, that file didn't exit. On my system, that file is located at /etc/emulationstation/es_systems.cfg. Are you sure ES will pick it up?

EDIT - Decided to write the script in /home/pi/RetroPi and changed the es_systems.cfg to match. I then installed the hascheevos script. It's scraping MegaDrive right now. Looks like it's gonna take some time, but that's ok. The text file it creates is viewable through SFTP, so I checked it and it is grabbing full path names like it should! I wonder if a future version of this could maybe include some intelligence.. Have it scrape the entire /home/pi/RetroPie/roms path recursively and be like:

Skipping Atari 2600, no cheevos exist
Skipping Atari 5200, no cheevos exist
...etc

Then when it gets to a system it knows to have cheevos, it can go through and scan that folder. And then in the end, have it write out the text file including all systems rather that a separate file for each system.

EDIT - Another feature request.. maybe have a way to have it log the WARNING and ERROR games as well. If I have a game that has cheevos, but the script doesn't match the hash, I assume it's not going to work in RetroArch either. This is a bummer to me. I'd like to know what games need replaced with ones that DO work.

Also, perhaps another feature, or maybe part of the same. Have it list MISSING games. I say this because I know there are a lot of games that are HACKS that you simply don't get in a full ROM pack. To know there is a game available you don't have would be nice to see so you can go out and get it. It's related to the other request, so maybe just combining the two would help?

---> I think this belongs to a different thread that doesn't exit. Perhaps something like "HasCheevos Support" in the Ideas/Development?

EDIT - After the system was scraped, I installed the script to make symlinks to all the files in the /achievements directory. It was successful. Then I tried to do step #9 in the OP, and got error:

pi@retropie:~ $ ./custom-system-autoscraper.sh ~/RetroPie/roms/achievements/
ERROR: you must give a directory as argument.
Usage:
./custom-system-autoscraper.sh /path/to/custom/system/directory

I got this with /home/pi/RetroPie/roms/achievements/ as well... So I decided, meh, no metadata for now. Lets test it. Restarted ES.. and ES craps out. Says my systems file is missing or corrupt, and I'll have to manually recreate it. Then ES crashes back to terminal displaying:

lvl0:     es_systems.cfg is missing the <systemList> tag!
lvl0:     Error while parsing systems configuration file!

Wonderful. :( I suspect it's because of the custom system I added to that random folder somewhere... gonna test moving it into the actual existing es_systems.cfg file that works.

EDIT - More success, more set backs. I erased the file ~/.emulationstation/es_systems.cfg and added the contents of that to /etc/emulationstation/es_systems.cfg. That worked. When I rebooted, there is now an "achievements" system on my home screen. However, there is no working artwork for it (yet) so the screen is all messed up. No biggie for now. I tried to load up a game, and all it does is zoom in, go to black, and zoom back out. My SSH terminal shows this:

sh: 1: /home/pi/runcustom.sh: Permission denied
INFO: Failed to set keyboard mode

~~Do I need to set ownership or chmod or something to that file?~~ Bingo! Did a quick and dirty chmod a+x runcustom.sh (saw you did that in another step) and it worked! I was able to get the game to load up just fine. :) Now I just need to get a proper background and system svg from @TMNTturtlguy and then figure out why your script for metadata doesn't work. I also need to scrap the other systems too. BTW.. I tried that whole export Path blah blah stuff and it didn't do anything.

meleu

@hansolo77 hey dude, you are being a bit vindictive, aren't you? :-) I confuse you with all those scripts and commands and you confuse me with all those questions and edits in a same post. :D

Let me try:

when I say "create the /home/pi/bin/runcustom.sh script", the creation of the bin directory if it doesn't exist is implicit in that sentence.
"Also, are you sure that ~/.emulationstation/es_systems.cfg is the correct location for that file?"
Yes, I am. If you edit /etc/emulationstation/es_systems.cfg any change you made will be lost after an EmulationStation update.
the feature requests for the hascheevos tool can be made on this thread: https://retropie.org.uk/forum/topic/11859/what-about-adding-a-cheevos-flag-in-gamelist-xml
The error you're having ERROR: you must give a directory as argument. is because you are using the wrong script (sorry, I need to make it more clear in the OP). I've created a better script, which takes the path for a file and automatically creates the symbolic links and copy the original metadata. I talked about it in this post
I didn't understand this part of your post "I tried that whole export Path blah blah stuff and it didn't do anything".

hansolo77

LOL I'm going to bed now before my head explodes.

1 - Does it NEED to be in that folder at all though? Is it safe where I have it in /home/pi/?

2 - Does ES look in the ~/.emulationstation/es_systems.cfgfirst? If so, I'll just make a copy of the file to that location and edit from there. The error I got about that was because it was missing all the other systems, and thus the extra parts at the beginning.

3 - When I get up in the morning, I'll transfer over my requests for you there. :)

4 - I didn't really read through your thread here. Only to the point where you tagged my name and then the OP. So I will check back and see if you have a clearer update to the correct script.

5 - https://github.com/meleu/hascheevos#create-a-list-of-all-roms-that-have-cheevos-in-a-directory (step #4)

# adapt the path below to your setup!
export PATH="$PATH:/path/to/hascheevos/bin"

I tried to map it to /home/pi/hascheevos/bin, /home/pi/hascheevos/bin/hascheevos.sh, ~/hascheevos/bin/ etc.. none of them work. I read up on the export command. It's for making that path appear as though I'm always in that path, so I can run the command without having to type out the entire path to it. Well.. I can verify with the env command that it did add those paths, but they don't work. When I try to run the script locally without being in that folder, it just tells me it's not a valid file or something.

Anyway, yeah.. it's almost 3am here. I haven't been up this late in years lol. I need to rest my noggin. :)

meleu

@hansolo77 said in Create a custom ES system able to launch games for many systems:

1 - Does it NEED to be in that folder at all though? Is it safe where I have it in /home/pi/?

No. Yes, just change es_systems.cfg accordingly.

2 - Does ES look in the ~/.emulationstation/es_systems.cfgfirst?

Yes.

5 - https://github.com/meleu/hascheevos#create-a-list-of-all-roms-that-have-cheevos-in-a-directory (step #4)
# adapt the path below to your setup!
export PATH="$PATH:/path/to/hascheevos/bin"
I tried to map it to /home/pi/hascheevos/bin, /home/pi/hascheevos/bin/hascheevos.sh, ~/hascheevos/bin/ etc.. none of them work.

Oh, I got it. I would need to know what exactly you typed to help with this. But this is subject for that other thread. Feel free to ask there, but please give more details of what you typed and error you got.

Have a good night Han!

cyperghost

@meleu
Just logged in for small comment. You are awesome! I am still in holiday and enjoying it.

meleu

@cyperghost thanks for the kind words bro. Enjoy your time!

hansolo77

Yesterday I realized I had a lot of my modded roms (that I got for cheevos) only in my "Achievements" sub folders. So I moved those back into their root system folders with the rest of their rom brothers. That then prompted me to have to re-scrape all my roms so that when I migrate them over to the new custom system, they'll all still have proper metadata. Took me all day yesterday, then I realized I didn't have the naming scheme set right (the last scrape I did was for MAME so I had changed it). So today I'm re-scraping again, but this time I made sure to have the name scheme set right! Oh well, I'll get there eventually.

meleu

@hansolo77 what do you mean by "naming scheme set right"? What would be the right naming scheme?

hansolo77

@meleu I was using USX and had it configured to provide just the Game Name in the gamelist.xml. Since I'm using a no-intro romset, the resulting gamelist had a bunch of duplicate game names. This is a bother to me, because if there are multiple versions of games, I want to know which one I'm loading up. So now I'm scraping using the scheme "filename", so it will provide me with the extra bits of info at the end (like USA, Japan, v1.0, v1.2, etc.

TMNTturtlguy

@hansolo77 why did you have to rescrape? Just because you moved the roms to a different folder? You could just use notepad++ and do a replace all to all the files in the folder. Takes about 10 seconds.

hansolo77

There's more to it than that though. All my achievement artwork was saved with the folder name as part of the filename. So they all looked like _ACHIEVEMENTS_Sonic 1 (etc).mp4. It's all good. I'm in the process of making the symlinks now. Fingers crossed!

I also create a system logo for Achievements. I was going to upload it the ComicBook theme page, but everything says the file is corrupted or something. I even tried to copy it back over from the Pi and still no good. If it's corrupted, it shouldn't work with the Pi either right? I dunno. It works, that's all that matters.

hansolo77

I think I got it working! Took forever adding the last couple of systems, probably due to memory or something. All told, I now have a custom system named "Achievements", and there are currently 1426 games to choose from! :) I need to do some further checking and testing, but so far all the games look like they have proper metadata installed.

Next on my list - figure out how to get the ComicBook loading screens to install.

Thanks @meleu !!!

meleu

@hansolo77 said in Create a custom ES system able to launch games for many systems:

figure out how to get the ComicBook loading screens to install.

Didn't rpie-art work for you?

hansolo77

I never progressed any further with it because I can't get it to load up in the Retropie menu. I'm tackling this today.