hascheevos.sh - a tool for RetroPie/RetroAchievements users

meleu

@hansolo77 said in hascheevos.sh - a tool for RetroPie/RetroAchievements users:

Do we still need to pass the RetroAchievments username and password in the execution, or write it inside the hascheevos.sh script?

No need to edit the script at all.

You only have to provide username/password under some conditions. The script will complain if it needs the account data and they aren't provided
By the way, the README was updated accordingly.

hansolo77

I finished through a "scrape" of my rom system and had it generate that "all-in-one" collection. I restarted ES and enabled it in the Custom Collections menu. It seems to have worked partially. My list using the old method (scraping each system manually, then running the script to create a custom collection from it) generated a list of 1426 games when I did it a month or so ago. Using the new method, it only grabbed a total of 969 games. At first I thought at "well, some games probably got deleted from the RA server", but that's a large number to be missing. So I went back and checked a few games. RA's homepage shows that Friday the 13th (NES) is the current "Game of the Week". It exists on my old list but not in the new one. Something is screwy there. Also, when I go through the NEW list, a lot of the video snapshots aren't working and just show the JPG art instead, whereas the same game in the OLD list shows a video.

I will experiment more and give specific games tomorrow when I have the chance. It's like ALMOST there. :)

EDIT : I think the problem with the game list shortage is that ES is providing the list with only 1 interration of a game name. For instance, when I look at the contents of the cfg file it created, I have multiple entries for Final Fantasy II. I have the original NES version, a Translated NES version, an NES USA Prototype, and the SNES version. But ES only displays the game 1 time, even through the scraper found and added all 4 to the file. This probably happens many times throughout the gamelist and probably whereever there are multiple versions of the same game.

EDIT 2 : I need to do some more work it seems. I went into my ES settings and disabled the option "Parse Gamelists Only", then restarted ES. The NEW game list is now showing 1456 games. So now that's working. But now I have another problem. None of my GB and NES games are showing any metadata. Not sure what's going on there, and I'm way to tired now to mess with it anymore tonight. But I think I might have to rescrape my metadata. It looks like all your script does is check the hash and build the necessary list of games and their paths. So the changes @pjft did to get custom collections working is to have it parse the cfg with direct paths to them. If ES isn't displaying their metadata, it's missing altogether. I confirmed this by checking the regular NES gamelist and it's gone there too. A perfect system is so close though. :) That is, until the next big thing comes along that I'll want to add too.

pjft

@hansolo77 can you check the number of lines in the generated file, and/or send it over via pastebin?

I'm also not sure why it would have changed anything in terms of video vs image view. That should not be happening altogether, and if possible I'd like to see a video of it - one with the game in the original collection, with the video showing, and open the edit metadata menu for it, then exit it, and then one in the achievements collection, without the video playing, with opening the edit metadata menu as well.

You can do it with a phone, I just want to see if it's really going for the same file in ES.

Thanks.

hansolo77

@pjft I think the problem might be a corrupting SD card. This is the same card I've been using ever since I started my project. That includes all of my original build work, then the adding of video snaps, updated metadata, then the white screen fix, then the Mausberry Shutdown Switch fix, then the Achievements and Custom Collections. The testing of the shutdown scripts for metadata saving was intensive, and a lot of the time my system hung or crashed, resulting in my "pulling the plug" because the shutdown wasn't working. To that end, I think parts of my SD card may have been damaged. Prior to testing out this new cheevo script, I went to shut down my system to make a backup, just like I do any time I go to change things. Well, there was a screensaver running, I pushed the power button and waited. Nothing happened. So after a few minutes I just pulled the plug. When I connected the SD card to my computer for backing up, it offered me the option to FIX it (corrupted data). I chose not too because in my experience, Windows can't really fix a Linux partition with chkdsk. When I booted everything back up the first time, ES was reporting (in the terminal window after exiting) that it was unable to parse the NES game list. When I checked the file via SFTP, it had created a blank gamelist.xml and renamed original as gamelist.old. No really knowing what was going on, I just erased the blank one, and renamed the old one back to the correct way. Rebooted, ES was happy, NES gamelist was working correctly, etc. I then scraped with the cheevos script. Then, following instructions, I rebooted. The cheevo custom collection was there, but none of the NES games had working videos or art (metadata). Everything else was fine, and I didn't think twice about it. Then I noticed the shortage of games compared to the old cheevo list. That's when I tried turning off the "only parse game lists" switch in ES. Once I did that and restarted ES, the games all displayed correctly, although the metadata is still now showing. I think the root cause of the problem is the corruption from before, and possibly the original NES gamelist is bad. So I'm in the process now of re-scraping a new NES gamelist and will see if the results are better afterwards.

Long story short - I'm re-scraping a new gamelist because it got f*ed up. :)

robertybob

Apologies, I haven't logged in for a couple of weeks, I will try my best to check this out :)

hansolo77

@meleu It's been about a month, and I decided to do a rescrape with the hascheevos script. I first ran with the --update flag to get the latest version. Then started the full scrape to collections. It went through just fine. But when it completed, I was presented with the same list (same number of games) I had before. I thought maybe it just needed to refresh the list so I rebooted the Pi. Still nothing new. Surely there have been more games added in the last month. Does the script not overwrite the previously existing collection file? Do I need to run this outside of ES (I was actually playing a game during the entire scrape). Have you noticed anything like this in your tests?

meleu

@hansolo77 I suspect you're seeing your own custom collection lists and not the new ones generated by the script.

Please, try to go to main menu -> game collections settings -> custom game collections and see if there are new collections named ACHIEVEMENTS {SYSTEM_NAME}.

hansolo77

When I look at my available Custom Collections, it only lists the 1 ACHIEVEMENTS (remember, I'm using the single collection list, not split). When I SFTP into the Pi, I have a file named custom-achievements.cfg and custom-achievements.cfg.old although I think the .old was one I had made before using your updated script. The date on the new one is correct though (11/132017). So it did build a new one. There must not be any new games... Or does your Github repository of the hashes need updated? Watching the scrape, it looks like the first thing it does is download the hashes locally to my Pi, it that correct?

meleu

@hansolo77 please, execute the command below:

bash -x ./hascheevos.sh --collection "~/RetroPie/roms/nes/CONTRA ROM FILE NAME.nes" 2> debug.txt

It'll generate a file named debug.txt. Please post its content in codepad.org.

hansolo77

@meleu Ok, might be an hour or so. I went ahead and erased my collection from the folder and started a re-scrape. I'll post a link to it when it finishes. Or can I do a simultaneous run if I open another ssh window?

meleu

@hansolo77 said in hascheevos.sh - a tool for RetroPie/RetroAchievements users:

Or can I do a simultaneous run if I open another ssh window?

yep, you can do simutaneously. Remember, the command I suggested above is "scraping cheevos" only for one ROM (and I suggested the Contra game from NES).

I'm not sure if I'm going to be able to debug it right now, though.

hansolo77

@meleu Not sure if this helps. Did you want me to put it exactly like you had it, or did you want me to fill in a rom name that has Contra in it?
http://codepad.org/bkUDYtEU

EDIT - debug with Contra (USA).nes filename just incase:
http://codepad.org/D0NLnjD3

meleu

@hansolo77 hehe. I meant your actual contra rom file. But my command is wrong any way. Please, repeat that command but replacing ~with /home/pi.

hansolo77

@meleu Ok, so

bash -x ./hascheevos.sh --collection "/home/pi/RetroPie/roms/nes/Contra (USA).nes" 2> debug.txt

results in:
http://codepad.org/nazUHpn0

meleu

@hansolo77 looks like everything ran fine

This game has been added to "/home/pi/.emulationstation/collections/custom-achievements nes.cfg".

Do you have that 👆 file on your system?

hansolo77

@meleu I do now, but I didn't before.
Incidentally, I noticed some interesting results while scraping in the log:

Checking "/home/pi/RetroPie/roms/mastersystem/Super Tennis (USA, Europe).sms"...
--- Genesis: ce4b4ca78c681bccb77c2d383c6a1243
--- plain MD5: 2db9404fe79593fd2379921ca822103a
--- unable to get game ID.
"/home/pi/RetroPie/roms/mastersystem/Super Tennis (USA, Europe).sms" has no cheevos. :(

Why would it say "Genesis:" if it's Mastery System file? And this game was also unable to detect a "game ID". My scroll-back log only goes as far back to still be SMS, yet it looks like it's doing that "Genesis" bit on all of the games. Here's a working result:

Checking "/home/pi/RetroPie/roms/mastersystem/Strider (USA, Europe).sms"...
--- Genesis: 2bc1f448529ee58162da600d966d9a14
--- plain MD5: 48af5ae2cad7c48c81d7c2553a8ed426
--- game ID: 250
--- Game Title: "Strider"
--- This game has been added to "/home/pi/.emulationstation/collections/custom-achievements.cfg".
--- "/home/pi/RetroPie/roms/mastersystem/Strider (USA, Europe).sms" HAS CHEEVOS!

I also noticed some indications of needing a username/password for some SNES games, but it went by too quickly to be able to copy/paste it. The script never prompted for a login, just went on to the next one.

cubed3d

Just a quick question -
Why would hascheevos report this:

pi@RetroPie:~ $ sudo ~/hascheevos/bin/hascheevos.sh --collection --system all
Checking JSON hash libraries...
Done!
ERROR: not a RetroPie system.
The "--collection" option is available only for RetroPie systems.

Obviously running RetroPie on a RPi3 - Roms located on a mounted network share. Everything else working perfectly, except for hascheevos :/

Edit Just tried running it on a single rom as a test and got the following:

pi@RetroPie:~ $ sudo ~/hascheevos/bin/hascheevos.sh "/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc"
Checking JSON hash libraries...
Done!
Checking "/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc"...
cheevos_eval_md5(): Value too large for defined data type
"/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc" has no cheevos. :(

Could this be to do with the ROMs being on a network share? Thanks in advance.

meleu

@cubed3d that problem happened because of the sudo. There's no need to use the script this way. Calling the script with sudo is confusing it about the homedir. I'll try to make the script smart enough to deal with it ;-)

cubed3d

@meleu Thanks for the quick reply!

Had a try without the sudo as suggested, a bit better progress:

pi@RetroPie:~ $ ~/hascheevos/bin/hascheevos.sh --collection --system all        Checking JSON hash libraries...
Done!
WARNING: ignoring "pcengine": not found.
WARNING: ignoring "segacd": not found.
WARNING: ignoring "xbox360": not found.
WARNING: ignoring "atari": not found.
Checking "/home/pi/RetroPie/roms/megadrive/3 Ninjas Kick Back (USA).md"...
cheevos_eval_md5(): Value too large for defined data type
"/home/pi/RetroPie/roms/megadrive/3 Ninjas Kick Back (USA).md" has no cheevos. :(

Checking "/home/pi/RetroPie/roms/megadrive/6-Pak (USA).md"...
cheevos_eval_md5(): Value too large for defined data type
"/home/pi/RetroPie/roms/megadrive/6-Pak (USA).md" has no cheevos. :(

For each and every ROM, cancelled it shortly after.
Seems its having trouble determining the MD5 hash of each ROM file now, I will try with locally stored ROMs and report back!

cubed3d

Unmounted network share and tried locally:

pi@RetroPie:~/RetroPie/roms/snes $ ~/hascheevos/bin/hascheevos.sh "/home/pi/RetroPie/roms/snes/Super Mario Kart (Europe).zip"
Checking JSON hash libraries...
Done!
Checking "/home/pi/RetroPie/roms/snes/Super Mario Kart (Europe).zip"...
--- SNES: d504d075130c4a544814eb21e691f904
--- game ID: 232
--- Game Title: "Super Mario Kart"
--- "/home/pi/RetroPie/roms/snes/Super Mario Kart (Europe).zip" HAS CHEEVOS!

Local ROMs working. Now I'm thinking it may have something to do with write permissions? Is there a more verbose output to debug this?