hascheevos.sh - a tool for RetroPie/RetroAchievements users

pjft · 20 Oct 2017, 06:31

@hansolo77 can you check the number of lines in the generated file, and/or send it over via pastebin?

I'm also not sure why it would have changed anything in terms of video vs image view. That should not be happening altogether, and if possible I'd like to see a video of it - one with the game in the original collection, with the video showing, and open the edit metadata menu for it, then exit it, and then one in the achievements collection, without the video playing, with opening the edit metadata menu as well.

You can do it with a phone, I just want to see if it's really going for the same file in ES.

Thanks.

hansolo77 · 20 Oct 2017, 23:20

@pjft I think the problem might be a corrupting SD card. This is the same card I've been using ever since I started my project. That includes all of my original build work, then the adding of video snaps, updated metadata, then the white screen fix, then the Mausberry Shutdown Switch fix, then the Achievements and Custom Collections. The testing of the shutdown scripts for metadata saving was intensive, and a lot of the time my system hung or crashed, resulting in my "pulling the plug" because the shutdown wasn't working. To that end, I think parts of my SD card may have been damaged. Prior to testing out this new cheevo script, I went to shut down my system to make a backup, just like I do any time I go to change things. Well, there was a screensaver running, I pushed the power button and waited. Nothing happened. So after a few minutes I just pulled the plug. When I connected the SD card to my computer for backing up, it offered me the option to FIX it (corrupted data). I chose not too because in my experience, Windows can't really fix a Linux partition with chkdsk. When I booted everything back up the first time, ES was reporting (in the terminal window after exiting) that it was unable to parse the NES game list. When I checked the file via SFTP, it had created a blank gamelist.xml and renamed original as gamelist.old. No really knowing what was going on, I just erased the blank one, and renamed the old one back to the correct way. Rebooted, ES was happy, NES gamelist was working correctly, etc. I then scraped with the cheevos script. Then, following instructions, I rebooted. The cheevo custom collection was there, but none of the NES games had working videos or art (metadata). Everything else was fine, and I didn't think twice about it. Then I noticed the shortage of games compared to the old cheevo list. That's when I tried turning off the "only parse game lists" switch in ES. Once I did that and restarted ES, the games all displayed correctly, although the metadata is still now showing. I think the root cause of the problem is the corruption from before, and possibly the original NES gamelist is bad. So I'm in the process now of re-scraping a new NES gamelist and will see if the results are better afterwards.

Long story short - I'm re-scraping a new gamelist because it got f*ed up. :)

robertybob · 21 Oct 2017, 20:38

Apologies, I haven't logged in for a couple of weeks, I will try my best to check this out :)

hansolo77 · 14 Nov 2017, 17:24

@meleu It's been about a month, and I decided to do a rescrape with the hascheevos script. I first ran with the --update flag to get the latest version. Then started the full scrape to collections. It went through just fine. But when it completed, I was presented with the same list (same number of games) I had before. I thought maybe it just needed to refresh the list so I rebooted the Pi. Still nothing new. Surely there have been more games added in the last month. Does the script not overwrite the previously existing collection file? Do I need to run this outside of ES (I was actually playing a game during the entire scrape). Have you noticed anything like this in your tests?

meleu · 14 Nov 2017, 17:56

@hansolo77 I suspect you're seeing your own custom collection lists and not the new ones generated by the script.

Please, try to go to main menu -> game collections settings -> custom game collections and see if there are new collections named ACHIEVEMENTS {SYSTEM_NAME}.

hansolo77 · 14 Nov 2017, 18:30

When I look at my available Custom Collections, it only lists the 1 ACHIEVEMENTS (remember, I'm using the single collection list, not split). When I SFTP into the Pi, I have a file named custom-achievements.cfg and custom-achievements.cfg.old although I think the .old was one I had made before using your updated script. The date on the new one is correct though (11/132017). So it did build a new one. There must not be any new games... Or does your Github repository of the hashes need updated? Watching the scrape, it looks like the first thing it does is download the hashes locally to my Pi, it that correct?

meleu · 14 Nov 2017, 20:06

@hansolo77 please, execute the command below:

bash -x ./hascheevos.sh --collection "~/RetroPie/roms/nes/CONTRA ROM FILE NAME.nes" 2> debug.txt

It'll generate a file named debug.txt. Please post its content in codepad.org.

hansolo77 · 14 Nov 2017, 20:17

@meleu Ok, might be an hour or so. I went ahead and erased my collection from the folder and started a re-scrape. I'll post a link to it when it finishes. Or can I do a simultaneous run if I open another ssh window?

meleu · 14 Nov 2017, 20:22

@hansolo77 said in hascheevos.sh - a tool for RetroPie/RetroAchievements users:

Or can I do a simultaneous run if I open another ssh window?

yep, you can do simutaneously. Remember, the command I suggested above is "scraping cheevos" only for one ROM (and I suggested the Contra game from NES).

I'm not sure if I'm going to be able to debug it right now, though.

hansolo77 · 14 Nov 2017, 20:40

@meleu Not sure if this helps. Did you want me to put it exactly like you had it, or did you want me to fill in a rom name that has Contra in it?
http://codepad.org/bkUDYtEU

EDIT - debug with Contra (USA).nes filename just incase:
http://codepad.org/D0NLnjD3

meleu · 14 Nov 2017, 21:09

@hansolo77 hehe. I meant your actual contra rom file. But my command is wrong any way. Please, repeat that command but replacing ~with /home/pi.

hansolo77 · 14 Nov 2017, 21:18

@meleu Ok, so

bash -x ./hascheevos.sh --collection "/home/pi/RetroPie/roms/nes/Contra (USA).nes" 2> debug.txt

results in:
http://codepad.org/nazUHpn0

meleu · 14 Nov 2017, 21:23

@hansolo77 looks like everything ran fine

This game has been added to "/home/pi/.emulationstation/collections/custom-achievements nes.cfg".

Do you have that 👆 file on your system?

hansolo77 · 14 Nov 2017, 21:34

@meleu I do now, but I didn't before.
Incidentally, I noticed some interesting results while scraping in the log:

Checking "/home/pi/RetroPie/roms/mastersystem/Super Tennis (USA, Europe).sms"...
--- Genesis: ce4b4ca78c681bccb77c2d383c6a1243
--- plain MD5: 2db9404fe79593fd2379921ca822103a
--- unable to get game ID.
"/home/pi/RetroPie/roms/mastersystem/Super Tennis (USA, Europe).sms" has no cheevos. :(

Why would it say "Genesis:" if it's Mastery System file? And this game was also unable to detect a "game ID". My scroll-back log only goes as far back to still be SMS, yet it looks like it's doing that "Genesis" bit on all of the games. Here's a working result:

Checking "/home/pi/RetroPie/roms/mastersystem/Strider (USA, Europe).sms"...
--- Genesis: 2bc1f448529ee58162da600d966d9a14
--- plain MD5: 48af5ae2cad7c48c81d7c2553a8ed426
--- game ID: 250
--- Game Title: "Strider"
--- This game has been added to "/home/pi/.emulationstation/collections/custom-achievements.cfg".
--- "/home/pi/RetroPie/roms/mastersystem/Strider (USA, Europe).sms" HAS CHEEVOS!

I also noticed some indications of needing a username/password for some SNES games, but it went by too quickly to be able to copy/paste it. The script never prompted for a login, just went on to the next one.

cubed3d · 15 Nov 2017, 12:08

Just a quick question -
Why would hascheevos report this:

pi@RetroPie:~ $ sudo ~/hascheevos/bin/hascheevos.sh --collection --system all
Checking JSON hash libraries...
Done!
ERROR: not a RetroPie system.
The "--collection" option is available only for RetroPie systems.

Obviously running RetroPie on a RPi3 - Roms located on a mounted network share. Everything else working perfectly, except for hascheevos :/

Edit Just tried running it on a single rom as a test and got the following:

pi@RetroPie:~ $ sudo ~/hascheevos/bin/hascheevos.sh "/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc"
Checking JSON hash libraries...
Done!
Checking "/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc"...
cheevos_eval_md5(): Value too large for defined data type
"/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc" has no cheevos. :(

Could this be to do with the ROMs being on a network share? Thanks in advance.

meleu · 15 Nov 2017, 13:29

@cubed3d that problem happened because of the sudo. There's no need to use the script this way. Calling the script with sudo is confusing it about the homedir. I'll try to make the script smart enough to deal with it ;-)

cubed3d · 15 Nov 2017, 13:35

@meleu Thanks for the quick reply!

Had a try without the sudo as suggested, a bit better progress:

pi@RetroPie:~ $ ~/hascheevos/bin/hascheevos.sh --collection --system all        Checking JSON hash libraries...
Done!
WARNING: ignoring "pcengine": not found.
WARNING: ignoring "segacd": not found.
WARNING: ignoring "xbox360": not found.
WARNING: ignoring "atari": not found.
Checking "/home/pi/RetroPie/roms/megadrive/3 Ninjas Kick Back (USA).md"...
cheevos_eval_md5(): Value too large for defined data type
"/home/pi/RetroPie/roms/megadrive/3 Ninjas Kick Back (USA).md" has no cheevos. :(

Checking "/home/pi/RetroPie/roms/megadrive/6-Pak (USA).md"...
cheevos_eval_md5(): Value too large for defined data type
"/home/pi/RetroPie/roms/megadrive/6-Pak (USA).md" has no cheevos. :(

For each and every ROM, cancelled it shortly after.
Seems its having trouble determining the MD5 hash of each ROM file now, I will try with locally stored ROMs and report back!

cubed3d · 15 Nov 2017, 14:11

Unmounted network share and tried locally:

pi@RetroPie:~/RetroPie/roms/snes $ ~/hascheevos/bin/hascheevos.sh "/home/pi/RetroPie/roms/snes/Super Mario Kart (Europe).zip"
Checking JSON hash libraries...
Done!
Checking "/home/pi/RetroPie/roms/snes/Super Mario Kart (Europe).zip"...
--- SNES: d504d075130c4a544814eb21e691f904
--- game ID: 232
--- Game Title: "Super Mario Kart"
--- "/home/pi/RetroPie/roms/snes/Super Mario Kart (Europe).zip" HAS CHEEVOS!

Local ROMs working. Now I'm thinking it may have something to do with write permissions? Is there a more verbose output to debug this?

meleu · 15 Nov 2017, 14:27

@cubed3d said in hascheevos.sh - a tool for RetroPie/RetroAchievements users:

Is there a more verbose output to debug this?

Yes. Do what @hansolo77 did here and post the debug.txt file on codepad.org

cubed3d · 15 Nov 2017, 18:15

@meleu
Ouput to debug.txt

Nothing really standing out except for the MD5 check error. scratches head