what about adding a cheevos flag in gamelist.xml ?

meleu

@pjft Just a heads up, I'm working hard on the cheevos scraper side and am having satisfactory results.

The delay was caused by the fact that the hash calculation for MegaDrive, NES and SNES roms isn't always a plain MD5. Then I had to write my own hash calculator. Thanks to the open source philosophy I could have been strongly inspired by RetroArch code.

Just for curiosity, this is my cheevos hash calculator: https://github.com/meleu/share/blob/master/cheevoshash.c

Now I'm working on the bash script to deal with gamelist.xml files. It's working fine right now. But the code still have that prototype looking. :)

I just need to polish the bash script and then we will be ready to scrape our ROM sets tagging the files that has cheevos!

Cheers!

pjft

@meleu Thank you for the update good sir!

I have not had much time recently, unfortunately, but adding the flag, sorting and filtering shouldn't be a big issue. :)

I've been meaning to tidy up the custom collections work before moving on to something else, hence my silence here, but this is committed to on my end. I'll definitely make this happen :)

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :) Well done.

Thanks!

meleu

@pjft said in what about adding a cheevos flag in gamelist.xml ?:

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :)

Hey bro, don't feel like that, I loved to code that! And to be honest I don't feel like I did that "on my own". I ~~stole~~ was strongly inspired by RetroArch code. :)

meleu

Progress log

ROM hash calculator for cheevos: DONE! (as I said in a post above)
A script to check if a ROM file has cheevos: DONE! (but there is room for small improvements)

You guys can already test the script's effectiveness! Just follow the instructions in the README of the repository https://github.com/meleu/hascheevos

Next step

write the script that will create/update the <achievements> metadata in the gamelist.xml.

pjft

@meleu Well done good sir!

I'm sure you thought of this, but just in case if I may recommend something, I'd create a backup of the gamelists before actually overwriting them, in case something goes wrong in the new script . :)

sselph

@meleu Cool. Sorry I didn't see this eariler, but looks like you have it all sorted out. I also have hashing code for several rom types. I have a shasum utility here that demonstrates its use:
https://github.com/sselph/scraper/blob/master/shasum/shasum.go#L14
If you need to reference how I handle other systems, the implementation that transforms the bits to something that hashes nicely is here:
https://github.com/sselph/scraper/blob/master/rom/hash/hash.go

But the only other one that is tricky is N64. .Z64 and .N64 have each byte swapped around differently. Atari 7800 and Lynx have optional headers that won't match no-intro hash if they are there but those are easy enough to detect and remove.

meleu

@sselph thanks for the references man! :-)

For RetroAchievements, the only non-plain MD5 hashes are for SNES, MegaDrive and NES ROMS. All other systems (including N64) use plain MD5. Then I think everything is fine. :-)

Cheers!

meleu

I've just made a small improvement on the hascheevos.sh script and now you guys have an easy way to get a list of your ROMs that have cheevos.

From the repo's README:

Create a list of all ROMs that have cheevos in a directory.

The only thing the script puts on stdout are the file name of ROMs that have cheevos. Everything else are printed in stderr. Then if you want a list of all ROMs that have cheevos in a directory, do something like this:

$ ./hascheevos.sh -u USER -p PASSWORD /path/to/megadrive/* > ~/megadrive-roms-with-cheevos.txt

Don't worry about non-ROM files in the same directory (like gamelist.xml or .srm files), the script ignores files with invalid extensions. ;-)

hansolo77

Getting closer to a reality! :)

pjft

Hi @meleu .

Just letting you know that I hope to pick this up shortly.

Best.

meleu

@pjft I'll try to finish the scraper (the part that updates the gamelist.xml) this weekend (if the kids permit)! :-)

pjft

@meleu not a bother.

I might put together a build for that, for you to test anyway. Hopefully that'll help.

If time allows for it, of course:)

meleu

@pjft no need to upload a binary for that. Just the code is enough. ;-)

Thanks dude!

robertybob

Is this something that can be added to the RetroPie Setup menu, or intergrated into the built-in ES scraper? Or is this gong to be another cool tool that gets lost in the forums? :(

hansolo77

REQUEST - Update the tool to spit out additional texts. Specifically, I'd like to see what games exist with cheevos that I don't have, so I know to go out and find them.

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

QUESTION - I'm happy with the way it works now, but what if I choose to run the scraper again.. will it make duplicate copies in the texts, thus making me have multiple copies of the same game when I run the tool to auto-add it to a custom system?

PokeEngineer

@robertybob

I second this.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

REQUEST - Update the tool to spit out additional texts. Specifically, I'd like to see what games exist with cheevos that I don't have, so I know to go out and find them.

I would love this feature too, but I'm afraid it's not going to happen.

Why?

Because with the [currently?] RetroAchievements.org isn't smart enough to provide a list with all games that have cheevos. Implementing what you're suggesting with the current "technology" would stress the RA server (and would take ~~ages~~ several hours to accomplish the task).

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

Not sure what you mean. Can you expand that?

QUESTION - I'm happy with the way it works now, but what if I choose to run the scraper again.. will it make duplicate copies in the texts, thus making me have multiple copies of the same game when I run the tool to auto-add it to a custom system?

No. The add-game-to-custom-system.sh doesn't create duplicated links.

hansolo77

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

Not sure what you mean. Can you expand that?

Just as means to know if I should scrape again.. Like in the Retropie-Setup script, it tells you when the last time was that you last updated the script. I'd like to have a similar system in place, that checks when the last time was you last scraped. Even better would be a simple version number check. See if your version is the same you have in your github databases. That way I'm not needlessly re-scraping all my games again a week later if you haven't updated the databases.

This actually brings up another thing I noticed. When it was going through checking each rom, it was actually going back and saying something like "checking on retroachievements". Is it doing that, or is that just a bit of bad wording since you have it scraping from github?

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

Just as means to know if I should scrape again.

OK, I got it. I'll try to think in something to avoid re-scraping everything. Would you be kind enough to make those requests on the repository issue tracker? I think I'll forget if this stays here on the forums. :-)

https://github.com/meleu/hascheevos/issues

When it was going through checking each rom, it was actually going back and saying something like "checking on retroachievements".

It's really doing that. If the script fails to find info in the github files, it tries to get from RetroAchievements.org.

hansolo77

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

OK, I got it. I'll try to think in something to avoid re-scraping everything. Would you be kind enough to make those requests on the repository issue tracker? I think I'll forget if this stays here on the forums. :-)

https://github.com/meleu/hascheevos/issues

Will Do!

It's really doing that. If the script fails to find info in the github files, it tries to get from RetroAchievements.org.

Well it was doing it for what appeared to be ALL of them. So if the server crashed yesterday or was slow, it was probably my fault.