what about adding a cheevos flag in gamelist.xml ?

hansolo77

It's all good. I didn't even get to say the BEST part, about how this "problem solving" stuff is supposed to save the company, yet it cost them over $1M a year to pay the guys who are training us to solve problems and then they ignore the solutions we present them. It's laughable. :)

So yeah, cheevos... Is there even a system available that you can use to run automatic backups to github like I suggested? I don't know much about internet infrastructure, so the capabilities available are beyond my knowledge. If you can do it manually, by downloading the db's and storing them elsewhere, there's gotta be a way to automate it. I'm just fearful when I hear phrases like:

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

I'll try to update those files once a week, but I cannot guarantee.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

Is there even a system available that you can use to run automatic backups to github like I suggested?

Yes, the git command. :-)

Your suggestion about automating the updating process is cool, but unfortunately I don't have a Linux system online 24/7.

To be really honest I thought about you and @grenade44 (or other RetroPie/cheevos users, maybe @Sano ) to help me in keeping the repo up to date. I just need to make my tool user friendly enough and then I'll talk to you guys.

By te way, if someone could kindly provide me a Linux shell account (on a 24/7 online system) for this task, it would be useful for the community. I promess I won't abuse the system resources (everything could be monitored, no need for privacy).

hansolo77

What about a seedbox?... or a router?

meleu

@pjft Just a heads up, I'm working hard on the cheevos scraper side and am having satisfactory results.

The delay was caused by the fact that the hash calculation for MegaDrive, NES and SNES roms isn't always a plain MD5. Then I had to write my own hash calculator. Thanks to the open source philosophy I could have been strongly inspired by RetroArch code.

Just for curiosity, this is my cheevos hash calculator: https://github.com/meleu/share/blob/master/cheevoshash.c

Now I'm working on the bash script to deal with gamelist.xml files. It's working fine right now. But the code still have that prototype looking. :)

I just need to polish the bash script and then we will be ready to scrape our ROM sets tagging the files that has cheevos!

Cheers!

pjft

@meleu Thank you for the update good sir!

I have not had much time recently, unfortunately, but adding the flag, sorting and filtering shouldn't be a big issue. :)

I've been meaning to tidy up the custom collections work before moving on to something else, hence my silence here, but this is committed to on my end. I'll definitely make this happen :)

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :) Well done.

Thanks!

meleu

@pjft said in what about adding a cheevos flag in gamelist.xml ?:

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :)

Hey bro, don't feel like that, I loved to code that! And to be honest I don't feel like I did that "on my own". I ~~stole~~ was strongly inspired by RetroArch code. :)

meleu

Progress log

ROM hash calculator for cheevos: DONE! (as I said in a post above)
A script to check if a ROM file has cheevos: DONE! (but there is room for small improvements)

You guys can already test the script's effectiveness! Just follow the instructions in the README of the repository https://github.com/meleu/hascheevos

Next step

write the script that will create/update the <achievements> metadata in the gamelist.xml.

pjft

@meleu Well done good sir!

I'm sure you thought of this, but just in case if I may recommend something, I'd create a backup of the gamelists before actually overwriting them, in case something goes wrong in the new script . :)

sselph

@meleu Cool. Sorry I didn't see this eariler, but looks like you have it all sorted out. I also have hashing code for several rom types. I have a shasum utility here that demonstrates its use:
https://github.com/sselph/scraper/blob/master/shasum/shasum.go#L14
If you need to reference how I handle other systems, the implementation that transforms the bits to something that hashes nicely is here:
https://github.com/sselph/scraper/blob/master/rom/hash/hash.go

But the only other one that is tricky is N64. .Z64 and .N64 have each byte swapped around differently. Atari 7800 and Lynx have optional headers that won't match no-intro hash if they are there but those are easy enough to detect and remove.

meleu

@sselph thanks for the references man! :-)

For RetroAchievements, the only non-plain MD5 hashes are for SNES, MegaDrive and NES ROMS. All other systems (including N64) use plain MD5. Then I think everything is fine. :-)

Cheers!

meleu

I've just made a small improvement on the hascheevos.sh script and now you guys have an easy way to get a list of your ROMs that have cheevos.

From the repo's README:

Create a list of all ROMs that have cheevos in a directory.

The only thing the script puts on stdout are the file name of ROMs that have cheevos. Everything else are printed in stderr. Then if you want a list of all ROMs that have cheevos in a directory, do something like this:

$ ./hascheevos.sh -u USER -p PASSWORD /path/to/megadrive/* > ~/megadrive-roms-with-cheevos.txt

Don't worry about non-ROM files in the same directory (like gamelist.xml or .srm files), the script ignores files with invalid extensions. ;-)

hansolo77

Getting closer to a reality! :)

pjft

Hi @meleu .

Just letting you know that I hope to pick this up shortly.

Best.

meleu

@pjft I'll try to finish the scraper (the part that updates the gamelist.xml) this weekend (if the kids permit)! :-)

pjft

@meleu not a bother.

I might put together a build for that, for you to test anyway. Hopefully that'll help.

If time allows for it, of course:)

meleu

@pjft no need to upload a binary for that. Just the code is enough. ;-)

Thanks dude!

robertybob

Is this something that can be added to the RetroPie Setup menu, or intergrated into the built-in ES scraper? Or is this gong to be another cool tool that gets lost in the forums? :(

hansolo77

REQUEST - Update the tool to spit out additional texts. Specifically, I'd like to see what games exist with cheevos that I don't have, so I know to go out and find them.

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

QUESTION - I'm happy with the way it works now, but what if I choose to run the scraper again.. will it make duplicate copies in the texts, thus making me have multiple copies of the same game when I run the tool to auto-add it to a custom system?

PokeEngineer

@robertybob

I second this.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

REQUEST - Update the tool to spit out additional texts. Specifically, I'd like to see what games exist with cheevos that I don't have, so I know to go out and find them.

I would love this feature too, but I'm afraid it's not going to happen.

Why?

Because with the [currently?] RetroAchievements.org isn't smart enough to provide a list with all games that have cheevos. Implementing what you're suggesting with the current "technology" would stress the RA server (and would take ~~ages~~ several hours to accomplish the task).

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

Not sure what you mean. Can you expand that?

QUESTION - I'm happy with the way it works now, but what if I choose to run the scraper again.. will it make duplicate copies in the texts, thus making me have multiple copies of the same game when I run the tool to auto-add it to a custom system?

No. The add-game-to-custom-system.sh doesn't create duplicated links.