what about adding a cheevos flag in gamelist.xml ?

pjft

@meleu Well done good sir!

I'm sure you thought of this, but just in case if I may recommend something, I'd create a backup of the gamelists before actually overwriting them, in case something goes wrong in the new script . :)

sselph

@meleu Cool. Sorry I didn't see this eariler, but looks like you have it all sorted out. I also have hashing code for several rom types. I have a shasum utility here that demonstrates its use:
https://github.com/sselph/scraper/blob/master/shasum/shasum.go#L14
If you need to reference how I handle other systems, the implementation that transforms the bits to something that hashes nicely is here:
https://github.com/sselph/scraper/blob/master/rom/hash/hash.go

But the only other one that is tricky is N64. .Z64 and .N64 have each byte swapped around differently. Atari 7800 and Lynx have optional headers that won't match no-intro hash if they are there but those are easy enough to detect and remove.

meleu

@sselph thanks for the references man! :-)

For RetroAchievements, the only non-plain MD5 hashes are for SNES, MegaDrive and NES ROMS. All other systems (including N64) use plain MD5. Then I think everything is fine. :-)

Cheers!

meleu

I've just made a small improvement on the hascheevos.sh script and now you guys have an easy way to get a list of your ROMs that have cheevos.

From the repo's README:

Create a list of all ROMs that have cheevos in a directory.

The only thing the script puts on stdout are the file name of ROMs that have cheevos. Everything else are printed in stderr. Then if you want a list of all ROMs that have cheevos in a directory, do something like this:

$ ./hascheevos.sh -u USER -p PASSWORD /path/to/megadrive/* > ~/megadrive-roms-with-cheevos.txt

Don't worry about non-ROM files in the same directory (like gamelist.xml or .srm files), the script ignores files with invalid extensions. ;-)

hansolo77

Getting closer to a reality! :)

pjft

Hi @meleu .

Just letting you know that I hope to pick this up shortly.

Best.

meleu

@pjft I'll try to finish the scraper (the part that updates the gamelist.xml) this weekend (if the kids permit)! :-)

pjft

@meleu not a bother.

I might put together a build for that, for you to test anyway. Hopefully that'll help.

If time allows for it, of course:)

meleu

@pjft no need to upload a binary for that. Just the code is enough. ;-)

Thanks dude!

robertybob

Is this something that can be added to the RetroPie Setup menu, or intergrated into the built-in ES scraper? Or is this gong to be another cool tool that gets lost in the forums? :(

hansolo77

REQUEST - Update the tool to spit out additional texts. Specifically, I'd like to see what games exist with cheevos that I don't have, so I know to go out and find them.

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

QUESTION - I'm happy with the way it works now, but what if I choose to run the scraper again.. will it make duplicate copies in the texts, thus making me have multiple copies of the same game when I run the tool to auto-add it to a custom system?

PokeEngineer

@robertybob

I second this.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

REQUEST - Update the tool to spit out additional texts. Specifically, I'd like to see what games exist with cheevos that I don't have, so I know to go out and find them.

I would love this feature too, but I'm afraid it's not going to happen.

Why?

Because with the [currently?] RetroAchievements.org isn't smart enough to provide a list with all games that have cheevos. Implementing what you're suggesting with the current "technology" would stress the RA server (and would take ~~ages~~ several hours to accomplish the task).

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

Not sure what you mean. Can you expand that?

QUESTION - I'm happy with the way it works now, but what if I choose to run the scraper again.. will it make duplicate copies in the texts, thus making me have multiple copies of the same game when I run the tool to auto-add it to a custom system?

No. The add-game-to-custom-system.sh doesn't create duplicated links.

hansolo77

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

REQUEST - Update the tool to keep a log of the date you last checked for cheevo games, then compare the dates with TODAY to see if it needs re-scraped.

Not sure what you mean. Can you expand that?

Just as means to know if I should scrape again.. Like in the Retropie-Setup script, it tells you when the last time was that you last updated the script. I'd like to have a similar system in place, that checks when the last time was you last scraped. Even better would be a simple version number check. See if your version is the same you have in your github databases. That way I'm not needlessly re-scraping all my games again a week later if you haven't updated the databases.

This actually brings up another thing I noticed. When it was going through checking each rom, it was actually going back and saying something like "checking on retroachievements". Is it doing that, or is that just a bit of bad wording since you have it scraping from github?

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

Just as means to know if I should scrape again.

OK, I got it. I'll try to think in something to avoid re-scraping everything. Would you be kind enough to make those requests on the repository issue tracker? I think I'll forget if this stays here on the forums. :-)

https://github.com/meleu/hascheevos/issues

When it was going through checking each rom, it was actually going back and saying something like "checking on retroachievements".

It's really doing that. If the script fails to find info in the github files, it tries to get from RetroAchievements.org.

hansolo77

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

OK, I got it. I'll try to think in something to avoid re-scraping everything. Would you be kind enough to make those requests on the repository issue tracker? I think I'll forget if this stays here on the forums. :-)

https://github.com/meleu/hascheevos/issues

Will Do!

It's really doing that. If the script fails to find info in the github files, it tries to get from RetroAchievements.org.

Well it was doing it for what appeared to be ALL of them. So if the server crashed yesterday or was slow, it was probably my fault.

lilbud

I think cheevos are fine, but what would really be a worthwhile addition would be backloggery support: http://www.backloggery.com/

meleu

Hello friends!

I've got distracted with PS Vita hacking and hunting RetroAchievements for some Mega Man games. But now I finally made an useful progress on my "cheevos scraper".

You can check it in this repository: https://github.com/meleu/hascheevos

The README has instructions on how to use it.

An example of how useful the tool can be: In order to update your NES' gamelist.xml adding <achievements>true</achievements> for every game that has cheevos, you just need to use this command:

$ ./hascheevos.sh -u USER -p PASSWORD --system nes --scrape

Now let's wait if @pjft can find sometime to work on EmulationStation code in order to make the <achievements> flag filterable. :-)

hansolo77

Exciting progress! So just to be clear.. in order to access the new feature (adding the flag), I will need to rebuild the binary correct? I'm not particularly eager to get this specific feature right now, I'd just like to know. Quick question.. is there a reason a binary hasn't been made yet, to skip having to build it? 2nd quick question.. any progress been made on checking to see what new roms exist without having to completely rescrape?

Kudos!

cyperghost

@hansolo77

in order to access the new feature (adding the flag), I will need to rebuild the binary correct?

Yes!

Quick question.. is there a reason a binary hasn't been made yet, to skip having to build it?

As @pjft is the develeoper for adding new flags to XMLs (as he did by adding favorite selections) it's up to him for a release.

So things will surely be done, just let them grow. I highly appreciate such a possibility as the RetroAchievements makes the gaming experience really better ... :)

@meleu
I will point to your latest thread with my RA-automate-script