what about adding a cheevos flag in gamelist.xml ?

R1n0X

@pjft I think that we could use a filter in the option "SORT GAMES" in menu

meleu

@r1n0x @grenade44 @hansolo77 and all cheevos users. I'm trying to be very clear now of what I am asking for ( @pjft please confirm).

I want to "enter" in a system (that view where you see the games of a system), press select (it will open the "OPTIONS" screen), choose the "FILTER GAMELIST" option (it will open the "FILTER GAMELIST BY" screen) and there will be a filter named "ACHIEVEMENTS".

Look the screenshots below:

press select when you are in system's game list

in the menu below there will be an ACHIEVEMENTS option

hansolo77

Well, seems like I'm getting outvoted. Everybody is more or less going with the filter option. I'd still really like to have a custom menu on the systems screen. Might just have to use a collection like originally planned.

meleu

@hansolo77 maybe my "cheevos scraper" can be useful for your use case too. I'll post here when it's usable.

hansolo77

No doubt it'll be useful. Up till now it's all had to be done manually, but if a quick and easy automated way to at least identify the games in your list that ARE compatible can be done, that's the way to go.

meleu

Just an update:
I'm making some progress with my "cheevos scraper", but I'm afraid it would overload the RetroAchievements server (wich we, cheevos users, know isn't that powerful). I'm in contact with the RetroAchievements.org guys to find the less stressfull way to retrieve the needed info from the site.

hansolo77

Boy do I know that all to well. I hate doing everything necessary for a cheevo, just to have it not pop because the server is down. I wish there was a good way to help get them to relocate to an upgraded server.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

I wish there was a good way to help get them to relocate to an upgraded server.

Being a patron looks like a good way to help ;-)
https://www.patreon.com/retroachievements

pjft

@meleu let's start a separate thread for the implementation. I'm sure we'll find a reasonable and scalable way to do so.

Shall we? :)

meleu

@pjft great! I would like to discuss a little more with the RetroAchievements guys about the "cheevos scraper" and its impact on server's performance before releasing my tool to the community.

But we can go ahead with the implementation in ES! Sure! Let's go!!

pjft

@meleu I was actually referring to the scraper implementation :) I'd like to see if there's a way to make it scalable with minimum overhead on the server.

I was just offering to create a separate thread to bounce ideas there on the implementation end in regards to the server calls, just that.

Would you want to create a separate thread for that?

meleu

@pjft Well, I think it's more about RetroAchievements.org and their available php requests. Maybe it's better to discuss on their forums, on the thread I linked above.

On that thread I detailed the method I'm using to know if a game has cheevos. My concern about overloading the server is because the available transaction where I can see if a game has cheevos processes/returns a lot of data from where I need only a small snippet. Maybe Scott (that site's maintainer/admin) could create a php request to return only the info I need.

pjft

@meleu Got it, I'll follow up there.

There are two stress points in your design: one is the one you identified, which is the data that needs to be transferred per request; but the other is the number of requests overall.

I added my comments there.

Thanks!

sselph

@meleu Let me know if there is something I can do to help. It would be best to open a github issue so I don't miss it.

meleu

@sselph hey mate. I'll do it when I have something more concrete. The current status of this feature is "waiting RetroAchievements admin say what should be the nicer way to retrieve this info without stressing his server."

I'll open an issue on your scraper's repo if we get some progress.

Cheers!

meleu

@pjft @hansolo77 A heads up answering here the question from the other thread...

@hansolo77 said in Please Test: Creating Custom Collections:

did the post @meleu started about adding gamelist.xml support for achievements get deleted? I can't find it.

The post is this very same here.

And about the "cheevos scraper", I'm working on it right now.

We discussed in a RetroAchievements forum topic and IMHO if I release my "cheevos scraper" to the community it would stress the RA server. To avoid it, I created the repo hascheevoswith files separated by console. Each file has all registered game IDs in RetroAchievements database for the respective console and a true or false saying if the game has cheevos.

I'll try to update those files once a week, but I cannot guarantee.

pjft

@meleu thanks for the update, just saw your repo. A question: could you not store the hashes as well so you don't need to query the RA server at all?

meleu

@pjft said in what about adding a cheevos flag in gamelist.xml ?:

A question: could you not store the hashes as well so you don't need to query the RA server at all?

Yes, I could. But I decided to not.

Why?

getting the hashlibraries from RA server isn't expensive (one request per console, it costs less than accessing one RA web page, for example).
those guys update the hash database more than once a week (IMHO). Then my copy of hashes would be outdated quite frequently.

hansolo77

Thanks for the update. I couldn't find this thread because I was looking for "achievements" and not "cheevo" lol.

It's a shame we can't get anywhere with this... You say that storing the hashes on github is a bad idea because the Db's are updated multiple times a week and will become outdated. Yet you also say the impact on the server is minimal if you're doing a system/console grab. Is there no way to automate a request at say 2am when most people are sleeping, then just have a 24-hour system? Have the script that updates github's hash database on something server side maybe at Retropi's website. Then whenever a user want's to update their lists, it connects to github's db's instead.

Something else I was thinking about is how you could display it in ES. You could probably get ES to display something like "Online Database Last Updated XXXXXX" and "Local System Last Updated XXXXXXX" meaning your RetroPie... But that's if you wanna get all pretty. updating the gamelist.xml would probably need to be done outside of ES to prevent overwrites. So you'd need to write a nifty little addon for the Retropie-Setup menus.

But I think the first hurdle to tackle would be to get something set up to automate grabbing the hashes. Maybe 48 hours instead of 24 would be good too? Once you get that established, writing another script/program to add the true/false to their respective gamelists should be relatively easy I would think.

meleu

@hansolo77 Calm down bro! Otherwise that migraine will come back! :D

The hascheevos repo is only the supply for a "cheevos scraper" script I'm writing. The script will automatically update your gamelist.xml files.

Once your gamelist.xml has the <achievements> metadata you'll be able to filter the games that has cheevos using the way I described here (of course, it will be implemented by @pjft in the future. This feature isn't available in ES yet!)

I hope to release something useful this weekend (at least to scrape cheevos for Gameboy[Color|Advance] games).