what about adding a cheevos flag in gamelist.xml ?

pjft

@meleu let's start a separate thread for the implementation. I'm sure we'll find a reasonable and scalable way to do so.

Shall we? :)

meleu

@pjft great! I would like to discuss a little more with the RetroAchievements guys about the "cheevos scraper" and its impact on server's performance before releasing my tool to the community.

But we can go ahead with the implementation in ES! Sure! Let's go!!

pjft

@meleu I was actually referring to the scraper implementation :) I'd like to see if there's a way to make it scalable with minimum overhead on the server.

I was just offering to create a separate thread to bounce ideas there on the implementation end in regards to the server calls, just that.

Would you want to create a separate thread for that?

meleu

@pjft Well, I think it's more about RetroAchievements.org and their available php requests. Maybe it's better to discuss on their forums, on the thread I linked above.

On that thread I detailed the method I'm using to know if a game has cheevos. My concern about overloading the server is because the available transaction where I can see if a game has cheevos processes/returns a lot of data from where I need only a small snippet. Maybe Scott (that site's maintainer/admin) could create a php request to return only the info I need.

pjft

@meleu Got it, I'll follow up there.

There are two stress points in your design: one is the one you identified, which is the data that needs to be transferred per request; but the other is the number of requests overall.

I added my comments there.

Thanks!

sselph

@meleu Let me know if there is something I can do to help. It would be best to open a github issue so I don't miss it.

meleu

@sselph hey mate. I'll do it when I have something more concrete. The current status of this feature is "waiting RetroAchievements admin say what should be the nicer way to retrieve this info without stressing his server."

I'll open an issue on your scraper's repo if we get some progress.

Cheers!

meleu

@pjft @hansolo77 A heads up answering here the question from the other thread...

@hansolo77 said in Please Test: Creating Custom Collections:

did the post @meleu started about adding gamelist.xml support for achievements get deleted? I can't find it.

The post is this very same here.

And about the "cheevos scraper", I'm working on it right now.

We discussed in a RetroAchievements forum topic and IMHO if I release my "cheevos scraper" to the community it would stress the RA server. To avoid it, I created the repo hascheevoswith files separated by console. Each file has all registered game IDs in RetroAchievements database for the respective console and a true or false saying if the game has cheevos.

I'll try to update those files once a week, but I cannot guarantee.

pjft

@meleu thanks for the update, just saw your repo. A question: could you not store the hashes as well so you don't need to query the RA server at all?

meleu

@pjft said in what about adding a cheevos flag in gamelist.xml ?:

A question: could you not store the hashes as well so you don't need to query the RA server at all?

Yes, I could. But I decided to not.

Why?

getting the hashlibraries from RA server isn't expensive (one request per console, it costs less than accessing one RA web page, for example).
those guys update the hash database more than once a week (IMHO). Then my copy of hashes would be outdated quite frequently.

hansolo77

Thanks for the update. I couldn't find this thread because I was looking for "achievements" and not "cheevo" lol.

It's a shame we can't get anywhere with this... You say that storing the hashes on github is a bad idea because the Db's are updated multiple times a week and will become outdated. Yet you also say the impact on the server is minimal if you're doing a system/console grab. Is there no way to automate a request at say 2am when most people are sleeping, then just have a 24-hour system? Have the script that updates github's hash database on something server side maybe at Retropi's website. Then whenever a user want's to update their lists, it connects to github's db's instead.

Something else I was thinking about is how you could display it in ES. You could probably get ES to display something like "Online Database Last Updated XXXXXX" and "Local System Last Updated XXXXXXX" meaning your RetroPie... But that's if you wanna get all pretty. updating the gamelist.xml would probably need to be done outside of ES to prevent overwrites. So you'd need to write a nifty little addon for the Retropie-Setup menus.

But I think the first hurdle to tackle would be to get something set up to automate grabbing the hashes. Maybe 48 hours instead of 24 would be good too? Once you get that established, writing another script/program to add the true/false to their respective gamelists should be relatively easy I would think.

meleu

@hansolo77 Calm down bro! Otherwise that migraine will come back! :D

The hascheevos repo is only the supply for a "cheevos scraper" script I'm writing. The script will automatically update your gamelist.xml files.

Once your gamelist.xml has the <achievements> metadata you'll be able to filter the games that has cheevos using the way I described here (of course, it will be implemented by @pjft in the future. This feature isn't available in ES yet!)

I hope to release something useful this weekend (at least to scrape cheevos for Gameboy[Color|Advance] games).

hansolo77

I know it's not available yet lol. I was just trying to suggest some ideas. One of the things we're working hard on at work is "Problem Solving Projects".. Yes, we're a grocery store chain. But there are many issues in the company with the way things are done. And things have always stayed the same. These new projects are putting the grunt employees, the ones who DO the work, front and center to coming up with ways to do things better. One of the key components of a successful problem solving project is to be open to the ideas of many, rather than be close minded to just your own. To that end, that's merely what I was trying to accomplish in my previous post. Not as a directorate for a way to do it, but more of a suggestion of a possible solution.

Another key part of our projects is identifying the problem, not the goal. We know our goal, but we have to figure out what the root cause of the problem getting there is. From what it sounds like (and I agree up to this point), the problem is the RetroAchievement's website is incapable of supporting the load required to handle a flood of calls for the information. So a possible "countermeasure" is to mirror the data at another site. This is where you're progressing, and I fully support it as well. Now the problem, since that idea has come up, is that you have no way to maintain the sustainability of the mirror. So my suggestion of an off-hours script to work automatically was just an idea to help provide a possible countermeasure for that problem. It may turn out to be bogus, or the solution you're looking for. But please understand, I wasn't trying to tell you how to do it. We're all in this together.

There's a lot of backstory here about this problem solving that I'd love to discuss with people. The problem with it is that the company as a whole LOVES the fact that we're solving some of the BIG problems. But then we get new management staff or corporate level people who are opposed to change. So what inevitably happens is things go back to the old ways. It's pretty bad right now. Things are worse now than they were before. We had a working solution, but the sustainability left us when corporate level decisions were made to go back the way they were. Now the company is suffering a dramatic (we're talking something like 40%) drop in stock value, and they're announcing hour cuts, forcing early retirements, and closing complete stores. This is right on the heals of a new store franchise competitor moving into the area. 2 other major chains in the area have already cut their losses and ran. My company is trying to hold on to what they have, but are doing it the wrong way. Years ago I worked for a store chain called Big Bear. They were the biggest Midwest (USA) grocery chain until another company called Giant Eagle moved into the neighborhood. Out of fear, the owners of Big Bear sold the company to a group of people who ran businesses and not grocery stores. They had no experience in the customer service / point of sale aspect of life. They fired all their corporate staff with forced early retirements, began tearing down stores, and cutting hours in an attempt to gain some financial momentum. It worked for a while and they saw a substantial net gain from their previous year's income. So they spent that money to build a brand new state-of-the art store. The most expensive undertaking the company had ever made. Their goal was to attract the Giant Eagle customers and bring back the old ones. It flopped. The new managers they hired were from a state (in the USA) where the company didn't exit. These managers were businessmen, not trained in customer service, but trained in making money. So wherever they saw fit, they cut hours, increased costs, fired loosely, and hired strictly. I was full time when this happened, the week they scheduled me 12 hours I quit. Weekly sales had gone, and were now monthly sales. These sales were not set by the store but by pencil pushers in Wall Street. The individual stores could set their own prices based on the market and clientele in the area. Our manager was ignorant enough to upcharge something like 20% on everything. The excuse being upkeep for the 21st century store. When in fact, he would get a 35% bonus each week of whatever the store made over the allotted budget. The grunts, including me, were making $9.50 top out pay after 10 years of service. Barely scraping something like $17k a year after taxes. Meanwhile, he was making a SIX digit salary and weekly 4-digit bonuses. The company was originally local to where I live and spread out to 3 other states. In the end, only the local stores remained, a company halfway across the country owned it and operated it, and I was no longer working for them. 6 months after I quit, they declared bankruptcy. NOW.. there's a new threat to the job I'm working at today. I'm seeing the exact same thing happen. A new company is coming in and trying to take over our territory. Forced retirements are happening, the president and CEO of the company has been replaced twice in the last 5 years, hours are cut so deep we can only afford 2 people a day in the department at THE busiest store in the company in the state, and the 3rd busiest store in the state combined with all others (including Walmart, Giant Eagle, Sams Club, Aldi's, Save Alot, Big Lots, etc). We bring in the most money for this company, and they want to cut our hours even more to the point of 1 person a day in the department. We can't do it with 2 and they want 1. These are corporate and management level issues. I'm not even a department head. I'm just a grunt clerk. But I see this, and I have to work these hours by myself with lines of pissed off people wondering where my help is. I don't want to quit, but I'm THIS close... I've been full time for 2 years, have 8 years in the company. I'm making $11.40 an hour. According to my union hand book, after 72 months I should be topped out at $15.25. Hired in 2009.. 2017-2009=8 years... 8*12=95 months.. 95-72=24. 24 months without a raise. I'm 2 years over due for this raise! The same thing happened to me at Big Bear. They still owe me $2.5 grand in back pay that I'll never see because they went bankrupt. I'll be damned if this happens to me again. What can I do? I've talked to the Union rep countless times. "Oh yeah, it'll take 90 days from the time you went full time till you see it" or "I was mistaken, because of the year you were hired you're on a different contract so it'll be 6 months from your full time date." Then it's "oh yeah they won't be implementing any wage increases till after the start of the new fiscal year". Then March comes around and he's "You won't see it until a year after you went full time." Ok, so it's now 2 years and I still don't have it. He's in the store this week and said "Yeah you won't see it till after the day you got hired." So that will be September some time. I'm really pissed about it. He told me 90 days 2 years ago. If you don't know, just say you don't know and I'll ask somebody else. Stop lying hoping it'll just happen and I'll forget about it. So I called the HR department. Nobody answered the phone except a secretary. I explain to her what's going on, she transfers me to the head of payroll. That person's not in, so I leave a message. I leave a message every day for 3 days. No calls. Nobody talks to me at all. She's probably on vacation but I can't help remember what Big Bear did and think they're purposely avoiding me because they can't afford to pay me the back pay and raise. I'd hire a lawyer if I was able to afford one. UUUHG! And you tease about my migraines?

Oh and, sorry.. Didn't realize I was writing a book about my life here lol.

meleu

@hansolo77 WOW!! I think I'll wait Netflix adapt your post to a series. :D

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

I'd hire a lawyer if I was able to afford one.

If only you could afford a ship and find a very hairy mate... You could move to the "logistics" area cough cough smuggling cough cough and deliver all those Chinese gaming gadgets all over the world...

(Hey friend, I'm sorry for not being able to help. I wish you good times.)

hansolo77

It's all good. I didn't even get to say the BEST part, about how this "problem solving" stuff is supposed to save the company, yet it cost them over $1M a year to pay the guys who are training us to solve problems and then they ignore the solutions we present them. It's laughable. :)

So yeah, cheevos... Is there even a system available that you can use to run automatic backups to github like I suggested? I don't know much about internet infrastructure, so the capabilities available are beyond my knowledge. If you can do it manually, by downloading the db's and storing them elsewhere, there's gotta be a way to automate it. I'm just fearful when I hear phrases like:

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

I'll try to update those files once a week, but I cannot guarantee.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

Is there even a system available that you can use to run automatic backups to github like I suggested?

Yes, the git command. :-)

Your suggestion about automating the updating process is cool, but unfortunately I don't have a Linux system online 24/7.

To be really honest I thought about you and @grenade44 (or other RetroPie/cheevos users, maybe @Sano ) to help me in keeping the repo up to date. I just need to make my tool user friendly enough and then I'll talk to you guys.

By te way, if someone could kindly provide me a Linux shell account (on a 24/7 online system) for this task, it would be useful for the community. I promess I won't abuse the system resources (everything could be monitored, no need for privacy).

hansolo77

What about a seedbox?... or a router?

meleu

@pjft Just a heads up, I'm working hard on the cheevos scraper side and am having satisfactory results.

The delay was caused by the fact that the hash calculation for MegaDrive, NES and SNES roms isn't always a plain MD5. Then I had to write my own hash calculator. Thanks to the open source philosophy I could have been strongly inspired by RetroArch code.

Just for curiosity, this is my cheevos hash calculator: https://github.com/meleu/share/blob/master/cheevoshash.c

Now I'm working on the bash script to deal with gamelist.xml files. It's working fine right now. But the code still have that prototype looking. :)

I just need to polish the bash script and then we will be ready to scrape our ROM sets tagging the files that has cheevos!

Cheers!

pjft

@meleu Thank you for the update good sir!

I have not had much time recently, unfortunately, but adding the flag, sorting and filtering shouldn't be a big issue. :)

I've been meaning to tidy up the custom collections work before moving on to something else, hence my silence here, but this is committed to on my end. I'll definitely make this happen :)

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :) Well done.

Thanks!