what about adding a cheevos flag in gamelist.xml ?

hansolo77

Thanks for the update. I couldn't find this thread because I was looking for "achievements" and not "cheevo" lol.

It's a shame we can't get anywhere with this... You say that storing the hashes on github is a bad idea because the Db's are updated multiple times a week and will become outdated. Yet you also say the impact on the server is minimal if you're doing a system/console grab. Is there no way to automate a request at say 2am when most people are sleeping, then just have a 24-hour system? Have the script that updates github's hash database on something server side maybe at Retropi's website. Then whenever a user want's to update their lists, it connects to github's db's instead.

Something else I was thinking about is how you could display it in ES. You could probably get ES to display something like "Online Database Last Updated XXXXXX" and "Local System Last Updated XXXXXXX" meaning your RetroPie... But that's if you wanna get all pretty. updating the gamelist.xml would probably need to be done outside of ES to prevent overwrites. So you'd need to write a nifty little addon for the Retropie-Setup menus.

But I think the first hurdle to tackle would be to get something set up to automate grabbing the hashes. Maybe 48 hours instead of 24 would be good too? Once you get that established, writing another script/program to add the true/false to their respective gamelists should be relatively easy I would think.

meleu

@hansolo77 Calm down bro! Otherwise that migraine will come back! :D

The hascheevos repo is only the supply for a "cheevos scraper" script I'm writing. The script will automatically update your gamelist.xml files.

Once your gamelist.xml has the <achievements> metadata you'll be able to filter the games that has cheevos using the way I described here (of course, it will be implemented by @pjft in the future. This feature isn't available in ES yet!)

I hope to release something useful this weekend (at least to scrape cheevos for Gameboy[Color|Advance] games).

hansolo77

I know it's not available yet lol. I was just trying to suggest some ideas. One of the things we're working hard on at work is "Problem Solving Projects".. Yes, we're a grocery store chain. But there are many issues in the company with the way things are done. And things have always stayed the same. These new projects are putting the grunt employees, the ones who DO the work, front and center to coming up with ways to do things better. One of the key components of a successful problem solving project is to be open to the ideas of many, rather than be close minded to just your own. To that end, that's merely what I was trying to accomplish in my previous post. Not as a directorate for a way to do it, but more of a suggestion of a possible solution.

Another key part of our projects is identifying the problem, not the goal. We know our goal, but we have to figure out what the root cause of the problem getting there is. From what it sounds like (and I agree up to this point), the problem is the RetroAchievement's website is incapable of supporting the load required to handle a flood of calls for the information. So a possible "countermeasure" is to mirror the data at another site. This is where you're progressing, and I fully support it as well. Now the problem, since that idea has come up, is that you have no way to maintain the sustainability of the mirror. So my suggestion of an off-hours script to work automatically was just an idea to help provide a possible countermeasure for that problem. It may turn out to be bogus, or the solution you're looking for. But please understand, I wasn't trying to tell you how to do it. We're all in this together.

There's a lot of backstory here about this problem solving that I'd love to discuss with people. The problem with it is that the company as a whole LOVES the fact that we're solving some of the BIG problems. But then we get new management staff or corporate level people who are opposed to change. So what inevitably happens is things go back to the old ways. It's pretty bad right now. Things are worse now than they were before. We had a working solution, but the sustainability left us when corporate level decisions were made to go back the way they were. Now the company is suffering a dramatic (we're talking something like 40%) drop in stock value, and they're announcing hour cuts, forcing early retirements, and closing complete stores. This is right on the heals of a new store franchise competitor moving into the area. 2 other major chains in the area have already cut their losses and ran. My company is trying to hold on to what they have, but are doing it the wrong way. Years ago I worked for a store chain called Big Bear. They were the biggest Midwest (USA) grocery chain until another company called Giant Eagle moved into the neighborhood. Out of fear, the owners of Big Bear sold the company to a group of people who ran businesses and not grocery stores. They had no experience in the customer service / point of sale aspect of life. They fired all their corporate staff with forced early retirements, began tearing down stores, and cutting hours in an attempt to gain some financial momentum. It worked for a while and they saw a substantial net gain from their previous year's income. So they spent that money to build a brand new state-of-the art store. The most expensive undertaking the company had ever made. Their goal was to attract the Giant Eagle customers and bring back the old ones. It flopped. The new managers they hired were from a state (in the USA) where the company didn't exit. These managers were businessmen, not trained in customer service, but trained in making money. So wherever they saw fit, they cut hours, increased costs, fired loosely, and hired strictly. I was full time when this happened, the week they scheduled me 12 hours I quit. Weekly sales had gone, and were now monthly sales. These sales were not set by the store but by pencil pushers in Wall Street. The individual stores could set their own prices based on the market and clientele in the area. Our manager was ignorant enough to upcharge something like 20% on everything. The excuse being upkeep for the 21st century store. When in fact, he would get a 35% bonus each week of whatever the store made over the allotted budget. The grunts, including me, were making $9.50 top out pay after 10 years of service. Barely scraping something like $17k a year after taxes. Meanwhile, he was making a SIX digit salary and weekly 4-digit bonuses. The company was originally local to where I live and spread out to 3 other states. In the end, only the local stores remained, a company halfway across the country owned it and operated it, and I was no longer working for them. 6 months after I quit, they declared bankruptcy. NOW.. there's a new threat to the job I'm working at today. I'm seeing the exact same thing happen. A new company is coming in and trying to take over our territory. Forced retirements are happening, the president and CEO of the company has been replaced twice in the last 5 years, hours are cut so deep we can only afford 2 people a day in the department at THE busiest store in the company in the state, and the 3rd busiest store in the state combined with all others (including Walmart, Giant Eagle, Sams Club, Aldi's, Save Alot, Big Lots, etc). We bring in the most money for this company, and they want to cut our hours even more to the point of 1 person a day in the department. We can't do it with 2 and they want 1. These are corporate and management level issues. I'm not even a department head. I'm just a grunt clerk. But I see this, and I have to work these hours by myself with lines of pissed off people wondering where my help is. I don't want to quit, but I'm THIS close... I've been full time for 2 years, have 8 years in the company. I'm making $11.40 an hour. According to my union hand book, after 72 months I should be topped out at $15.25. Hired in 2009.. 2017-2009=8 years... 8*12=95 months.. 95-72=24. 24 months without a raise. I'm 2 years over due for this raise! The same thing happened to me at Big Bear. They still owe me $2.5 grand in back pay that I'll never see because they went bankrupt. I'll be damned if this happens to me again. What can I do? I've talked to the Union rep countless times. "Oh yeah, it'll take 90 days from the time you went full time till you see it" or "I was mistaken, because of the year you were hired you're on a different contract so it'll be 6 months from your full time date." Then it's "oh yeah they won't be implementing any wage increases till after the start of the new fiscal year". Then March comes around and he's "You won't see it until a year after you went full time." Ok, so it's now 2 years and I still don't have it. He's in the store this week and said "Yeah you won't see it till after the day you got hired." So that will be September some time. I'm really pissed about it. He told me 90 days 2 years ago. If you don't know, just say you don't know and I'll ask somebody else. Stop lying hoping it'll just happen and I'll forget about it. So I called the HR department. Nobody answered the phone except a secretary. I explain to her what's going on, she transfers me to the head of payroll. That person's not in, so I leave a message. I leave a message every day for 3 days. No calls. Nobody talks to me at all. She's probably on vacation but I can't help remember what Big Bear did and think they're purposely avoiding me because they can't afford to pay me the back pay and raise. I'd hire a lawyer if I was able to afford one. UUUHG! And you tease about my migraines?

Oh and, sorry.. Didn't realize I was writing a book about my life here lol.

meleu

@hansolo77 WOW!! I think I'll wait Netflix adapt your post to a series. :D

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

I'd hire a lawyer if I was able to afford one.

If only you could afford a ship and find a very hairy mate... You could move to the "logistics" area cough cough smuggling cough cough and deliver all those Chinese gaming gadgets all over the world...

(Hey friend, I'm sorry for not being able to help. I wish you good times.)

hansolo77

It's all good. I didn't even get to say the BEST part, about how this "problem solving" stuff is supposed to save the company, yet it cost them over $1M a year to pay the guys who are training us to solve problems and then they ignore the solutions we present them. It's laughable. :)

So yeah, cheevos... Is there even a system available that you can use to run automatic backups to github like I suggested? I don't know much about internet infrastructure, so the capabilities available are beyond my knowledge. If you can do it manually, by downloading the db's and storing them elsewhere, there's gotta be a way to automate it. I'm just fearful when I hear phrases like:

@meleu said in what about adding a cheevos flag in gamelist.xml ?:

I'll try to update those files once a week, but I cannot guarantee.

meleu

@hansolo77 said in what about adding a cheevos flag in gamelist.xml ?:

Is there even a system available that you can use to run automatic backups to github like I suggested?

Yes, the git command. :-)

Your suggestion about automating the updating process is cool, but unfortunately I don't have a Linux system online 24/7.

To be really honest I thought about you and @grenade44 (or other RetroPie/cheevos users, maybe @Sano ) to help me in keeping the repo up to date. I just need to make my tool user friendly enough and then I'll talk to you guys.

By te way, if someone could kindly provide me a Linux shell account (on a 24/7 online system) for this task, it would be useful for the community. I promess I won't abuse the system resources (everything could be monitored, no need for privacy).

hansolo77

What about a seedbox?... or a router?

meleu

@pjft Just a heads up, I'm working hard on the cheevos scraper side and am having satisfactory results.

The delay was caused by the fact that the hash calculation for MegaDrive, NES and SNES roms isn't always a plain MD5. Then I had to write my own hash calculator. Thanks to the open source philosophy I could have been strongly inspired by RetroArch code.

Just for curiosity, this is my cheevos hash calculator: https://github.com/meleu/share/blob/master/cheevoshash.c

Now I'm working on the bash script to deal with gamelist.xml files. It's working fine right now. But the code still have that prototype looking. :)

I just need to polish the bash script and then we will be ready to scrape our ROM sets tagging the files that has cheevos!

Cheers!

pjft

@meleu Thank you for the update good sir!

I have not had much time recently, unfortunately, but adding the flag, sorting and filtering shouldn't be a big issue. :)

I've been meaning to tidy up the custom collections work before moving on to something else, hence my silence here, but this is committed to on my end. I'll definitely make this happen :)

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :) Well done.

Thanks!

meleu

@pjft said in what about adding a cheevos flag in gamelist.xml ?:

I've just gone through your hash calculator and my only thought is "oh my, I'm sorry you had to go through that on your own" :)

Hey bro, don't feel like that, I loved to code that! And to be honest I don't feel like I did that "on my own". I ~~stole~~ was strongly inspired by RetroArch code. :)

meleu

Progress log

ROM hash calculator for cheevos: DONE! (as I said in a post above)
A script to check if a ROM file has cheevos: DONE! (but there is room for small improvements)

You guys can already test the script's effectiveness! Just follow the instructions in the README of the repository https://github.com/meleu/hascheevos

Next step

write the script that will create/update the <achievements> metadata in the gamelist.xml.

pjft

@meleu Well done good sir!

I'm sure you thought of this, but just in case if I may recommend something, I'd create a backup of the gamelists before actually overwriting them, in case something goes wrong in the new script . :)

sselph

@meleu Cool. Sorry I didn't see this eariler, but looks like you have it all sorted out. I also have hashing code for several rom types. I have a shasum utility here that demonstrates its use:
https://github.com/sselph/scraper/blob/master/shasum/shasum.go#L14
If you need to reference how I handle other systems, the implementation that transforms the bits to something that hashes nicely is here:
https://github.com/sselph/scraper/blob/master/rom/hash/hash.go

But the only other one that is tricky is N64. .Z64 and .N64 have each byte swapped around differently. Atari 7800 and Lynx have optional headers that won't match no-intro hash if they are there but those are easy enough to detect and remove.

meleu

@sselph thanks for the references man! :-)

For RetroAchievements, the only non-plain MD5 hashes are for SNES, MegaDrive and NES ROMS. All other systems (including N64) use plain MD5. Then I think everything is fine. :-)

Cheers!

meleu

I've just made a small improvement on the hascheevos.sh script and now you guys have an easy way to get a list of your ROMs that have cheevos.

From the repo's README:

Create a list of all ROMs that have cheevos in a directory.

The only thing the script puts on stdout are the file name of ROMs that have cheevos. Everything else are printed in stderr. Then if you want a list of all ROMs that have cheevos in a directory, do something like this:

$ ./hascheevos.sh -u USER -p PASSWORD /path/to/megadrive/* > ~/megadrive-roms-with-cheevos.txt

Don't worry about non-ROM files in the same directory (like gamelist.xml or .srm files), the script ignores files with invalid extensions. ;-)

hansolo77

Getting closer to a reality! :)

pjft

Hi @meleu .

Just letting you know that I hope to pick this up shortly.

Best.

meleu

@pjft I'll try to finish the scraper (the part that updates the gamelist.xml) this weekend (if the kids permit)! :-)

pjft

@meleu not a bother.

I might put together a build for that, for you to test anyway. Hopefully that'll help.

If time allows for it, of course:)