Versatile C++ game scraper: Skyscraper
-
No worries, I thought there was some confusion, but thanks for the heads up, and as always, thanks for helping.
-
Perfect, thanks for adding .cue to the list of files.
No worries about the region functionality, if that won't change in the foreseeable future, I could always pop the EU games in another directory, scrape as US, move the EU games back, then scrape again as EU but use the switch to skip the previously scraped files. That should work...
I wonder how Skraper does it, perhaps it looks at the filename and makes a judgement. I have no idea but it does work quite well.
Cheers for all of the help so far.
-
@movisman Can you give me examples of your filenames? I've decided to implement the region override (after a bit of thinking I realized this was a pretty easy thing to implement). But I need filename examples of how they are designated. For instance, are they called "Gamename (Europe).zip"? or maybe "Gamename (eu).zip"?
EDIT: The implementation is done now. I've implemented it so it looks for "europe", "usa", "japan", and "world" in the filename, but only if it comes after the first "(" and the user hasn't manually set the "--region" flag. It will then add that region to the top of the priority list automatically.
EDIT2: @movisman Skyscraper 2.8.2 now tested and released, please update and test out the region detection. You should get the artwork from the region that's in the filename if they are from the above mentioned regions.
-
Skyscraper version 2.8.2 released: https://github.com/muldjord/skyscraper
- Now autodetects region from filename and prioritizes it when scraping with "screenscraper" unless user has forced region with "--region" option (thank you to "movisman" for suggesting this)
- Fixed issue where "wheel" artwork would not be fetched for specified region even though it exists
- Added ".cue" file extension to Amiga
-
Amazing!! Thank you for implementing this!
I have been out this evening, but will give it a go shortly and let you know.You are correct, most of the files are (Europe), (USA), (Japan), and (World). Some are a combination, eg. (US, Europe) - I guess in this instance it should scrape the US one.
Will let you know shortly how it goes. I'll test on one of the folders which has these brackets on every file (megadrive + snes).
Cheers
-
@movisman For the ones with (Europe, USA) or similar, it will look for the first one first, then the second one and so forth. So in this examples it would look for 'eu' first. And if that isn't found, it will look for 'us'. And if that isn't found, it will go back to it's original prioritization.
-
Just tried Megadrive, and the good news is it works way better, it is picking up the region as you specify above.
However, something I noticed previously which seems to be the same here, is some (Europe) games scraped as EU have the US marquee for some reason. Three examples:
Another World (Europe) - shows the 'Out Of This World' logo, which is US
Thunder Force IV (Europe) - shows the 'Lightening Force' logo, which is US
Chaos Engine, The (Europe) - shows the 'Soldiers of Fortune' logo, which is USThe box is the European type, so that's good - it's just the marquee which displays incorrectly.
Does this need the same treatment with the region autodetect you implemented?
I don't believe this is a screenscraper.fr issue, because if I scrape these with skraper, the marquees are all detected as the correct EU ones. Also if I do a manual lookup, say on Another World, you can see the 'wheel' lists the Europe and US logo correctly, as does marquees etc.
Thanks!
-
@movisman Probably because they are cached and therefore not rescraped. Try scraping with '--refresh'.
-
Yep I was just editing my post to say I have already refreshed with --refresh.
-
@movisman The method I've implemented concerns all data that has a region, so it should work for all types.
I'll need to look into this. "Another World" is probably a good examples to test it with, so I'll try that.
-
Command used for testing:
/opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/megadrive -o /home/pi/RetroPie/roms/megadrive/media --relative -p megadrive -s screenscraper -u username:password --videos
username:password was filled out correctly of course.
Cheers
-
@movisman Just tested with "Another World (Europe)" for snes. It works as expected. I don't get the "Out of this world" wheel. And if I force us region it also correctly shows the "Out of this world" resources. I'm not sure what could be causing your issue. Have you perhaps scraped the platform with other sources aswell? It might use the artwork from cache from other sources.
EDIT: It seems like SS is down at the moment? I can't scrape anything from it right now. It worked just a few minutes ago.
EDIT2: It's working again it seems. Odd.
-
Hi,
No I have only scraped against screenscraper.fr so far, as it has pretty much everything I need, and if it's missing a video I obtain one and submit it. Maybe I could clear the db folder out for megadrive.
I was working with the Megadrive version of Another World, perhaps that version is worth a try?
Or any of the above three examples maybe?
Cheers
-
Just tried to purge the DB for megadrive within the GUI, and I ran the command above again, but the same occurs. Correct boxart but the US marquee/wheel on those three files I mentioned, but they have an (Europe) filename and european boxart.
This was happening before btw when I was playing around with the region switch. Not sure this happens for all roms, but I am not sure what the differentiator is.
-
@movisman Just tested it with "Another World (Europe)" for Megadrive, and you're right, it grabs the "Out of this world" marquee for some reason. I'll look into it.
-
Nice one. I'm glad you can replicate it! I would need to check, but I think it happens on other systems too. I can probably get you more files to test against if required.
Cheers
-
@movisman I found the reason. I look for the "wheel" and if that isn't found I look for the "wheel-hd" types when scraping the wheel, and for "Another World" there only is a us version of that. The "eu" wheel shows up on the website, but not in the data I get when searching for the game. I'll need to figure out why that is. Perhaps it's hiding behind some other type.
Ok, so this is the reason: For those games they have the "wheel" artwork with "us" region. But it also has the "wheel-hd" artwork with "eu" region.
EDIT: Ok, so I've fixed this now. I've made it so it doesn't just look for "wheel" but every type that has "wheel" in it. So it'll also look through the "wheel-hd" and other wheel types. I've tested this and it works. I just need to make sure it doesn't break any of the other types.
EDIT2: There, fixed! I decided to just re-release 2.8.2 with the fix, so please update again. If you are updating using the "update_skyscraper.sh" script (if you installed it using the RetroPie-Setup you aren't) you'll have to do "rm /home/pi/skysource/VERSION" before updating, otherwise it'll just think you already have the latest version. Also, thank you @movisman for letting me know about this issue. I would never have noticed myself. I would just have assumed it didn't have the wheel for the region.
-
Hi,
Ah, ok just getting my head around it - so what you're saying is because you look for 'wheel' first, it is grabbing the US version because no EU version exists for that? But ignoring the fact that an HD wheel is available for EU?
Is there a way to search both wheel and wheel HD first before going to a fallback?
I also noticed, with Thunder Force, there is only a World and US wheel. No Europe. But World should be the same as Europe. But what must be happening there, is because my rom is (Europe), it looks for that wheel, cannot locate it, so it then goes for the standard priority list for skyscraper, which I believe has US as the next one. I'd have to change the priorities so that 'world' is ahead of 'europe'. This means it would get the correct wheel I guess. Or what i'll likely do is submit the correct logo with 'Europe' as the region.
-
@movisman said in Versatile C++ game scraper: Skyscraper:
Is there a way to search both wheel and wheel HD first before going to a fallback?
Yes, this is exactly what I just changed.
-
Cool! I will give it a try and see how it works. Sorry I didn't see your edit above already explaining that.
Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.
The reason I was thinking that, an example:
GameX (Europe).zipWants European boxart and logo/game title, but it isn't available on screenscraper.fr. However World is there, and US is there. World seems to me, to be more likely to have the same boxart and/or title as the European release. However on default priority, if EU cannot be found, it will go for the US release, which is more likely to have a different name, and definitely a different box.
The same goes if you have a (USA) release, if this isn't available but World and Europe are, it could go for World, and then Europe after that.
Does that make sense? I might be missing something to suggest this isn't a good idea, but I think that would possibly work better?
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.