Versatile C++ game scraper: Skyscraper

movisman

Command used for testing:

/opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/megadrive -o /home/pi/RetroPie/roms/megadrive/media --relative -p megadrive -s screenscraper -u username:password --videos

username:password was filled out correctly of course.

Cheers

muldjord

@movisman Just tested with "Another World (Europe)" for snes. It works as expected. I don't get the "Out of this world" wheel. And if I force us region it also correctly shows the "Out of this world" resources. I'm not sure what could be causing your issue. Have you perhaps scraped the platform with other sources aswell? It might use the artwork from cache from other sources.

EDIT: It seems like SS is down at the moment? I can't scrape anything from it right now. It worked just a few minutes ago.

EDIT2: It's working again it seems. Odd.

movisman

@muldjord

Hi,

No I have only scraped against screenscraper.fr so far, as it has pretty much everything I need, and if it's missing a video I obtain one and submit it. Maybe I could clear the db folder out for megadrive.

I was working with the Megadrive version of Another World, perhaps that version is worth a try?

Or any of the above three examples maybe?

Cheers

movisman

@muldjord

Just tried to purge the DB for megadrive within the GUI, and I ran the command above again, but the same occurs. Correct boxart but the US marquee/wheel on those three files I mentioned, but they have an (Europe) filename and european boxart.

This was happening before btw when I was playing around with the region switch. Not sure this happens for all roms, but I am not sure what the differentiator is.

muldjord

@movisman Just tested it with "Another World (Europe)" for Megadrive, and you're right, it grabs the "Out of this world" marquee for some reason. I'll look into it.

movisman

@muldjord

Nice one. I'm glad you can replicate it! I would need to check, but I think it happens on other systems too. I can probably get you more files to test against if required.

Cheers

muldjord

@movisman I found the reason. I look for the "wheel" and if that isn't found I look for the "wheel-hd" types when scraping the wheel, and for "Another World" there only is a us version of that. The "eu" wheel shows up on the website, but not in the data I get when searching for the game. I'll need to figure out why that is. Perhaps it's hiding behind some other type.

Ok, so this is the reason: For those games they have the "wheel" artwork with "us" region. But it also has the "wheel-hd" artwork with "eu" region.

EDIT: Ok, so I've fixed this now. I've made it so it doesn't just look for "wheel" but every type that has "wheel" in it. So it'll also look through the "wheel-hd" and other wheel types. I've tested this and it works. I just need to make sure it doesn't break any of the other types.

EDIT2: There, fixed! I decided to just re-release 2.8.2 with the fix, so please update again. If you are updating using the "update_skyscraper.sh" script (if you installed it using the RetroPie-Setup you aren't) you'll have to do "rm /home/pi/skysource/VERSION" before updating, otherwise it'll just think you already have the latest version. Also, thank you @movisman for letting me know about this issue. I would never have noticed myself. I would just have assumed it didn't have the wheel for the region.

movisman

@muldjord

Hi,

Ah, ok just getting my head around it - so what you're saying is because you look for 'wheel' first, it is grabbing the US version because no EU version exists for that? But ignoring the fact that an HD wheel is available for EU?

Is there a way to search both wheel and wheel HD first before going to a fallback?

I also noticed, with Thunder Force, there is only a World and US wheel. No Europe. But World should be the same as Europe. But what must be happening there, is because my rom is (Europe), it looks for that wheel, cannot locate it, so it then goes for the standard priority list for skyscraper, which I believe has US as the next one. I'd have to change the priorities so that 'world' is ahead of 'europe'. This means it would get the correct wheel I guess. Or what i'll likely do is submit the correct logo with 'Europe' as the region.

muldjord

@movisman said in Versatile C++ game scraper: Skyscraper:

Is there a way to search both wheel and wheel HD first before going to a fallback?

Yes, this is exactly what I just changed.

movisman

@muldjord

Cool! I will give it a try and see how it works. Sorry I didn't see your edit above already explaining that.

Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.

The reason I was thinking that, an example:
GameX (Europe).zip

Wants European boxart and logo/game title, but it isn't available on screenscraper.fr. However World is there, and US is there. World seems to me, to be more likely to have the same boxart and/or title as the European release. However on default priority, if EU cannot be found, it will go for the US release, which is more likely to have a different name, and definitely a different box.

The same goes if you have a (USA) release, if this isn't available but World and Europe are, it could go for World, and then Europe after that.

Does that make sense? I might be missing something to suggest this isn't a good idea, but I think that would possibly work better?

movisman

@muldjord

Separate question, can I scrape a single rom when using the --localdb switch? For some reason, that doesn't work for me. Running a command on a single rom, for example:

/opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/megadrive -o /home/pi/RetroPie/roms/megadrive/media --relative -p megadrive -s screenscraper -u username:password --videos "insertromnameandpathhere.zip"

Will successfully scrape a single rom to the localdb.

However, when I then run the same command as above but with the localdb as source to pull the rom into the gamelist, it doesn't do it. I believe the same happens when using the query function too.

If I then remove the single rom+quotes at the end of the above command to pull the entire localdb for that system into the gamelist.xml, it will bring the rom in successfully, but unfortunately at the expense of any manual edits I made to the other games (it will overwrite any minor changes you made).

Any idea?

Thanks

muldjord

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord
Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.
I did this originally, but it turned out many world entries have the japanese versions, so I changed it.

muldjord

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord

Separate question, can I scrape a single rom when using the --localdb switch? For some reason, that doesn't work for me. Running a command on a single rom, for example:

That is not possible and won't be possible. It would require me to inject only selected entries into the gamelist which is something I don't want to fiddle with.

movisman

@muldjord said in Versatile C++ game scraper: Skyscraper:

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord
Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.
I did this originally, but it turned out many world entries have the japanese versions, so I changed it.

Ah ok, I see. That's fair enough.

muldjord

@movisman Let me elaborate a bit. I'm not saying it won't ever be able to do that (actually that is what I wrote but whatever), but it's a rather big code rewrite I would have to do for that functionality. And personally I wouldn't ever use it. I change the data in my localdb instead and rescrape. Much better imo.

movisman

@muldjord said in Versatile C++ game scraper: Skyscraper:

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord

Separate question, can I scrape a single rom when using the --localdb switch? For some reason, that doesn't work for me. Running a command on a single rom, for example:

That is not possible and won't be possible. It would require me to inject only selected entries into the gamelist which is something I don't want to fiddle with.

Cool, that's fine, I just wanted to check if it was possible or if I was doing something wrong. If I want to scrape a single game but not have it fiddle with the existing gamelist entries, is my only option at the moment either to edit the db.xml file for that system, or scrape the file to a fresh gamelist.xml and merge it into my existing one?

Thanks!

muldjord

@movisman Second option is your best option as you've already edited the gamelist manually. The first one won't work. If you've already manually edited your gamelist, it won't help to change the data in your localdb files. It will still overwrite the things you've changed upon a rescrape.

movisman

@muldjord

Ah no yeah I understood that, I haven't made loads of edits to the point where I couldn't rescrape. To be honest since you've improved the region stuff it's going to be quicker for me to rescrape anyway to pick up the correct artwork, so I could rescrape, edit db.xml after and then scrape from the DB.

movisman

@muldjord

Hiya,

Since I updated this morning, for some reason queries against SH1, CRC, MD5 are not working? It just comes back with 'game xxx not found' on pass 1. Adding the maximum verbosity on the command line shows the correct SHA1 and platform, but it never finds it. I am using literally exactly the same command as I was using previously to obtain a correct scrape for mismatched files. Tried a variety of valid checksums off screenscraper.fr.

Running without query is fine though.

Do you know what might be happening?

Thanks!

movisman

FYI,

An example command used was:
/opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/dreamcast -o /home/pi/RetroPie/roms/dreamcast/media --relative -p dreamcast -s screenscraper -u username:password --videos --nobrackets --query sha1=058b2eda664ee953caeb5ad35efbbdbcfc60d85f "/RetroPie/roms/dreamcast/Crazy Taxi (USA).chd" --verbosity 3

It made a start, then bombs out:

Starting scraping run on 1 files using 3 threads.
Sit back, relax and let me do the work! :)

#1/1 (T1) Pass 1 ---- Game 'Crazy Taxi (USA)' not found :( ----

Debug output:
Tried with: 'sha1=058b2eda664ee953caeb5ad35efbbdbcfc60d85f'
Platform: dreamcast

0/1/1

This wasn't happening before, but thought i'd share the command I used. I tried a few others, changed my file extensions around etc, but the same result.

Thanks