Versatile C++ game scraper: Skyscraper

quicksilver

@muldjord I scraped some PSP ROMs about 6 hours ago and didn't have any issues. So if there is a problem it just started recently.

muldjord

Thanks, yeah, I just tried it again and now it seems to be working... Oh well. :)

EDIT: Asked on their Discord and turned out it was a bug in the API that they've now fixed.

muldjord

Skyscraper 3.4.1 released: https://github.com/muldjord/skyscraper

Further optimized artwork space requirements. Now checks if original takes up less space than resized artwork, then forces use of original for those cases
The 'thegamesdb' module now also supports wheel and marquee for the games that have them (Thank you to 'tv21' for pointing this out)
Updated developer and publisher json list for 'thegamesdb'

thegamesdb now supports retrieving wheel and marquee artwork resource types. And I optimized the artwork resource export pipeline a bit further. If you scraped with 3.4.0 there's no need to redo it, it's only a small difference.

A Former User

Any chance you can updated the Windows version to 3.4.1? Unless you have an update in the pipeline.

muldjord

@LiveFastCyYoung Sure, I've updated it to the latest 3.4.2 now. Get it here: http://www.muldjord.com/downloads/Skyscraper_3.4.2_unsupported_win_version.zip

A Former User

@muldjord Much appreciated!

muldjord

Skyscraper 3.4.3 released: https://github.com/muldjord/skyscraper

Implemented workaround to incorrectly formatted JSON returned from 'screenscraper' when checking user credentials
All arcade platforms now use 'flyer' from 'screenscraper' for cover artwork instead of 'box-2D'

There have been issues logging in to screenscraper lately. This stems from a bug in their code which formats the returned JSON incorrectly after checking the credentials. This leads to a parse error in the Skyscraper JSON parser which then lead to a failed login. I've implemented a workaround that fixes the JSON before parsing it. I have reported the bug to them, I hope they fix it soon so I can remove this workaround again.

User @aidy80-s suggested I use the flyer artwork from screenscraperfor the arcade platforms. This was a brilliant idea, so this has now been implemented as well.

A Former User

Have you ever looked into adding the LaunchBox DB as a scraping source?

https://gamesdb.launchbox-app.com/

muldjord

@LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

Have you ever looked into adding the LaunchBox DB as a scraping source?

https://gamesdb.launchbox-app.com/

Yes, several times actually. I can't remember exactly why, but as I recall it is not an open API like other sources use.

quicksilver

Certain files get scraped as ZZZnotagame instead of just ignored (especially +StartDOSbox.sh) Its really annoying to have to manually rename these after scraping. Is there anyway to avoid this?

muldjord

@quicksilver Hmm, I already have a filter to avoid these, but maybe some of them goes under the radar. Can you provide a specific filename for an entry that returns this?

quicksilver

@muldjord Ok cool, if that is the case let me try doing a cache refresh and then rescrape first to see if its fixed now and I'll report back.

quicksilver

Refreshing the cache and rescraping took care of the issue. Thanks for the quick response.

Edit: Just as a side note to anyone else having the same issue. You'll need to purge your cache for that specific system otherwise when you rescrape it will grab the ZZZnotagame info from your cache and cause the problem all over again.

AlCzervik

@muldjord I cannot figure it out. I don't have the credentials commented out, they are properly written, yet I continue to get a "Received invalid ScreenScraper server response, maybe their server is having issues, forcing 1 thread..." and then nothing but empty Json scrapes. Please help! This is with version 3.4.3 BTW.

muldjord

@AlCzervik It works fine, just tested it. Maybe their servers were having issues (they are having a lot of issues in general because of so many people scraping at the moment), please try again.

AlCzervik

@muldjord I'll take another look but are there other things I can check that could cause that issue? Is there a way I can test the API?

muldjord

@AlCzervik Hmm, actually that response is also shown if your credentials are wrong. It needs to look exactly like this in the config.ini:

[screenscraper]
userCreds="user:pass"

But with your own user and pass of course. If you've entered that correctly and don't have any stray configs messing with that, it will work.

AlCzervik

@muldjord That's helpful! Quotes required? I know use case is important on the variable, is it important with the username/password? Perhaps I should inquire with screenscraper directly to ensure there's nothing wrong with my account? I only recently got to the point where I can download data, perhaps that's the issue.

muldjord

@AlCzervik It could be. The service is currently closed for some users. I'm not sure what tiers are closed exactly.

I don't think the quotes are necessary but I know it works with them, so I would use them. But if you get that error at the beginning it either means the server is down or you entered the credentials incorrectly. I'm guessing the latter as many people seem to get it wrong.

AlCzervik

@muldjord Thanks for your help. I'll have to start with them if that response is related to credentials. And I've checked and rechecked that it is spelled correctly in the config.ini file. Not sure what else to try at this point. Thanks again.