[Help] Skyscraper 3.0.1 "query" option fails to scrape from screenscraper

jurassicplayer

So I've been bashing my head on this for a little bit now, but first, context.
Setup: RPi 3B+, RetroPie 4.4, Skyscraper 3.0.1, verified dumps with No-Intro daily dats, logged into screenscraper.fr (1 thread), ROMs are .sfc in .7z archives.

Initially scraped with the command: /opt/retropie/supplementary/skyscraper/Skyscraper -p snes -s screenscraper

It failed to scrape 184 out of 723, below are examples taken from ~/.skyscraper/skipped-screenscraper.txt (just the few that I personally tried to scrape again using the query method).
Examples:

'3 Ninjas Kick Back (USA)', No returned matches
'Aero the Acro-Bat (USA)', No returned matches
'Zoop (USA)', No returned matches

Now my understanding of what the "--query" option can do with screenscraper is to ignore the actual file's data (crc, md5, sha1, rom name, rom size, etc) and search screenscraper's database using a provided crc, md5, etc. that is theoretically already in their database. So, I took some checksums from said pages and fed those separately into skyscraper.

Zoop (https://screenscraper.fr/gameinfos.php?plateforme=4&gameid=2816):

06F0E94C
0B7A524E5470ECA3ED29B75E427A7CF4
051C767B843FC4112E0B06C2F81B65ABBAC9E7FC

Commands tested (All three failed):
/opt/retropie/supplementary/skyscraper/Skyscraper -p snes -s screenscraper --query "crc=06F0E94C" "/home/pi/RetroPie/roms/snes/Zoop (USA).7z"
/opt/retropie/supplementary/skyscraper/Skyscraper -p snes -s screenscraper --query "md5=0B7A524E5470ECA3ED29B75E427A7CF4" "/home/pi/RetroPie/roms/snes/Zoop (USA).7z"
/opt/retropie/supplementary/skyscraper/Skyscraper -p snes -s screenscraper --query "sha1=051C767B843FC4112E0B06C2F81B65ABBAC9E7FC" "/home/pi/RetroPie/roms/snes/Zoop (USA).7z"

Then I thought to try a different test and instead I used a ROM which had already been successfully scraped, Kirby Super Star (USA).7z.
Returned the expected result of scraping the information from screenscraper:
/opt/retropie/supplementary/skyscraper/Skyscraper -p snes -s screenscraper "/home/pi/RetroPie/roms/snes/Kirby Super Star (USA).7z"
alt text

Failed to find the ROM info (https://screenscraper.fr/gameinfos.php?plateforme=4&gameid=2216):
/opt/retropie/supplementary/skyscraper/Skyscraper -p snes -s screenscraper --query "md5=CB76EA8AC989E71210C89102D91C6C57" "/home/pi/RetroPie/roms/snes/Kirby Super Star (USA).7z"
alt text

If there is any other information that I should provide, or maybe if there is just some major misunderstanding on my part, I'd be glad to get some sort of enlightenment of what I'm seeing.

muldjord

There seems to be an issue with the Screenscraper database at the moment. No checksum only searches seem to come back with results. This used to work just fine. So let's give it a week and see if it they are working on fixing it.

mitu

@muldjord I had the opposite experience the other day (searches by name would yield an error and return nothing or return a random game). This was fixed yesterday, but I didn't try using a checksum only search - maybe that's failing too.

muldjord

@mitu I think they had similar issues about a year ago, where a faulty entry would be returned for many queries. Let's hope it settles soon. :)

muldjord

@jurassicplayer Just to follow up on this. This has now been fixed in the 3.0.3 version of Skyscraper. It was caused by a change in requirements in their APIv2 and slightly invalid XML being returned from Screenscraper for certain games. I've worked around the latter and added the required changed in 3.0.3. Please test.

jurassicplayer

@muldjord I went through and rescraped all of my games that were originally skipped with the latest 3.0.3 version. It was able to find all of the games that it previously skipped without intervention except for one particular game.

M&M's - Break 'em (https://screenscraper.fr/gameinfos.php?gameid=84087&action=onglet&zone=gameinfosroms)

A4EA65C0
8C5C04D6559A700903C7186E0BAEE2FE
5DC56DB613A0A27B7724CBE8889FD2BA3A0993D9

Aside from that particular game, Skyscraper appears to be able to scrape everything else that I have perfectly (provided screenscraper.fr has the entry, which seems to be missing a number of megadrive games, but that is unrelated).

Your work is much appreciated. Thanks.

muldjord

@jurassicplayer Tested with that md5 sum and it is caused by the same flaw in the returned XML as I just fixed. My fix doesn't work on this but I've just improved it so it should work all-round from 3.0.4.