Scraper not working

gokuh306

@thestarglider from what I can tell, the database information is correct. I know that they were having server issues as well with heavy load. I added a lot of information to games, it was approved, and the updated game data is showing up, but it's not coming up during scrapes.

muldjord

@gokuh306 It's not the data in the database that's incorrect. It's how they do a filename search match that turns up faulty results.

@thestarglider Could you perhaps run the following command and paste the output for me? That can help me conclude whether the checksums are unique or not:

sha1sum FILENAME.pbp

Just run that command on all your versions of the Tomb Raider pbp files. Then paste it here with the exact filenames and checksums (the output of the sha1sum command).

Btw, if you're having these issues, you can manually look up a correct checksum at ScreenScraper for any rom connected to the game you want, and then scrape the game with:

Skyscraper  -p psx -s screenscraper --query sha1=CHECKSUMFOUNDATSCREESCRAPER FILENAME.pbp

That will force it to register the game as the one linked to that particular checksum.

thestarglider

@muldjord

My original File: 6c76668ebf7dd8290668ba4d81a6d1dbd1facfa8 TombRaider.PBP
Another file: 17291f067313e67f40a9bbdeadb81d27cc397218 TombRaider1.PBP

(the games were actually 'Tomb Raider (USA).PBP' and 'Tomb Raider.PBP', but the command you asked me to use seemed to have a problem with filenames that include spaces).

I deleted the third one I had, and the source for it is no longer available.

Frankly though it's too much effort for me to have to check each of the games that no longer scrape properly and do all you suggest just to get them right. I'd rather just remove any scrape info from the games altogether and leave the default unscraped lists up.

muldjord

@thestarglider No, that's just because you need to escape filenames with spaces. That is a global thing, not just that command. You can do "Filename with space.pgp" or Filename\ with\ space.pgp. Both will work. But thanks, I'll look into them. I'm asking because I want to rule out that this is a Skyscraper issue.

EDIT: Ok, so I checked those two, and my suspicion was correct. The checksums don't exist in their database, so it relies on the filenames entirely. And it does poorly at that which causes the issue of the faulty results. I would have to do further local tests to rule them out. I might do that in a future release.

muldjord

@thestarglider Great news!!! Please update Skyscraper to 3.4.5 and try the Tomb Raider games again. I just had a chat with the ScreenScraper guys and it seems this was in fact an issue, and it has now been fixed in their code (as I suspected it was not a Skyscraper issue, but please update to 3.4.5 anyway)! Scraping Tomb Raider (USA).pbp will now result in the correct game. It should also work for all other games now, unless the filename is not in their database of course. Let me know if you test this out. I've tested it myself and it seems to work. :)

dman

Is screen scraper down? I tried to scrape a few and its not working for me. I scraped a month back and it was fine.

gokuh306

the website seems to be completely down at the moment

muldjord

@dman Yes, their API is entirely down atm.

gokuh306

Website is working, so API might also be working now.

gokuh306

API is working and enabled for non registered users. The built-in scraper should also work now.

gokuh306

It does appear that a lot of filename and CRC data is missing, for now. I have a lot of games that scraped fine before not being found.

dman

still not working for me but it may be something else

thestarglider

@muldjord success! I updated and scraped and all the culprits were found correctly (two weren't but they always were tricky ones to scrape so not an issue).

Many thanks for the heads up!