Scraping totally fails

beppi

Good day.
I decided to scrape some new rom, I have a Raspberry Pi 3B+ with retropie / emulationstation installed.
Everything has always been fine and I have a lot of roms that have been correctly scraped in the past. I work with Skyscraper, that I now updated to the latest 3.7.7 version.
Now, when I tried to scrape again, after some months of not doing it, I continuously got the same result: game not found.
The request succeeds but no data is retrieved.

I tried everything but no way...:

I tried to scrape different systems, both rare and common, no game found.
I tried to use both Skyscraper and the default scraper from inside emulationstation... no game found.
I tried both tgdb and screenscraper... no game found.
I tried to force scraping of a rom that I know exists on both websites and that I successfully scraped in the past, and I even tried to change its name to match letter-by-letter the correct name on the site, so there's no real reason for the message... no game found.

ALL these commands give me the same frustrating error: game not found. I could scrape not even one single game, on any platform and from any site.

I'm pretty sure that there are no errors on this side, I use scripts that have always worked... now I cannot do it anymore, is there something I've been missing in these months? Did those services shutdown? Did some API change so that the request fails in some way?

I know that the problem seems stupid, but I'm really stuck, I've tried anything, thank you if you would like to give me some help.

Addon: network works fine and I used free registered users on both tgdb and screenscraper

max

Not sure if it's the same / related: couple of days ago I scraped my games (I just set up everything from scratch, only had a couple of games) and it would only find a few ones out of 20; e.g. none of the ScummVM games were found.

Fast forward, yesterday I re-ran it and found almost all of them.

beppi

@max
Thank you Max, I hope that this is what you said, a temporary problem.
It's really strange that every single rom searched on every single site fails, even if they are present when searched manually on the web site.
A question: as you know does exist an alternative way, instead of scraping some site, to get those mp4s, jpgs and infos? some torrent maybe, to fill up the cache without pulling the data from those central sites?

WhiteT982

@beppi
If you want to do it manually I’ve used emumovies. There’s also ARRM which is another scraper you could try. Not sure if it uses sites other than tgb or screenscraper though.

nexusone13

Hi @WhiteT982 ,
ARRM can scrape on this sites : Screenscraper, HFS-DB, TheGamesDb.net, LaunchBox, Steam, IGDB , GOG , MobyGames and GameTDB

beppi

Thanks all folks, I'm really getting crazy.
I tried to scrape using skraper, and it works. Only problem... it runs on Windows and I need it running on the raspberry. The only ption I see now is to scrape on Windows, create a script to change manually all the paths and informations that are missing and move everything manually, a huge work!
Furthermore, also other scrapers fail and I'm really styuck in a problem that looks to me as a sci-fi movie... I made a lot of configurations on my raspi and I consider myself able to dive in deep in linux configurations using my beloved vi, but I fail here.
Anything else I tried, unfortunately, failed, and now I'm trying to reach the author of Skyscraper on github, but I doubt I will be able to fix it.

Quackwalks

@beppi What I do Is back up my roms directory into a windows folder. In Skraper, you can change the settings to use the windows directory and it'll still output game lists that work for retropie. I just put my game lists in my rom directories, then a folder in each system for the snaps and box arts. After you're done scraping, transfer your game lists and scraped media to your raspberry pi where they should go.

Sorry I can't help with the details, but I know this method works with enough tinkering.

beppi

@Quackwalks this is indeed the only solution i found so far, but it's not a complete solution because some information is missing, for example the information about the location of videos and screenshots, and Skyscraper was also creating a composition with the box for the screenshot that was integrated with the theme. Of course it's not a big deal but I would like to understand why Skyscraper cannot connect from raspi.
I feel that I could fix this problem if I only had some kind of log, monitor, or anything else, do you know if there is a log somewhere? I could not find it.

YFZdude

@beppi
There should be logs anytime something fails to scrape with skyscraper in the /home/pi/.skyscraper folder.

Crush

@Quackwalks said in Scraping totally fails:

@beppi What I do Is back up my roms directory into a windows folder. In Skraper, you can change the settings to use the windows directory and it'll still output game lists that work for retropie. I just put my game lists in my rom directories, then a folder in each system for the snaps and box arts. After you're done scraping, transfer your game lists and scraped media to your raspberry pi where they should go.

Sorry I can't help with the details, but I know this method works with enough tinkering.

Just make a network drive on windows out of \\retropie\roms so you can scrape using screenscraper on windows directly instead of having to transfer back and forth.

beppi

@YFZdude thank you, unfortunately there's nothing, because there's no error in Skyscraper: it just receives the information "not found", appearantly. I watched by capturing in real time written files both on .skyscraper directory, and on /var/log, and no log is being generated.

beppi

Thanks all for answering me, unfortunately there's no solution right now.
I make you just one question: are you able to scrape with Skyscraper from inside raspberry, on screenscraper.fr or do you get the same error "game not found"? thank you

kiro

@beppi trybtjis, it may work...https://github.com/zayamatias/retroscraper

Floob

@beppi
FWIW I use Skraper and scrape on windows, then just copy across the files.
I do that so I can copy it across to any destination quickly - so you only need to scrape once really.
The paths are all relative, so you dont need to change them

For example:
<image>./media/boxart2d/3-D Tic-Tac-Toe (USA).png</image> <marquee>./media/wheel/3-D Tic-Tac-Toe (USA).png</marquee> <video>./media/videos/3-D Tic-Tac-Toe (USA).mp4</video>

So in Skraper, the video path would be:
%ROMROOTFOLDER%\media\videos

Same as @Quackwalks really.

beppi

@Floob i fixed the problem. On retropie it still fails, but I mounted a disk in a Linux distribution and using the bind option I recreated the needed directory tree and run Skyscraper from there. It works, so it's obvious that there is a problem on my raspberry but I don't have any tool to address it. Thank you

kiro

@beppi could it be a permissions issue in your rpi? Pi user does not have write rights on the mount ?

Quackwalks

@Crush That is also an option, but sometimes I'd rather get everything scraped on a separate hard drive. Just makes me feel more secure in various ways. I'm probably just silly.