Scraping totally fails

max

Not sure if it's the same / related: couple of days ago I scraped my games (I just set up everything from scratch, only had a couple of games) and it would only find a few ones out of 20; e.g. none of the ScummVM games were found.

Fast forward, yesterday I re-ran it and found almost all of them.

beppi

@max
Thank you Max, I hope that this is what you said, a temporary problem.
It's really strange that every single rom searched on every single site fails, even if they are present when searched manually on the web site.
A question: as you know does exist an alternative way, instead of scraping some site, to get those mp4s, jpgs and infos? some torrent maybe, to fill up the cache without pulling the data from those central sites?

WhiteT982

@beppi
If you want to do it manually I’ve used emumovies. There’s also ARRM which is another scraper you could try. Not sure if it uses sites other than tgb or screenscraper though.

nexusone13

Hi @WhiteT982 ,
ARRM can scrape on this sites : Screenscraper, HFS-DB, TheGamesDb.net, LaunchBox, Steam, IGDB , GOG , MobyGames and GameTDB

beppi

Thanks all folks, I'm really getting crazy.
I tried to scrape using skraper, and it works. Only problem... it runs on Windows and I need it running on the raspberry. The only ption I see now is to scrape on Windows, create a script to change manually all the paths and informations that are missing and move everything manually, a huge work!
Furthermore, also other scrapers fail and I'm really styuck in a problem that looks to me as a sci-fi movie... I made a lot of configurations on my raspi and I consider myself able to dive in deep in linux configurations using my beloved vi, but I fail here.
Anything else I tried, unfortunately, failed, and now I'm trying to reach the author of Skyscraper on github, but I doubt I will be able to fix it.

Quackwalks

@beppi What I do Is back up my roms directory into a windows folder. In Skraper, you can change the settings to use the windows directory and it'll still output game lists that work for retropie. I just put my game lists in my rom directories, then a folder in each system for the snaps and box arts. After you're done scraping, transfer your game lists and scraped media to your raspberry pi where they should go.

Sorry I can't help with the details, but I know this method works with enough tinkering.

beppi

@Quackwalks this is indeed the only solution i found so far, but it's not a complete solution because some information is missing, for example the information about the location of videos and screenshots, and Skyscraper was also creating a composition with the box for the screenshot that was integrated with the theme. Of course it's not a big deal but I would like to understand why Skyscraper cannot connect from raspi.
I feel that I could fix this problem if I only had some kind of log, monitor, or anything else, do you know if there is a log somewhere? I could not find it.

YFZdude

@beppi
There should be logs anytime something fails to scrape with skyscraper in the /home/pi/.skyscraper folder.

Crush

@Quackwalks said in Scraping totally fails:

@beppi What I do Is back up my roms directory into a windows folder. In Skraper, you can change the settings to use the windows directory and it'll still output game lists that work for retropie. I just put my game lists in my rom directories, then a folder in each system for the snaps and box arts. After you're done scraping, transfer your game lists and scraped media to your raspberry pi where they should go.

Sorry I can't help with the details, but I know this method works with enough tinkering.

Just make a network drive on windows out of \\retropie\roms so you can scrape using screenscraper on windows directly instead of having to transfer back and forth.

beppi

@YFZdude thank you, unfortunately there's nothing, because there's no error in Skyscraper: it just receives the information "not found", appearantly. I watched by capturing in real time written files both on .skyscraper directory, and on /var/log, and no log is being generated.

beppi

Thanks all for answering me, unfortunately there's no solution right now.
I make you just one question: are you able to scrape with Skyscraper from inside raspberry, on screenscraper.fr or do you get the same error "game not found"? thank you

kiro

@beppi trybtjis, it may work...https://github.com/zayamatias/retroscraper

Floob

@beppi
FWIW I use Skraper and scrape on windows, then just copy across the files.
I do that so I can copy it across to any destination quickly - so you only need to scrape once really.
The paths are all relative, so you dont need to change them

For example:
<image>./media/boxart2d/3-D Tic-Tac-Toe (USA).png</image> <marquee>./media/wheel/3-D Tic-Tac-Toe (USA).png</marquee> <video>./media/videos/3-D Tic-Tac-Toe (USA).mp4</video>

So in Skraper, the video path would be:
%ROMROOTFOLDER%\media\videos

Same as @Quackwalks really.

beppi

@Floob i fixed the problem. On retropie it still fails, but I mounted a disk in a Linux distribution and using the bind option I recreated the needed directory tree and run Skyscraper from there. It works, so it's obvious that there is a problem on my raspberry but I don't have any tool to address it. Thank you

kiro

@beppi could it be a permissions issue in your rpi? Pi user does not have write rights on the mount ?

Quackwalks

@Crush That is also an option, but sometimes I'd rather get everything scraped on a separate hard drive. Just makes me feel more secure in various ways. I'm probably just silly.