Scraper not working when attempting to scrape games for descriptions and boxarts.

Emulatre71

Raspberry Pi 4
RetroPie Version: 4.7.11

So when attempting to scrape my collection it behaves as if scraper is unavailable or offline for both resource databases. I've made a few attempts at a later time and still nothing worthwhile, I tried to also do some of them one by one and that did not work.

After various attempts it did a few but not everything.

mitu

Assuming you're using the EmulationStation built-in scraper, the results will depend on the online site (from where the scraping is done) being available.
The built-in scraper can scrape from ScreenScraper.fr or ThegamesDb.net - which one did you use ? ScreenScraper is often overloaded and scraping will not work at the time - you can check on their site.
You can switch to the other scraping site (TheGamesDB.net) and see if you get better results, if there are games that don't have info then specify the game and platform.

Emulatre71

@mitu I used both sources from EmulationStation, I just could not remember the names of those 2 sources. Now that I know the sites I punched it in one of those site down detectors. It says both are not down. I'll just try again with both later. I was using the scraper for multiple systems such as Dreamcast, PS1, SNES, GBA, GBC, GameBoy.

TheGamesDB.net
https://www.isitdownrightnow.com/thegamesdb.net.html

ScreenScraper
https://www.isitdownrightnow.com/screenscraper.fr.html

mitu

@emulatre71 said in Scraper not working when attempting to scrape games for descriptions and boxarts.:

It says both are not down.

That's not what I said - if you look now on ScreenScraper.fr you'll see they're overloaded and requests from un-registered users are not working

Emulatre71

@mitu I tried this morning, not able to get them from TGDB.net either, is it possible that they are overloaded from traffic since ScreenScraper is overloaded still? Is it possible to edit RetroPie and add a third source, is there even a third source that will work with RetroPie. Would it work if you can try to borrow sources from libretro thumbnail database?
https://github.com/libretro-thumbnails/libretro-thumbnails

mitu

@emulatre71 said in Scraper not working when attempting to scrape games for descriptions and boxarts.:

Would it work if you can try to borrow sources from libretro thumbnail database?

No, the scrapers in EmulationStation only support 2 sources. You may use an external scraper - see the docs for details.

.. not able to get them from TGDB.net either...

I scraped from TGDB today and it worked fine for me. Even now, I had no problem scraping from it.

EDIT: one thing to note is that both sites have quotas (TGDB.net has a monthly quota and ScreenScraper a daily one). If you previously scraped from them and gone over quota, then you'll not be able to scrape from them.

Emulatre71

@mitu So if I ever needed to do a clean install of RetroPie, Would I be able to copy and paste the downloaded_image folder into its proper file path and RP detect them. There's a local art option but I would some info on how to use it. I've tried doing some thumbnails manually, its time consuming compared to Lakka OS. Would I be able to automate some stuff, the scraper has not been working well for me.

As for a thumbnail quota being meet, would I be able to get past that with IP address changing and mac address changing? How are mac address handled, do they change on RP or are they static?

Edit: When I went to retropie settings I saw this Screen scraper error show up.

mitu

@mitu So if I ever needed to do a clean install of RetroPie, Would I be able to copy and paste the downloaded_image folder into its proper file path and RP detect them.

You can copy the config folder to a new install and the gamelists and artwork would be transferred. Everything is stored under config/all/emulationstation.

There's a local art option but I would some info on how to use it.

The local art option makes EmulationStation look under the images sub-folder for image files with the <romname>-image and <romname>-video.mp4 name and show them automatically for image/video.

I've tried doing some thumbnails manually, its time consuming compared to Lakka OS. Would I be able to automate some stuff, the scraper has not been working well for me.

You can try one of the other scrapers. Skyscraper has more sources available than the built-in scraper and also user/pass support for Screenscraper (among other features). It can also import your existing images, as long as they're named properly.

As for a thumbnail quota being meet, would I be able to get past that with IP address changing and mac address changing? How are mac address handled, do they change on RP or are they static?

That's not how it works, changing your MAC address does not change your IP.

Edit: When I went to retropie settings I saw this Screen scraper error show up.

Yes, that usually shows up when trying to scrape from ScreenScraper and their servers are busy and not accepting connections from non-authenticated users.

Emulatre71

@mitu For the whole mac changing it was for changing my mac address on my RPI4 and also changing the IP address of my router. As for Retropie configuration having a static mac address or not I saw I was able to install macchanger on it and change it's mac address.

Scrapper has been acting odd so I had to retry a few times before it scrapped a rom. TheGameDB has not been working at all for me, I haven't played around with renaming roms to see if that works. For instance I have Darkstalkers 3 uncensored hack version, I'll play around with renaming it and maybe the files inside to see it works with SS scraper.

I do have a question about those with bin/cue file formats for PS1. It appears like this because the folders are in the ps1 rom folder, I did not take the contents and just throw them straight into the directory because I like to keep things organized. Are they supposed to be in a folder with their respective bin/cue files?

You don't see the scraper/boxart info until you click on it, this is how they appear so far. I see that I can have two images set to it, with the way that it is now.
RP scrapper.png

sleve_mcdichael

@emulatre71 said in Scraper not working when attempting to scrape games for descriptions and boxarts.:

I do have a question about those with bin/cue file formats for PS1. It appears like this because the folders are in the ps1 rom folder, I did not take the contents and just throw them straight into the directory because I like to keep things organized. Are they supposed to be in a folder with their respective bin/cue files?
You don't see the scraper/boxart info until you click on it, this is how they appear so far.

You can convert .bin/.cue sets to .chd using the CHDMAN tool, to clean up your folders to one file per disc (and potentially also save some space.) This will allow you to keep them all in the root psx folder without being too cluttered up with .bin's and .cue's. After this (you might need to re-scrape your games so it associates the new filenames), the menu art should again work as expected: https://retropie.org.uk/docs/CHD-files/#creating-chds-from-cd-roms

Then, for multi-disc games, you can rename the .chd's to .cd1, .cd2 etc. (to hide them from the gamelist) and use .m3u playlist files to clean up your game list menu to one menu item per game. Another advantage of this is on-the-fly disc swapping through the Retroarch menu, and easier save file management (one save file per game, named after the .m3u playlist, instead of a separate file for each disc named after the disc): https://retropie.org.uk/docs/Playstation-1/#m3u-playlists-for-cue-bins-or-chds