Adding Scraper to Emulationstation

sleve_mcdichael

@10vorne Skyscraper can use multiple sources (screenscraper, thegamesdb, etc.) If a title is not found at one source, try another one. You can use the "onlymissing" option to skip those titles already scraped by the first source and only do the remaining ones that have no data at all in the local cache.

If you have something really obscure that's not found in any of the scraping sources, you can download your own images and scrape local resources with the "import" function:

https://github.com/muldjord/skyscraper/blob/master/docs/IMPORT.md

Skyscraper can be run from command-line if you want to process individual roms:

https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

10vorne

My C64 collection is relatively extensive. I've been running the scraper over it for a few days now. From time to time I'll make the list. I've set that already scraped games should not be checked again. But what about the games that was not found? I have the feeling that they are checked again with each run?

mitu

@10vorne said in Adding Scraper to Emulationstation:

But what about the games that was not found? I have the feeling that they are checked again with each run?

If you're using the EmulationStation' scraper, yes, they'll be checked again.

10vorne

Sorry, I forgot. I use the skyscraper. And at the moment I have the feeling that the C64 list will never be finished because the games not found in the pre-run are checked again. And at some point the limit is reached, I create the list and I start all over again. But I could be wrong, as there are many cryptic file names in the C64 folders (newspaper cassettes, listings, etc.). If Skyscraper keeps checking them and doesn't find them, I would first remove these folders before scraping.

sleve_mcdichael

@10vorne I recommend using Skyscraper from command line for maximum performance:

https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

You could use the --startat/--endat options to scrape only sections of your library, or add the skipped games to an --excludefrom list so they won't be re-tried each time. Or you can scrape individual titles bu specifying them in a space-separated list after the command, i.e.:

Skyscraper -p snes -s screenscraper --flags videos,nocovers /path/to/game1.sfc /path/to/game2.zip

Paths can be relative or absolute. See docs for much, much more.

10vorne

@sleve_mcdichael Thank you. I'll see if I can handle it. For example, if I only scrape one directory, can I then scrape other directories before generating the gamelist, or do I have to do this after each directory? In other words, when would the crapped content expire if I didn't generate the gamelist?

sleve_mcdichael

@10vorne said in Adding Scraper to Emulationstation:

In other words, when would the crapped content expire if I didn't generate the gamelist?

Never, it is all stored permanently in your local cache (until you vacuum or purge it with the --cache option). You can scrape to your heart's content (or until your storage fills up -- my cache is about 2.4GB for 1100 games) and then generate the most complete gamelist you can muster with all the best datas from all the different scraping modules. You can even manage a "priorities.xml" file if you want (a different one for each system, if you want! Or not, you can use a global one too. If you want) so you can say like "well site A has better cover art than site B, and site C has the best descriptions but their library isn't very big so if C doesn't have one for a game, use site D before any of the rest, and for marquee images, try B first and then E if B doesn't have it, and..." and then when you go to generate the gamelist, it will sort everything by your chosen priorities and pick the highest match it has cached for each individual asset.

10vorne

@sleve_mcdichael Since I left the scraper run unattended and eventually return to the pi and I then mostly see the Skyskraper menu back in Retropie-Setup, I often don't know whether the scraper has finished the specified systems or whether my limit was reached. Can I read out with one command how much of the daily limit I have left?

sleve_mcdichael

@10vorne said in Adding Scraper to Emulationstation:

Can I read out with one command how much of the daily limit I have left?

If you scrape a title from command-line, it's reported in the cache output after each game is scraped:

pi@retropie:~ $ Skyscraper -p arcade -s screenscraper ~/RetroPie/roms/arcade/frogger.zip

[...]

Cache output:
Video conversion: Success!

'screenscraper' requests remaining: 19999

#1/1, (1/0)
Elapsed time   : 00:00:22
Est. time left : 00:00:00

10vorne

Just to be safe. In the Retropie setup I can create a config.ini for the Skyscraper. Do I understand it correctly that all default values are used, except for those that I switch on or off via the config.ini? So, for example, if I want to scrape the PC ROMs but not the subdirectories, do I just write --nosubdirs in the config.ini and trigger the process and leave all other settings at the default value?