Adding Scraper to Emulationstation

10vorne

Now that I've found out that by copying certain folders I can also update Retropie without losing my previous info, I'd also like to look into scrapers again.

I checked out the Retropie documentation. If I understand this correctly, I can only integrate two scrapers into the emulation station, correct? Honestly, I would prefer this way, so I could scrape every game I want to play, adjust the data and save it... and hopefully keep it forever :)

But at the moment I only see boxart, you can also use the existing scrapers here use for screenshots or add another scraper in the emulation station?

mitu

@10vorne said in Adding Scraper to Emulationstation:

I checked out the Retropie documentation. If I understand this correctly, I can only integrate two scrapers into the emulation station, correct? Honestly, I would prefer this way, so I could scrape every game I want to play, adjust the data and save it... and hopefully keep it forever :)

Yes, you can scrape individual games from EmulationStation or entire systems at a time. You don't 'integrate' the scrapers, they're already built-in - with the 2 scraping sources.

But at the moment I only see boxart, you can also use the existing scrapers here use for screenshots or add another scraper in the emulation station?

No, the internal scraper gets only 1 image, which is usually the boxart. If you want more than 1 image per game, you need also a theme which supports multiple images per game. Most (if not all) of the themes do not support more than 1 image per game (leaving aside the marquee, which is a different thing).

10vorne

@mitu The best combination for me would be box art and screenshots. I use Madness Magazine as a theme. The scrapers in Emulation Station get the boxart as you say. What's the best way to go about it now if I want to have screenshots? Is this also possible via the emulation station (supplement of a scraper with a screenshot function?) or only via the method of running the scraper over several ROMs at the same time? I would like to avoid having to play around with xml files and would like to be able to control and/or edit everything via the interface.

mitu

You can use an external screenscraper - Skyscraper or ARRM/Skraper if you wish to get screenshots instead of/in addition to boxart.

10vorne

@mitu
I now use Skyscraper and it basically works well for me. But there are always Roms that are not recognized. How do I improve here on individual roms? So I can go via the emulation station, but I only get the box artand no screenshots. Is there another possibility to scarpe again single roms?

sleve_mcdichael

@10vorne Skyscraper can use multiple sources (screenscraper, thegamesdb, etc.) If a title is not found at one source, try another one. You can use the "onlymissing" option to skip those titles already scraped by the first source and only do the remaining ones that have no data at all in the local cache.

If you have something really obscure that's not found in any of the scraping sources, you can download your own images and scrape local resources with the "import" function:

https://github.com/muldjord/skyscraper/blob/master/docs/IMPORT.md

Skyscraper can be run from command-line if you want to process individual roms:

https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

10vorne

My C64 collection is relatively extensive. I've been running the scraper over it for a few days now. From time to time I'll make the list. I've set that already scraped games should not be checked again. But what about the games that was not found? I have the feeling that they are checked again with each run?

mitu

@10vorne said in Adding Scraper to Emulationstation:

But what about the games that was not found? I have the feeling that they are checked again with each run?

If you're using the EmulationStation' scraper, yes, they'll be checked again.

10vorne

Sorry, I forgot. I use the skyscraper. And at the moment I have the feeling that the C64 list will never be finished because the games not found in the pre-run are checked again. And at some point the limit is reached, I create the list and I start all over again. But I could be wrong, as there are many cryptic file names in the C64 folders (newspaper cassettes, listings, etc.). If Skyscraper keeps checking them and doesn't find them, I would first remove these folders before scraping.

sleve_mcdichael

@10vorne I recommend using Skyscraper from command line for maximum performance:

https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

You could use the --startat/--endat options to scrape only sections of your library, or add the skipped games to an --excludefrom list so they won't be re-tried each time. Or you can scrape individual titles bu specifying them in a space-separated list after the command, i.e.:

Skyscraper -p snes -s screenscraper --flags videos,nocovers /path/to/game1.sfc /path/to/game2.zip

Paths can be relative or absolute. See docs for much, much more.

10vorne

@sleve_mcdichael Thank you. I'll see if I can handle it. For example, if I only scrape one directory, can I then scrape other directories before generating the gamelist, or do I have to do this after each directory? In other words, when would the crapped content expire if I didn't generate the gamelist?

sleve_mcdichael

@10vorne said in Adding Scraper to Emulationstation:

In other words, when would the crapped content expire if I didn't generate the gamelist?

Never, it is all stored permanently in your local cache (until you vacuum or purge it with the --cache option). You can scrape to your heart's content (or until your storage fills up -- my cache is about 2.4GB for 1100 games) and then generate the most complete gamelist you can muster with all the best datas from all the different scraping modules. You can even manage a "priorities.xml" file if you want (a different one for each system, if you want! Or not, you can use a global one too. If you want) so you can say like "well site A has better cover art than site B, and site C has the best descriptions but their library isn't very big so if C doesn't have one for a game, use site D before any of the rest, and for marquee images, try B first and then E if B doesn't have it, and..." and then when you go to generate the gamelist, it will sort everything by your chosen priorities and pick the highest match it has cached for each individual asset.

10vorne

@sleve_mcdichael Since I left the scraper run unattended and eventually return to the pi and I then mostly see the Skyskraper menu back in Retropie-Setup, I often don't know whether the scraper has finished the specified systems or whether my limit was reached. Can I read out with one command how much of the daily limit I have left?

sleve_mcdichael

@10vorne said in Adding Scraper to Emulationstation:

Can I read out with one command how much of the daily limit I have left?

If you scrape a title from command-line, it's reported in the cache output after each game is scraped:

pi@retropie:~ $ Skyscraper -p arcade -s screenscraper ~/RetroPie/roms/arcade/frogger.zip

[...]

Cache output:
Video conversion: Success!

'screenscraper' requests remaining: 19999

#1/1, (1/0)
Elapsed time   : 00:00:22
Est. time left : 00:00:00

10vorne

Just to be safe. In the Retropie setup I can create a config.ini for the Skyscraper. Do I understand it correctly that all default values are used, except for those that I switch on or off via the config.ini? So, for example, if I want to scrape the PC ROMs but not the subdirectories, do I just write --nosubdirs in the config.ini and trigger the process and leave all other settings at the default value?