Versatile C++ game scraper: Skyscraper

mitu

@Halvhjearne The screenscraper site dashboard shows their CPU at 294% right now, so I wouldn't be surprised if their system overload would result in the errors you're getting.

Halvhjearne

@mitu
scraping again without the --refresh seems to work fine ...

mitu

@Halvhjearne Why are you using --refresh by default ? It forces Skyscraper to re-fetch the data when you already have it in cache.

Halvhjearne

@mitu
im not using it by default, but i do run skyscraper sometimes with the --refresh to see if there is any missing artwork available ...

mitu

That's not what --refresh should be used for. If you don't have the corresponding artwork in the cache (marquee, cover, screenshot, video), Skyscraper would try and get it, so you don't have to add --refresh to get missing artwork. @muldjord can correct me if I'm wrong, but forcing --refresh just negates any advantage of having a cache.

muldjord

@mitu said in Versatile C++ game scraper: Skyscraper:

That's not what --refresh should be used for. If you don't have the corresponding artwork in the cache (marquee, cover, screenshot, video), Skyscraper would try and get it, so you don't have to add --refresh to get missing artwork. @muldjord can correct me if I'm wrong, but forcing --refresh just negates any advantage of having a cache.

This is true, but there is one issue where it helps. When the screenscraper servers are overloaded, some of the media might not be scraped for a certain game, even though the textual data is (each artwork resource is a separate request that might fail due to the high load). So in those (hopefully rare cases, especially for registered users) they will need to use --refresh to grab them.

With that said, I would always recommend only scraping a few roms with --refreshenabled. Instead, just add those few files to command line to grab the missing data. That is much better than scraping everything again.

quicksilver

@muldjord Is there a way to use the --refresh function only on roms that are detected as missing some media? I.e. screenshot, box art or description are missing? Im not sure if what I am asking makes sense. Unfortunately for me I scraped hundreds of roms before realizing that credentials for screenscraper made such a difference. Now there are tons of roms that are missing some media. Going through and checking them one by one will be a huge task.

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord Is there a way to use the --refresh function only on roms that are detected as missing some media? I.e. screenshot, box art or description are missing? Im not sure if what I am asking makes sense. Unfortunately for me I scraped hundreds of roms before realizing that credentials for screenscraper made such a difference. Now there are tons of roms that are missing some media. Going through and checking them one by one will be a huge task.

What you are asking makes perfect sense, but it is not possible to do this at the moment I'm afraid. I had not anticipated so many requests to be rejected, so this is actually quite a big problem at the moment. And it's a bit of a bad circle to get into. People will notice media being missing, and then they will start using refresh, which again puts even more load on the source servers and so on...

I am wondering if screenscraper has any sort of server caching installed on their service. This could potentially alleviate the problem. I've asked on their forum.

quicksilver

@muldjord So I'm slowly but surely getting all of ROMs scraped using skyscraper and the results are simply amazing. I should have done this long ago!

I just wanted to thank you for such a great scraper! I have read back through a lot of the posts on this thread and your commitment to creating a great program clearly shows. I hope you take great pride in this! The next time I'm at my computer I will be sure to send a tip your way. Thanks again for all your hard work!

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord So I'm slowly but surely getting all of ROMs scraped using skyscraper and the results are simply amazing. I should have done this long ago!

I just wanted to thank you for such a great scraper! I have read back through a lot of the posts on this thread and your commitment to creating a great program clearly shows. I hope you take great pride in this! The next time I'm at my computer I will be sure to send a tip your way. Thanks again for all your hard work!

You are very welcome and thanks! Skyscraper is sort of a passion project of mine and I probably spend more time on it than I should. :D

quicksilver

@muldjord sorry to be a nuisance again so soon, but I'm having issues generating Gameslist for "ports". Using the GUI I was unable to scrape any games because the platform is set as "ports" which the sources don't recognize as a valid platform. So I scraped them all from the commandline using pc as the platform. However, when I go to generate the Gameslist it just says the games cannot be found.

Edit: I see scummvm set to look for PC and Amiga matches. Perhaps ports can be set to look at pc?

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

Using the GUI I was unable to scrape any games because the platform is set as "ports" which the sources don't recognize as a valid platform.

The ports platform already has several platform aliases. Among them is pc, so it should work just fine with the file name based modules provided your scripts have a good naming convention (Such as Game Name.sh). It will scrape .sh file when using ports. screenscraper might give you issues though, as I don't send a proper system id with the query since I don't know what platform the games will be for from the get-go as I do for other platforms. So there you will have to work around it as you already do.

EDIT: I actually just noticed that I can make this work with screenscraper aswell. The PC Dos platform they have is the parent ID also for the Windows platform. So I can set that and it will work.

If you want to test it out you can simply set line 571 in screenscraper.cpp to read return "135";. Then compile Skyscraper with make and then sudo make install. But this will only work if you've compiled it yourself using the approach from the github documentation.

quicksilver

@muldjord the strange thing is thegamesdb will find the games if I search manually from the commandline with the platform set as pc. But if I search using the GUI it will return all games as not found.

I installed skyscraper through the retropie setup script. In order to test your fix would I have to uninstall and build it manually?

mitu

@quicksilver Hm, I'll see if I can reproduce this. Can you give me the details - name of the file, folder (I guess it's ports ?) ?

quicksilver

@mitu correct the game folders+the .sh scripts to run the games are in my /roms/ports/ directory.

If I attempt to scrape the .sh scripts on the command line using "pc" as the platform I will get successful scrapes for doom, doom 2, quake, hexen etc. But it won't work if the platform is set as "ports". Screenscraper shows a message that it's not a supported platform and the gamesdb can't find anything (unless I change the platform to PC like I said). Oddly enough, if I trick it by scraping my ports using PC as the platform the Gameslist will not generate anything.

Edit: and speaking of PC, it looks like skyscraper doesn't recognize .conf files as a valid extension in my PC folder.

mitu

@quicksilver said in Versatile C++ game scraper: Skyscraper:

if I trick it by scraping my ports using PC as the platform the Gameslist will not generate anything.

How exactly are you 'tricking it' ? Are you copying the .sh files to the pc folder and scraping it from there ?

quicksilver

@mitu no, the .sh files are for my ports games (doom, quake etc.) and they are in my ports folder. What I'm doing is changing the platform on the command line to "pc"

So for example this is the default and it returns no matching games:

Skyscraper -p ports -s thegamesdb ./RetroPie/roms/ports/doom.sh

But if I change to:

Skyscraper -p pc -s thegamesdb ./RetroPie/roms/ports/doom.sh

It will find the game no problem. I believe this even works for screenscraper as well if the name is a 100% match.

mitu

@quicksilver Ah, ok, then it's normal that generating the list won't work, since the cache entry is associated with the pc platform.

quicksilver

@mitu I figured it had something to do with that. But according to @muldjord, ports should already be looking for PC matches but unless I tell it to look for them by setting pc as the platform, it can't find them.

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@mitu I figured it had something to do with that. But according to @muldjord, ports should already be looking for PC matches but unless I tell it to look for them by setting pc as the platform, it can't find them.

Quick note: You should never give it a file name on command line that is not related to the current input folder (which in this case is roms/pc). If you do, it won't be included when you generate the game list afterwards since it is no longer in the input directory. You are basically caching data for a file that effectively doesn't exist when you generate the platform later on (since it will look for files in roms/ports at that point). :)

Ports currently looks for both Windows, Dos and PC as platforms when using the file name based modules (all modules except screenscraper). So that should work. If you use the fix I gave you earlier it might also work with screenscraper.