Versatile C++ game scraper: Skyscraper

muldjord · 3 Jul 2019, 23:58

Skyscraper 3.1.1 released: https://github.com/muldjord/skyscraper

Worked around a bug in the 'screenscraper' xml output improving stability
Added limiter to media getter functions for 'screenscraper' which should improve stability

I've been working to improve the stability of the screenscrapermodule. The fixes included in this release seem to have made quite a difference, so please update.

chubsta · 8 Mar 2019, 00:30

Quick question guys - if I am trying to update a set of roms, do I have to delete the game list, rescrape the roms, and generate the new game list, or just rescrape and generate?

muldjord · 3 Aug 2019, 00:47

@chubsta said in Versatile C++ game scraper: Skyscraper:

Quick question guys - if I am trying to update a set of roms, do I have to delete the game list, rescrape the roms, and generate the new game list, or just rescrape and generate?

Depends on what you mean by "update a set of roms". If you mean you want to update the data from the sources, then you need to regather the data first using the "--refresh" option and then regenerate the game list afterwards. But if you mean "update a set or roms" after you've just changed some artwork configuration or other local Skyscraper setting, then you just need to regenerate the game list. No need to regather the data as it is already cached and ready for use.

EDIT: This page explains it: https://github.com/muldjord/skyscraper/blob/master/docs/USECASE.md

chubsta · 8 Mar 2019, 01:00

@muldjord thanks for the fast reply - I have removed a load of roms from one of my Mame sets, and added some more. I don't have any cached info as I clear the cache once I have scraped, my concern was that when writing a new game list it would add the new info to the old and I would end up with both duplicate games and also games in the list that are no longer in the folder.

I guess in this case it is simple enough to just delete the old one and rescrape to give myself a fresh game list.

Great scraper BTW, works so much better than anything else I have tried!

ByteThis · 8 Mar 2019, 03:06

@muldjord said in Versatile C++ game scraper: Skyscraper:

Windows probably has the worst command prompt

And OS ever. I'll leave this here..lol.

Silent · 9 Mar 2019, 10:33

Hey,

Moonlight scriptmodule has now been merged into RetroPie-Setup. Hence, this is the correct time for me to summarize a request to add support for streaming media, like I did in my PR a while ago.

What I propose is:

Make .ml a valid extension for all platforms, like .zip .7z is at the moment. Technically you can stream anything from your PC, so there is no reason not to allow it everywhere.
Add steam platform support. Should have same aliases as PC platforms, maybe with PC platform also getting a Steam alias.
Add ps2 platform support. Even though there is no PS2 emulator on Pi, you can obviously stream it and technically RetroPie supports it. Most EmulationStation themes already support this platform, too.

Those are the changes I made in my local copy, and I think they should be sufficient. Technically, you could argue that ps3 (because of RPCS3) and x360 (because of Xenia) platforms also make sense, but I don't know if there are themes supporting that.

Thanks in advance!

quicksilver · 9 Mar 2019, 16:32

When scraping a game that won't be able to be found by it's checksum, Screenscraper will attempt to match by exact rom name from what I understand. However if there is more than one game with that exact title it will fail to scrape. I have a number of .sh and .conf files that cannot be found by checksum but fail based on ROM name because there are many versions of the game in screenscraper's database (quake II for example). Is there anyway to work around this?

muldjord · 9 Mar 2019, 21:15

@quicksilver said in Versatile C++ game scraper: Skyscraper:

When scraping a game that won't be able to be found by it's checksum, Screenscraper will attempt to match by exact rom name from what I understand. However if there is more than one game with that exact title it will fail to scrape. I have a number of .sh and .conf files that cannot be found by checksum but fail based on ROM name because there are many versions of the game in screenscraper's database (quake II for example). Is there anyway to work around this?

Not that I know of. This is a small limitation in how their database handles things. I do believe it only searches on the platform in question though, so this shouldn't happen too often as the platform id is also send in the requests.

muldjord · 9 Mar 2019, 21:15

@Silent Awesome, I'll look into it and include it in the next release. Thanks.

EDIT:

Only igdb has a platform for Steam related platform called steamos. I have added that to pc and steam but it is only relevant when scraping with igdb
steamplatform added with all relevant aliases including steamos
ps2 platform added with all relevant aliases
*.ml added for all platforms

I'll leave out ps3 and x360 for now. I don't want to spam platforms all over the place just for the sake of it.

quicksilver · 9 Mar 2019, 23:56

@muldjord strangely the built in scraper in emulation station will find the same titles using screenscraper as the source, unless of course it searches for ROM name differently. Either way, I was able to scrape those problematic games using different sources. Again, thank you for your help. I have been promoting skyscraper every chance I get!

muldjord · 10 Mar 2019, 00:18

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord strangely the built in scraper in emulation station will find the same titles using screenscraper as the source, unless of course it searches for ROM name differently. Either way, I was able to scrape those problematic games using different sources. Again, thank you for your help. I have been promoting skyscraper every chance I get!

Can you give me an exact file name and platform for a game it doesn't find that the built-in scraper does find when using screenscraper as source? I would like to test this.

quicksilver · 3 Oct 2019, 05:01

@muldjord "Oregon Trail II". It's a .conf file in my PC folder. Built in scraper will find it with screenscraper as the source but skyscraper will not. There were a couple others, but Im away from my pi and can't remember off the top of my head. But I'll check when next I'm home.

Edit: "Myst" is another one.

quicksilver · 10 Mar 2019, 05:03

@muldjord said in Versatile C++ game scraper: Skyscraper:

pc and ports have the exact same aliases

I wondering if the built in scraper doesn't have multiple aliases for PC and that's why it's able to find them.

mitu · 3 Oct 2019, 05:17

@quicksilver said in Versatile C++ game scraper: Skyscraper:

I wondering if the built in scraper doesn't have multiple aliases for PC and that's why it's able to find them.

Nope - the internal scraper (when using ScreenScraper) only knows about pc, for ports it ignores the platform and queries by name only.

EDIT: either that or the Ports system is considered as pc by Emulationstation.

quicksilver · 10 Mar 2019, 05:19

@mitu thanks for the confirmation. I would assume that's where the difference is then.

muldjord · 10 Mar 2019, 07:40

@mitu said in Versatile C++ game scraper: Skyscraper:

Nope - the internal scraper (when using ScreenScraper) only knows about pc, for ports it ignores the platform and queries by name only.

Oh, that's right. It uses the new search query instead of the jeuInfos, right? Then that's the difference. I might look into using the search query one at one point.

mitu · 10 Mar 2019, 08:23

@muldjord said in Versatile C++ game scraper: Skyscraper:

Oh, that's right. It uses the new search query instead of the jeuInfos, right? Then that's the difference. I might look into using the search query one at one point.

No, it's still using the game info endpoint (v2 WebApi). I've looked over using the search endpoint - see my post on their forum about the search returning to many unrelated results. However, the search is slower then the game info endpoint and I think it's not worth (at the moment) the effort to switch to it - considering their server(s) are already busy. Improving the search over the regular game info endpoint is a better investment at this point (checksum, search-by-name, search-by-id).

muldjord · 10 Mar 2019, 10:09

Skyscraper 3.1.2 released: https://github.com/muldjord/skyscraper

Added support for Moonlight '*.ml' extension for all platforms
Added 'ps2' platform for use with Moonlight and all other supported Ps2 extensions
Added 'steam' platform for use with Moonlight '*.ml' files
Fixed minor bug in CRC checksum calculator for 'screenscraper'
Improved workarounds for ScreenScraper XML errors

Yet again I fixed some errors in the screenscraper module. My CRC checksum calculator sometimes returned 7 char strings. This caused issues with the replies from screenscraper, so I've made sure I check the length and prepend a "0" when this happens. I also improved some of the workaround for the XML errors screenscraper sometimes send out.
Lastly you should now be able to scrape moonlight files with all platforms as requested by @Silent. Pretty untested though, so let me know how this works for you.

Silent · 3 Oct 2019, 11:25

@muldjord I quickly tested scraping a few "roms" from screenscraper/thegamesdb/igdb and I can confirm it seems to work fine :) Thanks a lot!

EDIT:
Quick observation: I tried to scrape this game off MobyGames:
https://www.mobygames.com/game/spintires-mudrunner

MobyGames name is "Spintires: MudRunner", whereas my .ml file is named "Spintires MudRunner". If you try to search for it like this on the website, you get normal results:
https://www.mobygames.com/search/quick?q=Spintires+Mudrunner

Yet, Skyscraper couldn't find it unless I leveraged aliasMap.csv to add : to the name. API's fault or Skyscraper not filtering results well enough?

Silent · 3 Oct 2019, 11:54

Another note: I can't find absolutely nothing for steam platform from thegamesdb scraper. My cache has entries for those, which means I used to be able to find things smoothly, but now I can't.

EDIT:
Also, looks like NVidia Gamestream added support for Forza Horizon 4:
https://www.mobygames.com/game/forza-horizon-4

Platform there shows as "Windows Apps", and looks like Skyscraper can't find it because of that either.