Versatile C++ game scraper: Skyscraper

muldjord · 10 Mar 2019, 07:40

@mitu said in Versatile C++ game scraper: Skyscraper:

Nope - the internal scraper (when using ScreenScraper) only knows about pc, for ports it ignores the platform and queries by name only.

Oh, that's right. It uses the new search query instead of the jeuInfos, right? Then that's the difference. I might look into using the search query one at one point.

mitu · 10 Mar 2019, 08:23

@muldjord said in Versatile C++ game scraper: Skyscraper:

Oh, that's right. It uses the new search query instead of the jeuInfos, right? Then that's the difference. I might look into using the search query one at one point.

No, it's still using the game info endpoint (v2 WebApi). I've looked over using the search endpoint - see my post on their forum about the search returning to many unrelated results. However, the search is slower then the game info endpoint and I think it's not worth (at the moment) the effort to switch to it - considering their server(s) are already busy. Improving the search over the regular game info endpoint is a better investment at this point (checksum, search-by-name, search-by-id).

muldjord · 10 Mar 2019, 10:09

Skyscraper 3.1.2 released: https://github.com/muldjord/skyscraper

Added support for Moonlight '*.ml' extension for all platforms
Added 'ps2' platform for use with Moonlight and all other supported Ps2 extensions
Added 'steam' platform for use with Moonlight '*.ml' files
Fixed minor bug in CRC checksum calculator for 'screenscraper'
Improved workarounds for ScreenScraper XML errors

Yet again I fixed some errors in the screenscraper module. My CRC checksum calculator sometimes returned 7 char strings. This caused issues with the replies from screenscraper, so I've made sure I check the length and prepend a "0" when this happens. I also improved some of the workaround for the XML errors screenscraper sometimes send out.
Lastly you should now be able to scrape moonlight files with all platforms as requested by @Silent. Pretty untested though, so let me know how this works for you.

Silent · 3 Oct 2019, 11:25

@muldjord I quickly tested scraping a few "roms" from screenscraper/thegamesdb/igdb and I can confirm it seems to work fine :) Thanks a lot!

EDIT:
Quick observation: I tried to scrape this game off MobyGames:
https://www.mobygames.com/game/spintires-mudrunner

MobyGames name is "Spintires: MudRunner", whereas my .ml file is named "Spintires MudRunner". If you try to search for it like this on the website, you get normal results:
https://www.mobygames.com/search/quick?q=Spintires+Mudrunner

Yet, Skyscraper couldn't find it unless I leveraged aliasMap.csv to add : to the name. API's fault or Skyscraper not filtering results well enough?

Silent · 3 Oct 2019, 11:54

Another note: I can't find absolutely nothing for steam platform from thegamesdb scraper. My cache has entries for those, which means I used to be able to find things smoothly, but now I can't.

EDIT:
Also, looks like NVidia Gamestream added support for Forza Horizon 4:
https://www.mobygames.com/game/forza-horizon-4

Platform there shows as "Windows Apps", and looks like Skyscraper can't find it because of that either.

muldjord · 3 Oct 2019, 12:50

@Silent said in Versatile C++ game scraper: Skyscraper:

Another note: I can't find absolutely nothing for steam platform from thegamesdb scraper. My cache has entries for those, which means I used to be able to find things smoothly, but now I can't.

Double negative, but I assume you meant "I can't find anything". I have added the *.ml extension to be id'et by filename since they are scripts. This might have fixed it but you will have to scrape all of your ml files all over again. I just re-released 3.1.2 for this. Maybe I should start doing betas instead.

I've also added the "windows apps" alias to steam and pc. So please re-update to 3.1.2.

EDIT: Figured out why you get no results with thegamesdb. They've apparently added new platforms and I haven't updated my internal list in a while since I mostly focus on retro platforms. So I'll need to figure that out... Let me look into this.

muldjord · 10 Mar 2019, 12:34

This post is deleted!

muldjord · 10 Mar 2019, 12:40

@Silent said in Versatile C++ game scraper: Skyscraper:

@muldjord I quickly tested scraping a few "roms" from screenscraper/thegamesdb/igdb and I can confirm it seems to work fine :) Thanks a lot!

EDIT:
Quick observation: I tried to scrape this game off MobyGames:
https://www.mobygames.com/game/spintires-mudrunner

MobyGames name is "Spintires: MudRunner", whereas my .ml file is named "Spintires MudRunner". If you try to search for it like this on the website, you get normal results:
https://www.mobygames.com/search/quick?q=Spintires+Mudrunner

Yet, Skyscraper couldn't find it unless I leveraged aliasMap.csv to add : to the name. API's fault or Skyscraper not filtering results well enough?

Seems like this is due to their API. The query Skyscraper sends for that file name is "spintires+mudrunner". This gives a result when using the website search, but not when using the API.

muldjord · 10 Mar 2019, 13:17

Ok, thegamesdb should provide more results for the steamplatform now. But I had to add aliases for the xbone and ps4 for it. It rarely returns PC results it seems, which is really odd. But not something I can change. Re-released 3.1.2 again... Please re-update.

SgtJimmyRustles · 10 Mar 2019, 14:55

I love this tool, thanks for making it. Really does wonders for scraping roms.

Ranma · 23 Mar 2019, 16:55

Would appreciate a tiny bit of help. I've read the documentation and various threads but yet to find an answer. So far doing really well with Skyscraper. It's amazing! But I'm having an issue with one or two games that share a name with each other and it's driving me crazy!

Particularly, there's a MAME title called Badlands. Then there's a MAME title called Bad Lands. It's always selecting the default incorrect one for my needs and I don't know how to get it to offer me a choice between them. I thought the --interactive option would help but it didn't seem to make any difference. How can I select the correct one? For info I am using Skyscraper on my PC running Manjaro Linux. It's otherwise perfect.

Any help with this very much appreciated. :-)

quicksilver · 23 Mar 2019, 17:13

@Ranma which source is returning the wrong result? And have you tried a different source?

Ranma · 23 Mar 2019, 18:13

@quicksilver
I'm using Screenscraper. I know they have the proper one I'm after because if I scrape from a Recalbox install I can select the correct one. Unfortunately the art style it uses doesn't fit what I want, and I don't know how to tailor it in Recalbox, so I'm using my desktop to scrape. It's quicker anyway.

mitu · 23 Mar 2019, 18:15

@Ranma Try using the --query option and select the right CRC for the ROM, as it's shown on the Screenscraper site. This will create the proper association between ROM and scrape source.

Ranma · 23 Mar 2019, 18:42

Thanks for replying but it's a loss as I can't search the Screenscraper site to find the damn info! Tbh I think the site is struggling at the moment. I'm getting an error when I try to use Recalbox to scrape too. Keeps saying disc full, but I know the SD card has loads of space.

EDIT: Ah looks like I'm right. Getting the output below using the --query flag you recommended.

The screenscraper service is currently too busy to handle requests from unregistered users. Sign up for an account at https://www.screenscraper.fr and support them to gain more threads. Then use the credentials with Skyscraper using the '-u [user:password]' command line option or by setting 'userCreds=[user:password]' in '~/.skyscraper/config.ini'.

I'll try it later. Thanks for the help though!

quicksilver · 23 Mar 2019, 19:42

@Ranma registration is free and is easy to add your credentials to skyscraper

Ranma · 24 Mar 2019, 10:38

Still not working even after registering. Driving me crackers!

Skyscraper -p mame-libretro -s screenscraper -u xxxxxxxx:xxxxxxxxxxxx --query "md5=[dfd57074a033d47df8e68d4c7b08013c]" /home/me/Downloads/mame-libretro/mame-libretro/badlands.zip

First 256 characters of answer was:
Champ crc, md5 ou sha1 erroné  
#1/1 (T1) Pass 1 ---- Game 'badlands' not found :( ----

But it's there! See? Same md5 and everything!

muldjord · 24 Mar 2019, 14:10

@Ranma said in Versatile C++ game scraper: Skyscraper:

Still not working even after registering. Driving me crackers!

Skyscraper -p mame-libretro -s screenscraper -u xxxxxxxx:xxxxxxxxxxxx --query "md5=[dfd57074a033d47df8e68d4c7b08013c]" /home/me/Downloads/mame-libretro/mame-libretro/badlands.zip

First 256 characters of answer was:
Champ crc, md5 ou sha1 erroné  
#1/1 (T1) Pass 1 ---- Game 'badlands' not found :( ----

But it's there! See? Same md5 and everything!

Why are you adding square brackets to the query? Remove those.

EDIT: Oh, you saw those in the docs and thought you needed to apply them. The square brackets are only there to show that it is a variable. I've changed them to < and > now. Maybe more people will understand that.

Ranma · 24 Mar 2019, 14:43

@muldjord
Ah I see! I've been using Linux for many years and everyone's approach to the syntax is always different from the last person's. If it's printed in the tutorial, I'll type it! Thanks for the heads up. :-)

EDIT: Finally, it worked! Many thanks.

muldjord · 24 Mar 2019, 15:13

@Ranma said in Versatile C++ game scraper: Skyscraper:

Ah I see! I've been using Linux for many years and everyone's approach to the syntax is always different from the last person's.

Agreed. I've recently changed all [ and ] to < and >. But you're right, it's hard to find concensus.

EDIT: Finally, it worked! Many thanks.

Awesome! :)