Versatile C++ game scraper: Skyscraper

Clyde

@muldjord Thank you for your ongoing work.

One (seemingly) small feature request. 😌 Next to the --symlink option, a --hardlink option would be great. Hardlinks would combine the benefits of saved storage space like symlinks, but would also work like normal files if the roms and/or media directories are copied to another medium. Would that be feasible for a coming update?

mitu

Actually, using a hardlink has a few limitations and caveats:

a hardlink cannot be created across filesystems - it's only between files on the same FS.
if you remove the source, then re-create it (with the same name), the hardlink would still contain the 1st source data, while the symlink would still work.

muldjord

@Clyde The framework I use does not support creating hardlinks only symlinks.

Clyde

@mitu Yes, both variants have their pitfalls (like symlinks' dependence on the source). I just would have liked to have both as options. :)

@muldjord A pity, but thanks for clearing it up.

steptoe

Hi,

Just wondering if you have any plans to add support for pinball in your scraper

I'm aware not many sites have pinball data on their videos or images. emumovies seems to be about the only one that has some videos/images

Mainly Future Pinball and Visual Pinball/PinMAME

Thanks

Clyde

@muldjord Happy holidays! I recently noticed a wrongly scraped game. Would it help if I reported this to you? If so, how should I do it?

steptoe

I'm trying to compiler the latest Skyscraper 3.3.5 for Windows using QT as described here :

*You can easily cross-compile it for Windows. Just download and install Qt 5 from the official site (open source version). When installing, put a checkmark in the latest mingw32 version, I don't support MSVC++ although it might work, not sure. When it's installed you'll find a Qt specific command prompt in your start menu. From there navigate to the Skyscraper source folder and run:
$ qmake
$ mingw32-make release

That will give you a compiled version in a subfolder called "release". Then you need to copy the needed dll's from the qt5 install folder into the release folder and start an ordinary command prompt there. Then you should be able to run it.*

Running the QT Creator and trying various new projects and not finding how to point at the Skyscraper source folder to compile a windows version

My intention is then look into using other 3rd party software to create a simple GUI for us Windows users. I use SkraperUI but Skyscraper gets better and better as I've followed its development for RetroPie which I also use via RetroSMC on our Pi/KODI box running OSMC

Thanks

muldjord

@steptoe said in Versatile C++ game scraper: Skyscraper:

Just wondering if you have any plans to add support for pinball in your scraper

No current plans, no.

I'm aware not many sites have pinball data on their videos or images. emumovies seems to be about the only one that has some videos/images

Skyscraper doesn't support emumovies as they don't have an open API. At least not to my knowledge. I've been looking for one but never found one. It has to be some kind of REST-based API, not just an ftp with images or something like that.

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord Happy holidays! I recently noticed a wrongly scraped game. Would it help if I reported this to you? If so, how should I do it?

That should be reported to the module that delivered the wrong data. I suggest registering with them and fixing it yourself if you can. That's the power of community based game info databases.

muldjord

@steptoe said in Versatile C++ game scraper: Skyscraper:

I'm trying to compiler the latest Skyscraper 3.3.5 for Windows

The howto describes how to compile it under Windows. You shouldn't use QT Creator to do that, just the Qt command prompt as the text you pasted also tells you to. But you're basically on your own here. I don't use Windows myself and quite frankly don't have much interest in that system. Sorry.

Remember that Skyscraper is licensed under the GPL if you end up using any of the Skyscraper code for any of your projects. :) Good luck with it!

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

That should be reported to the module that delivered the wrong data. I suggest registering with them and fixing it yourself if you can. That's the power of community based game info databases.

Ah, I thought that Skyscraper mistook one game for another (but on the second thought, only getting the video wrong then is illogical). I didn't think about checking the data on screenscraper.fr, though. And there it is: the wrong video (from Dangerous Xmas instead of Zettaizetsumei Dangerous Jiisan). I'll report it immediately, I have already an account there. edit: Done.

Thanks for pointing me to something obvious, I plead insanity because of a heavy cold I'm having at the moment. 🥴

Clyde

FYI: TheGamesDB is back online, but it needs a prefix /v1/ to work now.

(Thanks to @maserspark for reporting this.)

muldjord

@Clyde Thanks for the update. It seems they are running from an old backup, but I don't quite understand what they mean by

the API is also back online,
but using the prefix /v1/ which is both to get devs to explicitly make sure old data doesn't just crash their apps,
but this prefix will remain as our main endpoint for this api going forward

Sounds to me like it's only the old API that's online but it doesn't say so specifically.

Either way, great to see that they are slowly getting back online.

Clyde

@muldjord Indeed, it is great. Do you have to update SS to include the prefix?

muldjord

@Clyde Probably not if it's just the old API (which I think it is). I think we'll just have to wait this out until it's back online in v2.

EDIT: After re-reading their message a couple of times I think I get it. The 'v1' will stay, even when they get a more recent backup imported. So I've added the 'v1' prefix, and release 3.3.6.

muldjord

Skyscraper 3.3.6 released: https://github.com/muldjord/skyscraper

Skipped file now includes platform in the filename (Thanks to 'parker-hemphill' for suggesting this)
Now handles the '****' error message from ScreenScraper (Thank you to 'GiulioCentorame' for reporting this)
Updated 'thegamesdb' to include the 'v1' prefix (Thank you to 'Clyde' for letting me know)

This release fixes the 'thegamesdb' module, but be aware that their database is currently running on an old backup due to the December server shutdown. Thank you to @Clyde for letting me know about the 'v1' prefix. In other words: Updating to this release will make 'thegamesdb' work with Skyscraper again, but with old data. They are planning to get back to a more recent version soon, but it will still require the v1 prefix as I understand it, so it will work regardless.

steptoe

@muldjord said in Versatile C++ game scraper: Skyscraper:

Skyscraper 3.3.6 released: https://github.com/muldjord/skyscraper

Skipped file now includes platform in the filename (Thanks to 'parker-hemphill' for suggesting this)

Now handles the '****' error message from ScreenScraper (Thank you to 'GiulioCentorame' for reporting this)

Updated 'thegamesdb' to include the 'v1' prefix (Thank you to 'Clyde' for letting me know)

This release fixes the 'thegamesdb' module, but be aware that their database is currently running on an old backup due to the December server shutdown. Thank you to @Clyde for letting me know about the 'v1' prefix. In other words: Updating to this release will make 'thegamesdb' work with Skyscraper again, but with old data. They are planning to get back to a more recent version soon, but it will still require the v1 prefix as I understand it, so it will work regardless.

Your unofficial Windows version is on 3.30. I'm trying to compile the latest versions but not 100% what I need to compile. You give basic compile instructions for OSX and Linux but not Windows so can't really try SkyScraper fully on the PC. Thanks

muldjord

@steptoe It's not unofficial, it's just not officially supported (I don't use Windows). I know it's old, for that very reason. If I get the time today, I will create a 3.3.7 version of the Windows version.

EDIT: Windows version now updated to 3.3.7. Get it here

o0alucard0o

@muldjord

Thanks so much for this fantastic scraper ;-)

I'm trying to scrape igdb, but with no results

For example

Skyscraper -p steam -i /Volumes/Games/Roms/Steam -g "/Volumes/Games/Attract-Mode/romlists_b" -o "/Volumes/Games/Attract-Mode/scraper/Steam_b" -f attractmode -e "/Users/matteo/Games/skyscript-nuc-one/Steam.cfg" --addext "*.bat" --nosubdirs --nobrackets --unattendskip -s igdb -u IGDBKEY --pretend --verbosity 3

--- The following is a list of skipped games ---
'Cuphead', No returned matches
'GRIS', No returned matches
'Hollow Knight', No returned matches
'Monster Boy And The Cursed Kingdom', No returned matches
'My Friend Pedro', No returned matches
'Neon Chrome', No returned matches
'Nex Machina', No returned matches
'Ori and the Blind Forest Definitive Edition', No returned matches
'PinballArcade', No returned matches
'TrackMania Nations Forever', No returned matches

I'm able to find game through web interface

(sorry for my english)

muldjord

@o0alucard0o I just tried with "My Friend Is Pedro" and indeed something seems to be wrong... I'll look into it, stay tuned.