Versatile C++ game scraper: Skyscraper

MaroonOut09

@muldjord Thank you very much for your help!

muldjord

@maroonout09 You're welcome. Good luck with it! :)

fnkngrv

Just for reference we have also been testing this on our RetroPie base image for the Odroid XU4 and it does work well. The only item of note that we have found is that with that board a lot more folks use small EMMC or microSD cards for the base and then an ext drive for their games/media. With how the db is storing what we can tell are duplicates in the cache for quicker results when performing a rescrape it is easy to chew up the remaining space on the OS "drive" and filling it very quickly. Excellent work tho with how great the metadata that is returned for the gamelists and also the media itself.

muldjord

@fnkngrv Thank you, glad you like it. You can change the dbFolder with '-d' and I will make sure it can be set in the config.ini file for the next release aswell. Then you can create a config.ini and add the 'dbFolder="[db base folder]"' in the main section of it, and it will put the cache there for all platforms in subfolders. That should give you the dynamic you are looking for. Will be in 2.4.4.

AnalogHero

@muldjord Can you control in the sourcecode which image a scraper module downloads? When scraping amiga with openretro it sometimes gives strange results. For example RickDangerous_v1.3_2294.lha returns a screenshot from the trainermenu. ! It looks Like this but i cant find it on openretro.

Since screenscraper isnt an option with lha files, i manually imported some to replace the strange ones.

muldjord

@analoghero I found the flaw. It seems that it returns all screenshots, including those from the cracked versions, most of them are just hidden but still exist in the source. So my function to return the screenshot even looks through the hidden ones. I'll fix this in 2.4.4 so it only chooses between the main ones. Thank you for reporting this. :) It's really helpful!

AnalogHero

@muldjord Feeling bad for disturbing your break from Skyscraper. Dont know how but if i can help you with development in any other form then reporting minor bugs dont hesitate to ask. Maybe i can do something.

muldjord

@analoghero Reporting those bugs are really important, so it is immensely appreciated on my end. I wanna focus on the stability for a while, to make sure as many people as possible using Skyscraper have a problem-free experience with it.

I've cut down from about 3 hours daily to about 1 hour daily working on Skyscraper now. It seems to work pretty well for me. And with the help of you and others reporting problems, it has gotten really stable it seems, which makes me happy and motivates me to do further work. :)

Currently working on the MobyGames module. As mentioned earlier, the module won't be included in the Simple Mode scripts, simply because their limits would make it unusable. But it will be perfect for filling in the gaps for a few roms at a time.

EDIT: Oh, and I got my hands on a B+, so really looking forward to "starting over" and getting everything set up again. :)

edwardthehuman

How can I enable to show just the cover then after that the video?

AnalogHero

@edwardthehuman You can do this by editing the artwork.xml file in /home/pi/.skyscraper

Im no expert with editing this xml files. Please read: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md for more information.

AnalogHero

Happy easter to you and your families. Enjoy your time. Hope you dont find any empty easter presents, as its also aprils fools day.

edwardthehuman

Thanks @AnalogHero

Happy Easter to everyone!

muldjord

Skyscraper version 2.4.4 released: https://github.com/muldjord/skyscraper

Changed openretro 'wheel' type to 'marquee' type. It wasn't really a wheel
'openretro' now filters out variants to avoid bad screenshots (Thank you AnalogHero)
Now allows you to set custom dbFolder in config.ini both under main and platform sections
For Amiga it now searches for "Demo", "aga", "cdtv" and "cd32" in filenames and converts to [] bracket names (Thanks Dom)
Region 'wor' has been moved below 'eu' and 'us' for Screenscraper since it often returns Japanese titles (Thanks maroonout09)
Now turns '-' into ' ' in filenames before doing searches (Thanks maroonout09)
Simplified thread result communication with main thread from two functions to just one
Now again properly tests for a scraping run with 30 consecutive misses, then exits

Just wanted to get this out there while I am working on the mobygames module. In other words, the mobygames module is NOT implemented in this release.
Instead there's some user requested stuff and also some minor fixes. It's all there in the release notes. :)

bleuge

@muldjord You liar... xD
Thanks you very much!

muldjord

@bleuge I did take two days off ;)

AnalogHero

@muldjord Thanks for the update. How can i take advantage of the AGA brackets for amiga? When i run Skyscraper -p amiga --videos it doesnt change names.

muldjord

@analoghero If you're using the lha's and they have "aga" in their filename, it will create [AGA] brackets for them, unless you have '--nobrackets' defined.

AnalogHero

@muldjord Ok it works. Somehow i didnt update skyscraper facepalm

steptoe

Is there any way that this could be compiled for the Windows OS and a simple GUI to access ROMs via the PC instead of via commabd on the Raspberry Pi

If I wanted to do a full scrape of all my ROMs, while I try to get to grips with GNU emacs on Win7 so I can see a 'live' update as per your YouTube video so I can see exactly what changes do, is there a quick command via the Raspberry Pi to enable a full complete scan with snapshot and snapshot video (similar to sselph scraper) until I customise the your scraper for how I finally want it to look. I'm think then, I can either add to or use the scraped images and videos for a local cache to drastically speed up future scrapes and customizing

How did you manage to get the live update of the screenshot sample in your YouTube video, this would help me a lot in trying out different configs to get the one I'm happy with

Thanks

muldjord

@steptoe Somewhere in this thread there is a guide on how I made the "live view" but it is for Ubuntu Linux, which is what I run at home. I do not have a guide for Windows I'm afraid (and there won't be one as I don't use Windows).

You can compile it for Windows, but there will be tweaking to get it to work properly I think. I haven't done much testing of this myself, again, as I'm not really a Windows user.

But Qt5 is cross-platform, so with some tweaking it can be done. Just remember NOT to try and compile it through Cygwin, that will definitely fail. Instead use the Qt environment command prompt link that ships with the Qt5 installer.

Please read the documentation on the github page concerning locally caching data. Skyscraper already caches all data for later use, to significantly speed up the process of re-scraping. So whenever you have the artwork set up as you like, just scrape each platform with '-s localdb' and it will scrape using all cached data.