Versatile C++ game scraper: Skyscraper

StephanePare

Oh, I was also wondering if it would be possible to add the colecovision and pc engine console to the list of supported platforms

muldjord

@stephanepare Yes, Skyscraper automatically moves "The" to the end of names to make sure they are sorted correctly.

I will look into including colecovision and pc-engine soon. It's easily done. Thanks for the feedback.

paradadf

Hi @muldjord and thanks for the amazing job!
I've read your arguments about adding zip file support and would really appreciate you to reconsider.
The compression standards idea doesn't matter if you use no-intro romsets, which are pretty much the standard. It is not usual at all that you have more than one rom inside the zips too. People that have very large collections do save tons of space. Even if you you just have few roms, that should apply most of time for every single file. and even if you still think it is not the best to have compressed roms, there are lots of people that have a different opinion and right now they would need to do much stuff manually.
Best regards and thanks for bringing another scraper to the game!

LocVez

I don't suppose it would be possible to have an 'all' option for the -s switch that goes through every scraper for you for your selected system and then does the localdb one last to tie all results from all scrapers together?

I'm just not sure if I sould be overwriting the gamellist.xml and skipping games that already exist or not to try and get as complete a collection as possible and should I be running localdb after every other scraper finishes or can I continue using every scraper.

Thanks in advance.

muldjord

@paradadf I agree, I'll add zip support for all platforms.

muldjord

@LocVez I suggest simply scripting it to do so. That's what I do myself. :) And yes, just overwrite the gamelist.xml for each runthrough and don't ever skip. If you skip existing it won't add the resources for the games that have already been found. So overwrite, never skip.

When you run it with localdb last, it will create a "perfect" one using all resources from earlier scrapings. :)

muldjord

Skyscraper 1.8.2 released: https://github.com/muldjord/skyscraper

Added support for 'coleco' platform
Added support for 'pcengine' platform
Added zip support to all platforms per request from users

2 new platforms added and support for zip files for all platforms. :)

Have fun!

A Former User

Just tried this out for the first time by scraping my NES collection.

Wasn't terribly pleased with how it came out so I wanted to get rid of everything and rescrape with a different scraper. However, no matter what I do, I can't get rid of the descriptions and images this downloaded.

I have deleted the gameslist and downloaded_images from /opt/retropie/configs/all/emulationstation. There is nothing in those folders but the images and descriptions still show up in EmulationStation.

I have rebooted. Is there anywhere else these are stored? Thanks!

muldjord

@livefastcyyoung Read the readme. It's all well described in there. If you check the output of Skyscraper at the beginning, you'll also notice where the data is saved.

Concerning your trouble with descriptions you don't like, you can either delete the entire local nes database cache by deleting '~/.skyscraper/dbs/nes' and start over. I wouldn't do that though, I instead recommend simply finding out which source you prefer to have the descriptions coming from, and prioritizing that source for description in '~/.skyscraper/dbs/nes/priorities.xml', and then scrape using '-s localdb' afterwards.

Feel free to ask further questions, but please read the readmes thouroughly first. :) Thanks.

A Former User

I have deleted the entire /.skyscraper folder and it's still displaying the Skyscraper scraped art and game info. I'm at a loss.

muldjord

@livefastcyyoung Deleting Skyscrapers local resources won't alter your gamelist for EmulationStation. You need to delete the Emulationstation specific gamelist.xml and images folder to do that. As I mentioned, both are listed at the beginning when you scrape the platform.

A Former User

I have done that. There is no folder for NES in the downloaded_images and the gameslist.

I guess I'll have to just keep digging. I have deleted these before with no issue to rescrape. I have never had this issue before.

muldjord

@livefastcyyoung Those are not the right folders. Unless you set them manually when scraping, the correct folder is ~/RetroPie/roms/nes/images and ~/RetroPie/roms/nes/gamelist.xml. Doesn't it say that when you run the scraper? It should clearly state a gamelist and images folder at the very beginning.

Btw, you DON'T need to delete those to rescrape. Just choose to overwrite and NOT skip. But please, read the readme's.

LocVez

@muldjord Thank you for your advice on scripting the scraping. I have now done so but, alas, now find my IP banned from the games database. According to https://www.gameex.info/forums/topic/10930-gamesdbase/ - Would it be possible to add emumovies as a scrape resource, not only do they promote sharing but they are partnered with the games database and I have a lifetime sub with emumovies too....

Thanks again!

muldjord

@LocVez Yes, gamesdatabase will ban you if you overuse it.

I had a quick look at emumovies, I'll ask them for developer access to the api and let you know if I get it. Then I will include it if the API they provide fits with Skyscraper. It probably does, but I don't know atm. :)

Rion

@muldjord They probably will. Just as long as you can input your username and password.

Been a lifetime member there for ages.

Rion

Opps dobbel post..

LocVez

@muldjord Many thanks!

AnalogHero

@LocVez They banned me in the past, but after 2 weeks it worked again. Im glad it wasnt forever.

muldjord

Wooooooo!!! Video tutorial!!!

Please spread the word! :)

Please, please, PLEASE check this out if you are new to Skyscraper. I have a feeling most people only use Skyscraper to just scrape once using the default scraper and then leave it like that. It really is too bad, since the most important feature in Skyscraper is the local db cache. This video explains why it's important.