Versatile C++ game scraper: Skyscraper

[muldjord]

Skyscraper 2.0.3 released: https://github.com/muldjord/skyscraper

• Added '--lang' and '--region' command line options. Defaults to 'us' and 'wor' if none is provided.
• Language and region support now implemented for 'screenscraper' module.
• Removed 'stats' option as it didn't make much sense anymore.

You can now set language and region which will be used currently by the 'screenscraper' module only. In a later release I will implement the option to get a list of supported languages and regions. But for now I just wanted to enable it. It defaults to language 'us' and region 'wor' if none is provided.

[mitu]

@muldjord said in Versatile C++ game scraper: Skyscraper:

Either way, the psx platform won't work with the screenscraper module, simply because the raspberry pi doesn't have enough memory to checksum the iso files.

That's not necessarily true - the md5sum standard utility can operate in a constant memory space, no matter how large is the input stream/file. The checksumming does not require the whole file contents to be in memory, MD5 requires only reading sequentially 512 bits from the input.

[muldjord]

@mitu Thanks for pointing this out. I actually started doing sha1 and md5 summing differently in 2.0.2. Instead of feeding the functions the entire file data I feed it bits of 1 Meg until it's done. So that would actually mean that it has already been fixed. :)

[AnalogHero]

@muldjord Works with psx now. A little slow, but no problem if you dont have a big psx collection like me.

Thanks for the update. :)

[muldjord]

@analoghero Glad it works :)

I'm gonna start work on a gui now. This will be a whopper so it'll be a while before the next release.

[BenMcLean]

EmulationStation needs a way to quickly delete incorrect scraped info on a game in the GUI from the gamepad without deleting the game. That's more important than the actual scraper being improved IMO. But this is still great though. Thanks for all your hard work.

[A Former User]

@benmclean Yr barking up the wrong tree as @muldjord doesn't develop for ES.

@muldjord Is there a feature, or are you planning on a feature to scrape a single element and append the gamelist? Say, I wanted the title and genre from GamesDB, published and developed info from Screenscraper, etc.

[muldjord]

@livefastcyyoung There's not a feature to just scrape one title per se. But you can always just copy just 1 rom to a different folder, set the input folder to that folder with '-i'. Then scrape that rom using the different modules 1 by 1, then prioritize the resource type in '~/.skyscraper/dbs/[platform]/priorities.xml' and finish by scraping using '-s localdb'. That would give you exactly the result you are asking for, except it would have created a new gamelist.xml file with just that entry in it, so you'd have to copy the files over manually afterwards.

If you are in doubt about the features I mention, please check the readme's on github. It's all very well explained in there (although I am aware that it can seem confusing since Skyscraper can do so many things! Hence why I am working on a gui).

I do not have plans to enable scraping just 1 rom from a folder in a simple way. It doesn't work well with the current design of the code and how it creates the gamelist.xml, sorry.

[A Former User]

@muldjord I think I was misunderstood. I wasn’t referring to a single rom. I was referring to scraping a set of roms as we can now but setting a flag to just scrape the genre, release date, etc.

I’ve found no one database has all elements in great shape so putting together a gamelist by scraping the elements that database does best seems like the next best alternative.

Hopefully that made sense now.

[AnalogHero]

@LiveFastCyYoung Im not sure that this is what youre looking for but you can set priorities in the priorites.xml, but its only for a whole system. You cant say use info A on rom A and info B on rom B afaik.

A gui will really help here imo.

[muldjord]

@livefastcyyoung Makes perfect sense now. That is not currently supported by Skyscraper but it can do just what you want in a similar way. Have you read up on the '-s localdb' scraping module contained in Skyscraper? You can basically do what you are referring to simply prioritizing the resources correctly. It's all in the readme's. :)

[muldjord]

@AnalogHero Exactly, beat me to the punch :)

[AnalogHero]

@muldjord Tested c64 with screenscraper module. Doesnt produce any good output, infact none (aborts after 30 fails) although there are c64 games in the screenscraper database online. Any idea?

[muldjord]

@AnalogHero Basically c64 has a shitstorm of different versions of every single game available (I've literally seen 1 game with 130 different d64 images available, and that's just for the d64 format!). It is not very useful to scrape it with screescraper in my experience. But I'll look into it later and see if something is in fact buggy. :)

[AnalogHero]

@muldjord Yeah, true. Scraping with crc doesnt make sense here. Each crack would have a different checksum. Maybe going by name is better. Is this possible with screenscraper?

[muldjord]

@analoghero Unfortunately not. They only support checksum search and EXACT filename match (which is also one of the parses I currently do). So you'd have to use thegamesdb.

[muldjord]

Ok, so I am looking into creating a gui for Skyscraper, and I've been looking for a lightweight gui widget framework that works on the RetroPie distro without too much fuzz. I am currently using the Qt framework for a lot of the stuff in Skyscraper, which is really useful. BUT, I can't for the life of me get Qt to work without X11, which it can, but it just freezes my Pi.

So, any suggestions for lightweight gui frameworks? I thought about doing some with ncurses or similar, but I reeeeeally would like to have a graphical one and not a text-based one.

[mitu]

@muldjord Try running some of the samples from AGAR, it seems it can run of SDL and seems to support both framebuffer and GL based contexts.

EDIT: Scratch that, It seem the project is not active anymore and the SDL version supported is 1.2.

[Deltax5]

how do I install it. WHere do I put the files? I put the commend and says it can't find gt5-defalt packege. You need to explain where to put the files.

How to install Skyscraper

Install prerequisites

Install this package:

• $ sudo apt-get install qt5-default
• [enter your 'pi' user password, default is 'raspberry']
  I skiped this since it doesn't work.

Download and compile

Create a folder for the Skyscraper source, download the latest release, compile it and install it:

• $ cd /home/pi
• $ mkdir sources
• $ cd sources
• $ wget https://github.com/muldjord/skyscraper/archive/2.0.3.tar.gz
• $ tar xvzf 2.0.3.tar.gz
• $ cd skyscraper-2.0.3

Doesn't find these

• $ qmake
• $ make
• $ sudo make install
• [enter your 'pi' user password, default is 'raspberry']

[muldjord]

You don't put the files anywhere. Just follow the guide. Type in the commands exactly as they are shown but without the '$' and it will work if you are using a RetroPie image. If not, you are doing something wrong.