Versatile C++ game scraper: Skyscraper

muldjord

@incunabula I'm considering options for this, same thing I am working on with Used2BeRX. The problem here of course being that people tend to have varying ways of saving these files so I would need to handle "all of them" so to speak.

If would be really cool if you could sortof "import" a snap folder into the localdb and then just scrape using the 'localdb' module and it would use those snaps. But all local db resources are identified by sha1 checksum, and the snap images that are not easily identifiable other than from their filenames. And that poses a big problem.

I think a first implementation will be to define an xml format where your snaps would need to be in. Then you can scrape from that. That would then automatically add it to the localdb. So when you scrape from 'localdb' afterwards, it would use those snaps aswell as any other snaps you might have acquired.

EDIT: I realize what I just wrote might seem a bit fuzzy :D Bottom line is that I am considering it. And if I find "the right way"(tm) I will implement it.

muldjord

@incunabula said in Versatile C++ game scraper: Skyscraper:

That makes sense - the checksum could be anything depending on what application was used to compress the file, what level of compression was used, etc. Ok, i'm fine with unzipping these roms (rather small file sizes already) but i can confirm that MSX (using lr-bluemsx) and GG (using lr-genesis-plus-gx) both do in fact work with zipped roms.

Exactly. But I will add .zip to both MSX and GameGear, no problem.

EDIT: zip support is implemented in this release, feel free to test it: https://github.com/muldjord/skyscraper/archive/a820fbd8368372c09c5f3a00338dbf040bc23e43.zip

incunabula

@muldjord I understand, and i don't mean to be pushy by requesting new features. Just suggesting things that might be useful to others. So far 3 out of 3 platforms are working OK using the --nobrackets switch. :)

muldjord

@incunabula Suggestions are always more than welcome. :)

incunabula

Is adding support for other platforms as simple as adding additional if/else statements in the Skyscraper::run() function? I know nothing about C++ but have done a bit of VB .net so i apologize if this is an ignorant question or if i'm using the wrong terminology hehe :)

muldjord

@incunabula Almost. Adding new platforms is probably the easiest thing to do in Skyscraper. :) As long as the platform doesn't have any special demands. If it's just an ordinary platform with ordinary file formats that need to be run, then I can do it real easy. Just let me know which ones you'd like in there.

muldjord

Skyscraper 1.7.0 released: https://github.com/muldjord/skyscraper

MAJOR: Fixed and refined 'attractmode' frontend implementation, now works in a basic manner
'attractmode' can now skip existing entries
'emulationstation' now properly add brackets to 'name' on skipped entries
Added check for 'db.xml' when doing '--cleandb'
Refactored GameEntry variables
Changed GameEntry from struct to class
Added 'Overall title similarity' to final output
Added 'Overall completeness' to final output
Code refactoring here, there and everywhere
Now accepts results where we have low editDistance, but high similarity (For instance "Disney's Darkwing Duck" with fileName "Darkwing Duck").
Added '--nobrackets' option that disables and [] and () tags in the frontend game titles. (Thanks for the feedback 'incunabula')
Fixed bracket parsing
Now always uses completeBaseName since some filenames have more than one '.'
Completely rewrote sorting algorithm. 30 lines became one with a nifty C++11 lambda :D
Added zip format to GameGear and MSX platforms
Now uses filenames for output image files again

Have a great weekend everyone!

muldjord

Skyscraper 1.7.1c released: https://github.com/muldjord/skyscraper

Moved all source files to 'src' folder
'[homedir]/.skyscraper' is now default folder for all files used by Skyscraper
'/usr/local/bin/Skyscraper' is now default location for Skyscraper executable
Refined '--help' output a bit
Fixed lemon64 scraping
Added 'lemonamiga' scraping module
Added '--skipped' command line option
Added 'make install' for correct installation of files

This is more of a clean-up release. First of all, I've rewritten the compile and install procedure in the readme on github, so be sure to use it exactly as written. Basically what I've added is a "sudo make install" command, which will take care of installing the files in the correct locations on your Pi. This ensures that each time you compile a new version of Skyscraper, it will install it in the same place, thus perserving all of your configs and local dbs properly. It was a bit of a mess before, sorry about that.

In other news, I've fixed the lemon64 scraper. I've also added the 'lemonamiga' scraping module. I've also added the '--skipped' option which allows you to always include skipped entries in the resulting gamelists. It is especially useful for attractmode.

As always, if you run into trouble, let me know. I expect a few bugs in this release, since I've moved a lot of code around and refactored quite a bit to enable the '--skipped' option.

Have fun!

muldjord

Skyscraper 1.7.2 released: github.com/muldjord/skyscraper

Added 'uvlist' scraping module (http://www.uvlist.net)
Added rating resource and support
Added rating support to lemonamiga
Added rating support to lemon64
Added rating support to mobygames
Added rating support to uvlist

Enjoy!

UDb23

@muldjord said in Versatile C++ game scraper: Skyscraper:

I think a first implementation will be to define an xml format where your snaps would need to be in. Then you can scrape from that. That would then automatically add it to the localdb. So when you scrape from 'localdb' afterwards, it would use those snaps aswell as any other snaps you might have acquired.

I would suggest something really simple as first implementation: an "import to localdb" function that will take the scrape images from a local folder. The key as you mentioned should be: image file name = rom filename.

muldjord

@UDb23
Ok, so here's what I am thinking: I will create a folder called "~/.skyscraper/import". In this folder there will be a folder per supported import type. So there will be a folder called "snaps" and a folder called "boxart". If you place files inside this folder with the precise filename of one of the rom files you are going to scrape, and set the scraper to '-s import', it will then import these images into the localdb with the source of "import" or similar. Then, when you scrape using 'localdb' afterwards, it will use those files. So basically the import procedure becomes a scraper alongside 'mobygames' and others. The difference is that is scrapes using files from the "~/.skyscraper/import" folder.

I think this is a pretty good solution. It's pretty straightforward to do this with snaps, boxart, videos and other media files. I'll need to figure out how to do it for other resources. But let me start with media files.

How does that sound?

muldjord

@UDb23
Aaaan here it is:

Skyscraper 1.7.3 released: https://github.com/muldjord/skyscraper

Added 'import' scraper, scraping from resources located in '[homedir]/.skyscraper/import' folder
Added 'developer' support for 'uvlist' scraper
Improved html unescaping a lot
Cleaned up xml escaping

The 'import' release. So the big new thing in this update is that I now allow you to import your own 'snaps' and 'boxart' into the local database using the '-s import' scraper. It's a local scraper that scrapes from '[homedir]/.skyscraper/import' resources. Place your images in the folders within with the EXACT name of any rom you will be scraping, and it will import that image into the local database. Then, afterwards, scrape with the '-s localdb' scraper to make use of the artwork in your frontend of choice. (Read more under "Local data import" in the Github readme)

I will expand the import functionality over time to also allow importing of textual data. But for now, snaps and boxart seemed like the most important ones.

Also, I realize that Skyscraper now has a bunch of very abstract features that might seem a bit confusing, so I plan to make some videos explaining how to properly make use of the powerful features Skyscraper now provides. Simply scraping using one scraping module is just the basic usage. You can get much, MUCH better results very simply by also making use of the '-s localdb' scraper AFTER you've scraped with a bunch of web sources. They'll all be stitched together to form near-100% perfect results for the frontend you choose.

Anyways, more on this when I get the time to make those videos. :D

That's all for now, enjoy! :)

UDb23

@muldjord Great news ! Thanks .
This week I should be able to dedicate time to test both VIC20 scraping and local image import functionalities.

muldjord

@UDb23 Awesome! I'll probably expand it further during the coming days. I have some nifty ideas that will also allow for textual imports using a format defined in a definition file the user creates. More on this later. :)

AnalogHero

Hi,

first of all i must say thank you for this tool. It works great.
I have a question about the localdb. If i scrape a system it creates a folder with screenshot and covers in the dbs folder. Now lets say in the first run i end up with a few games missing either box art or screenshot.
How can i update the games which are missing something? Can you give some command line examples how to do that?

muldjord

@AnalogHero
Thank you, glad you find it useful!

It is quite simple. There are two options for getting the remaining screenshots and covers.

Scrape with more of the webscrapers, for instance 'Skyscraper -p [platform] -s mobygames' or 'Skyscraper -p [platform] -s screenscraper'. Then rescrape with 'Skyscraper -p [platform] -s localdb' afterwards. Each time you scrape with a web scraper, all of the data is cached. So the more web scrapers you've scraped a given platform with, the more complete results you'll get when you rescrape with 'localdb' afterwards.
The second option is to import your own covers and screenshots. You do this by placing your files in the '[homedir]/.skyscraper/import' subfolders. Then run "Skyscraper -p [platform] -s import". As long as the filenames of your images files match EXACTLY the rom filenames (without the suffix of .nes or similar of course), it will then import the images you provided into the localdb. Then rescrape with 'Skyscraper -p [platform] -s localdb' and it will then use the images you've imported.

Also, be sure to read the entire readme on github. It is all explained in more detail there. I plan to do some easy-to-understand videos when I get the time and motivation (I don't really like making videos, I'm more of a coder-dude).

Hope this made sense. :)

AnalogHero

Ahh i see. Wasnt sure how the cached data will be handled after scraping with more than one websource. Thanks for your help.

Edit: Wow. This works amazing. I was using sselphs scraper, but i liked the mixed screenshot/boxart/logo from the screenscraper tool. But screenscraper is windows and the pictures are a little too small when using the nes mini theme from @ruckage .
Now this is a perfect solution for me.

If i had one whish free: A small gui would be nice. (with info whats in your localdb and stats about your scraped games and whats missing)

muldjord

@AnalogHero A gui is something I'd really like to have. I am an experienced gui programmer aswell, so I certainly got the skills for it. Just need to get the command line version to a level where I feel I can leave it for a while. Then I might look into a gui. No promises, but it's certainly something I'd like to do.

AnalogHero

@muldjord Maybe i found a bug. Cant scrape mastersystem, it says no entries to scrape although i have games in there for sure. Paths seem to be correct.

Also looks like gamedatabase has blocked me. Anyone knows if they will unblock me ever again?

muldjord

@AnalogHero I got unblocked from gamesdatabase again, but never used it since. I took the hint :)

If it says no entries to scrape, probably you chose to skip existing? Or maybe your roms suffixes aren't supported. Where are your files located and what suffix do they have? 'mastersystem' only supports '.sms' according to the official RetroPie wiki, so that's what Skyscraper looks for.

If you're using zipped rom files I suggest unzipping them, even if the emulator supports zipped files. There are a couple of disadvantags to zipped roms. The major one is that many pieces of software uses the sha1 checksum of the rom data to recognize the rom. If you use a zipped rom, this feature is completely negated since a zipped rom can vary in size and content. It is always best to use the raw rom files in my opinion. But it is just that, an opinion. :) I merely wanted to point out why I feel zipped roms don't make much sense from a frontend and scraping perspective.