Versatile C++ game scraper: Skyscraper

AnalogHero

@muldjord Compiling right now. Will test when done. Maybe you can do test updates in a seperate git branch, compiled with an argument of update_skyscraper.sh.

@bleuge Three ` will display a code block. View http://commonmark.org/help/ for more information.

Edit: Looks like it worked now. Glad that i could help. Someone has to test it all out.

AnalogHero

@muldjord If i swap out my amiga collection with .lha files, will skyscraper recognise them as the same game? For example i delete Agony.uae and replace it with Agony_v1.3_0960.lha, will skyscrape use the avaiable local data from the dbs?

muldjord

@analoghero No, it will differ. If you switch to LHA's completely I would recommend deleting the entire amiga db folder as the cached resources would be of no use anymore. Skyscraper detects games from their sha1 checksums. In cases where it is a script (such as .uae) it will sha1 the filename instead of the file contents, since .uae's might change contents.

EDIT: Actually that is an interesting observation. If you change/delete your romset completely or partly, you would probably want to purge localdb of all resources not related to your roms. So in a future update, when I've had a few days for R&R, I will add the option of purging the cache of resources not related to the current roms you have in your input folder.

AnalogHero

@muldjord Hmm. Its a lot of work then, because i manually renamed the videos to match my .uae collection, now i have to do it again to match the .lhas. Good that i picked only the games i really like to play, and not just the entire collection.

muldjord

@analoghero Yeah, not much to do about that I'm afraid. Hopefully the LHA collection will be quite stable in its contents so it won't be necessary to do it again in the future.

bleuge

@AnalogHero Amiga newbie here, as i read, i assume the way to go in Amiga for Raspberry is lha?
Also I think if i want to mount an Amiga project, Amiberry is the best? Any tips building an image for Amiga?
Sorry the offtopic :D

@muldjord updated to latest, rerun my script, it seems it's working, sometimes it does very long pauses with no output, but for now it's running

AnalogHero

@bleuge I would recommend to wait a bit longer as RetroPie-setup hasnt updated yet to support the latest amiberry version. In the very near future it should be as easy as copying the games (in .lha format) to the /roms/amiga folder.

muldjord

@bleuge The long pauses are expected, that's because the threads run in parallel and sometimes sync up which gives those pauses in-between results. ;)

jlfernandez

Sorry if this has already been answered (I couldn't find a reply) but how can I get just a simple cover instead of any fancy artwork? I use videos whenever available so I don't need a screenshot or a rotated box or anything else. Just a simple 2D cover will do.

Thanks!

muldjord

@jlfernandez Artwork documentation is here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md

Basically what you want is for Skyscraper to export a screenshot that uses the cover as a layer.

So you'd create a '[homedir]/.skyscraper/artwork.xml' something like this:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" width="640" height="480">
    <layer resource="cover" height="480" align="center" valign="middle"/>
  </output>
</artwork>

Alternatively, if you want it exported directly as the cover, then just use the following:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="cover" height="480"/>
</artwork>

That will export the cover (as a cover) with a height of 480 pixels and width relative to that.

That should get you started. :)

bleuge

Skyscraper is so flexible i think many users don't use to its fully potential xD

I think this is correct: if i have already scraped my sets, my localdb is around 15-16GB. And i want to do things like this, export only the cover, re-build a graphic composition, etc... I just need to modify the artwork.xml file and do a -s localdb rescrape, am i right? Graphics will be rebuild and overwritten in the media folders?

Thanks

AnalogHero

@bleuge Id say yes. Thats one point for having a good localdb. If you have all needed images already downloaded, you can edit artwork.xml, run skyscraper with -s localdb. Not tested, but it should work offline, too.

bleuge

@muldjord Regarding --dbstats

As I have localdb mounted in external HD, I must use

Skyscraper --dbstats -p arcade -d /home/pi/RetroPie/dbs/arcade

I thought that I could only use -d /home/pi/RetroPie/dbs , but I must include platform also.

What I was strange is that I was testing --dbstats and did a wrong commandline

Skyscraper --dbstats -p arcade -d /home/pi/RetroPie/dbs/c64

So, checking c64 localdb with arcade platform, it showed data, don't know exactly what this means?

Don't you think is best to simply use the path of localdb 'without' platform (as already you have stated the platform you want the stats) and so this "bug" ? "feature"? is showing correct data.

Also, I know you have tons of ideas pending in Skyscraper, but I have two requests
Maybe --dbstats could just run without specifying platform and directly show the stats for the entire localdb?

Also, I've been thinking how could I know what games are missing data. I could process gamelist with an external tool.
But could this be interesting in Skyscraper?

A run of dbstats without platforms, between scraping sessions, could indicate users how many new data have you gathered in the new scrapping. And you could now how complete is your data sets with the --showmising command

Example:

Skyscraper --showmissing -> show report of missing datas/covers/etc for all platforms
Skyscraper --showmissing --verbosity 1 ->show missing datas/covers/etc for all games on each platform

I've been building a set of what I call "the best games" for console/arcade platforms, i've worked on this for many months, some platforms as snes o nes, need to be reduced because contains more than 200games in my initial selection and i think is too much. Also, i've included translatations, interesting hacks, etc...

Sorry for trying to give you more work xD But I love Skyscraper and currently I know is the best scraper around with a big difference.
The day you could implement MobyGames I think I am going to explode with excitement (imagine scraping TDC#14) this could be totally crazy

As always thank you very much and sorry the walls of text

muldjord

@bleuge I'm not gonna work on Skyscraper for a while :)

You can easily get the missing games simply by scraping the platform with any scraper and adding --pretend. It will load all of the data from cache, so it doesn't use network bandwidth from the sources. And the ones it doesn't have it will add to the [homedir]/.skyscraper/skipped-[module].txt files. :)

And yes, you have the basic idea of Skyscraper correct. The idea is, as you say, that you can change any of the options for Skyscraper, and simply rescrape with localdb. If you do so, you will get 99% the same result as scraping with the web sources all over again. So it saves a tonne of bandwidth for the sources and is a lot faster. So basically you can, in theory, redo your entire collection in Emulationstation just by rescraping with localdb. I don't think many other scrapers give you this option, if any.

The reason you have to provide a path for localdb is that they are platform specific. Having one database for all platforms would make it extremely slow, since I have to do lookups in ALL resources for each and every rom. It would make Skyscraper almost unusable.

Nothing keeps you from setting a "wrong" platform with --dbstats though if you also use '-d', it will display the stats correctly. :) I can see how that's confusing, but just think of it in a broader perspective. A localdb cache is not really platform specific. It is rom collection specific. But since everything is tied to platforms when you talk emulation, so does Skyscraper.

EDIT: MobyGames is high on my priority list. Just keep in mind that they have very strict connection limits, which basically means it is only usable to scrape about 10 roms an hour. But you could use it to fill in the gaps I think. So if you have one game that just doesn't scrape with the others, then you could scrape it on command line with MobyGames and maybe get a hit.

But yeah, for now, I'm taking a break. :)

muldjord

@analoghero Yes, that is exactly correct. Skyscraper is 100% capable of scraping offline if all games are in the local cache. :)

bleuge

@muldjord jeje ok, understood, you need a very well deserved rest!
Thanks for all the good work and enjoy!
P.D. Yes, skyscraper is unique featuring a localdb. That was a genius idea.

AnalogHero

@muldjord I know youre taking a break, and i dont want to disturb, so consider this as a thing you do when you like to. When scraping amiga, the names of the files (minus extension) are displayed in ES instead of the gamename. Example: Agony_v1.3_0960 should be Agony.
Dont know if other systems are affected too.

muldjord

@analoghero Either you're using '--forcefilename' or it hasn't been found. Those are the only two reasons for a game to show up with the filename. :) Otherwise, if it was found as "Agony" it will be shown as "Agony".

AnalogHero

@muldjord Strange issue. I never used --forcefilename.

#4/63 (T2) ---- Game 'Agony_v1.3_0960' found! :) ----
Scraper:        localdb
Search match:   100 %
Compare title:  'Agony'
Result title:   'Agony_v1.3_0960' (import)
Platform:       'Amiga' (thegamesdb)
Release Date:   '1992-01-01' (openretro)
Developer:      'Art and Magic' (openretro)
Publisher:      'Psygnosis' (openretro)
Players:        '1' (openretro)
Tags:           'Animalprotagonist, Autoscroll, Horizontal, Powerup, Shootemup, Sideways' (openretro)
Rating (0-1):   '0.7' (thegamesdb)
Cover:          YES (openretro)
Screenshot:     YES (openretro)
Wheel:          NO ()
Marquee:        NO ()
Video:          YES (import)

Description: (thegamesdb)
This horizontally-scrolling shoot 'em up features six long levels, all with detailed and mellow background graphics, aiming for a less hectic feel than contemporaries such as Project X. As a magician's apprentice, you have been turned into an owl to give you the best chance of destroying the many dark creatures to be faced, and thus discovering the secret of cosmic strength.

These dark creatures include piranhas, giant ants and mosquitoes. Extra weapons and invincibility periods can be collected. The technical details include 3 layers of multi-directional parallax scrolling, background animation, and different title and in-game music.

Elapsed time: 00:00:04
Estimated time: 00:01:10

Heres a example from my gamelist.xml:

 <?xml version="1.0"?>
<gameList>
  <game>
    <path>/home/pi/RetroPie/roms/amiga/Agony_v1.3_0960.lha</path>
    <name>Agony_v1.3_0960</name>
    <cover />
    <image>/home/pi/RetroPie/roms/amiga/media/screenshots/Agony_v1.3_0960.png</image>
    <marquee />
    <video>/home/pi/RetroPie/roms/amiga/media/videos/Agony_v1.3_0960.mp4</video>
    <rating>0.7</rating>
    <desc>This horizontally-scrolling shoot &apos;em up features six long levels, all with detailed and mellow background graphics, aiming for a less hectic feel than contemporaries such as Project X. As a magician&apos;s apprentice, you have been turned into an owl to give you the best chance of destroying the many dark creatures to be faced, and thus discovering the secret of cosmic strength.

These dark creatures include piranhas, giant ants and mosquitoes. Extra weapons and invincibility periods can be collected. The technical details include 3 layers of multi-directional parallax scrolling, background animation, and different title and in-game music.</desc>
    <releasedate>19920101</releasedate>
    <developer>Art and Magic</developer>
    <publisher>Psygnosis</publisher>
    <genre>Animalprotagonist, Autoscroll, Horizontal, Powerup, Shootemup, Sideways</genre>
    <players>1</players>
  </game>

muldjord

@analoghero Ah, that's because the title is prioritized from the "import" module. Just remove the "<source>import</source>" line from the "<order type="title">...</order>" in your priotities.xml under "[homedir]/.skyscraper/dbs/[platform]/priorities.xml" and rescrape with localdb.

:)