Versatile C++ game scraper: Skyscraper

SammyBoy

@muldjord One last question are you planning to add 3D boxart in a later version of Skyscraper (not 2.3.0)?

EDIT: Sorry I have another question regarding the screenscraper.fr region problem and that's that SNK VS. Capcom isn't getting scraped from screenscraper.fr whatsoever even though the game is from America I'll check tomorrow and see if it's the because of the filename.

muldjord

@sammyboy I will be looking into this yes. But it won't be the boxard from screenscraper, rather I'll render a 3d box myself from the wheel and cover photo. It won't be in 2.3.0 though.

Concerning your other question, it might be because '-m' isn't set. Try setting it ti '-m 0'. This has also been fixed in 2.3.0 together with the region problem.

SammyBoy

@muldjord That's pretty cool(the 3d boxart) and I'll try scraping neo geo pocket color tomorrow morning. Thank you so much

muldjord

I've decided to include the "gamebox" (a 3D glossy game box) effect in 2.3.0 aswell. It's a lot easier to do than I first thought. It's gonna be really cool!

AnalogHero

@muldjord ...then i need to rescrape again :) I hope its not much work but can you please add a few (common) example artwork.xml files? Maybe you can think of 2 or 3 useful settings (maybe round edges, 3d boxart with drop shadow in the bottom left :)

muldjord

@analoghero Yes, I plan to create a bunch of example artwork[n].xml.example files that will be distributed and installed to the "~/.skyscraper" folder on first run. I also plan to spice up the artwork documentation A LOT with effect examples. It'll take a while to create, but it'll be worth it to understand how everything works.

gamebox effect coming along really nicely btw! It looks really cool, with a glossy finish :)

SammyBoy

@muldjord For the 3D boxart are you going to use a case splice or make generate with the wheel, platform logo and developer logo?

muldjord

@sammyboy I haven't settled on the details yet. I do not have the platform or developer logos available to me, so it won't be that. The front of the box is of course the cover from the game.

EDIT: In a future release I might look into using platform or developer logos on the side. I have implemented a new resource system that allows you to use custom files from the "~/.skyscraper/resources" folder. So you can place png's or other image files in there and use them as masks, frames and the side of the gamebox rendering. So there's nothing holding you back from adding a "megadrive.png" to the resources folder and setting the 'side="megadrive.png"' on the gamebox effect. That would render the megadrive.png on the side of the box. But it would then use that for all platforms of course.

I need to ponder these things some more. But I really need to start nailing this down. I seem to keep coming up with new cool stuff that postpones the release. Maybe it's better to look into these things for future releases simply create the most basic goodlooking box for this release. After all I hadn't even planned to include the 'gamebox' effect in this release in the first place. :D

SammyBoy

@muldjord Can't wait for the release seems like the biggest to date! Can you do a favour for me and add .scummvm as a file format to scummvm because I like scraping from screenscraper.fr but they use a fake file format(.scummvm).

muldjord

@SammyBoy
'*.scummvm' added to scummvm platform. Will be in 2.3.0.

I am currently testing all of the new composite effects. I'm getting real close to being done with all the features I want in 2.3.0, and yes, it is indeed the biggest Skyscraper release to date. And also the toughest to code so far. :D But it's turning out much better than I had expected, so I really look forward to getting it out to you guys. It's a matter of days now unless I run into something unexpected.

AnalogHero

@muldjord I know youre busy with 2.3.0 which is released any moment, but i tried something which doesnt work for me.

Problem: Wings.uae is scraped as Wings of Fury.
Tried Screenscraper Wings.uae -p amiga but it just scrapes everything.

muldjord

@analoghero Are you in the correct folder? I mean, are you in the folder where "Wings.uae" is located? Also, I assume you meant "Skyscraper" not "Screenscraper".

Openretro doesn't seem to have "Wings" on their website which is really odd... So try scraping with '-s thegamesdb' instead. That might help.

AnalogHero

@muldjord Lol ofcourse Skyscraper, just a typo.

Yes im in the right folder, but even with -s thegamesdb added it asks me if i want to overwrite gamelist.xml and if i want to skip existing entries as usual.

muldjord

@analoghero Ah, I know why. You need to put the full path on the filename. I will look into fixing this for 2.3.0 aswell. Thanks for letting me know.

muldjord

Quick update: All of the effects are now done and working. There's one hiccup though. When I scrape a bunch of files with screenscraper I get a sporadic segfault (crash) after about 20 minutes in. Bad news is that it's not the same place each time which means this could take me a while to track down... I'll keep you updated on the progress.

AnalogHero

@muldjord Yes, with full path it works. I wonder why i didnt try that. In fact i tried Skyscraper ./Wings.uae -p amiga -s thegamesdb but that didnt work, too.

EDIT: Im confused now. When i see that the scraper has a name mismatch, as in my examples Wings gets confused with Wings of death or Wings of Fury, id like to rescrape that only game with another scraper (-s thegamesdb for example).

But the correct result doesnt get written to the gameslist.xml. Rescraping with -s localdb doesnt help either, as it sure gets mismatched again.
Tried scraping with -m 100 but this is a 100% match.

Another thing related: Suddenly all my Shadow of the beast games have no number in the name, dont have the right cover orscreenshot. Ill try deleting the dbs and see if it helps.

muldjord

@analoghero Yes, it's because I change directory to "~/.skyscraper" immediately after Skyscraper starts, to make it easier to get to the files I need. But I have a workaround in mind that will allow you to do both "Wings.uae" and "./Wings.uae".

muldjord

@analoghero You need to look at "~/.skyscraper/dbs/[platform]/priorities.xml". That file defines source priorities for the different resources. Currently you probably have openretro above thegamesdb for a bunch of the resource types which makes it prioritize the faulty entries from Openretro. So just move the "<source>thegamesdb</source> up above the "<source>openretro</source>" node for all resources and it should work as you expect.

EDIT: And yes, the gamelist.xml is NOT being written when scraping filenames on command line. It merely adds the data to the local cache so you can use it with localdb as you did.

EDIT2: I will advice against deleting the entire dbs folder. Not that it breaks anything, but if you have scraped all platforms, most of the dbs inside the folder are absolutely fine and will be useful for rescrapings later. Instead, consider deleting just the one folder inside dbs matching the platform you are having trouble with. But, best of all, consider using the prioritize feature as described in the documentation.

AnalogHero

@muldjord I just deleted the amiga folder insided the dbs folder. Its not that big loss.

I was thinking about priorities.xml but its a bit hard to decide, as sometimes it isnt clear which source is better or its depends on the game.

Another thing bugs me for long time: Can the result in number of players somehow streamlined? Most Emulationstation themes cant handle any text. 1 - 2 (2 simultaneous) would only display garbage.

hi_im_daft

Question: I know there's a way to do it, and it's probably posted somewhere but I haven't found it yet.

Can you change the priority for databases when the scraper runs? I don't like the current "primary" database, it throws in a lot of incorrect game names (or maybe just defaults to EU region titles instead of US), so I want to change it to a different one. I'm sure it's in a configuration somewhere to do it.