Versatile C++ game scraper: Skyscraper

muldjord

EDIT: Nevermind, read your post wrong. Looks ok to me, not sure what is wrong. I recommend using an xml based format instead, try that.

SammyBoy

@muldjord tried that and it is the exact same result (i tried it first)

SammyBoy

@muldjord what should I do?

muldjord

All I can say is that if you've followed the documentation completely, it WILL work. I'm assuming you might have something wrong with where you've placed the files, what you've named them or what command line you run Skyscraper with. And since I don't know any of these things, I can't tell you what to do.

SammyBoy

@muldjord i send pictures and the command I use is Skyscraper -p gba -s import

SammyBoy

Visit my Twitter Take a look at Sammy boy (@samsaju04_boy): https://twitter.com/samsaju04_boy?s=09

SammyBoy

@muldjord the pictures are on Twitter - the first thing you see

muldjord

Please see next post.

muldjord

Please try rerunning it with 'Skyscraper -p gba -s import --updatedb'. That might be the problem.

muldjord

Skyscraper 2.2.6a released: https://github.com/muldjord/skyscraper

Now always sets '--updatedb' when using 'import' scraping module

Get this release, then it'll work without '--updatedb'

SammyBoy

@muldjord Thank you so much!

jura

Hi @muldjord , I used your software for the scraping the images for roms and I loved it.

Seems to work almost OK, sometimes when game names are closely the same e.g. gameName 1 and gameName 2 it messes the description for these games. There's might be a chance that these descriptions are messed up in the location where it get it from at the first place, dunno.

When I checked the Lakka which is another Retro gaming fontend seems that they get images and thimbnails from the libretro. Dunno is it only based on the name of the game.

Any chance to use also this location which is used in Lakka for the images to the games? :)

https://github.com/libretro/libretro-thumbnails/tree/master/

muldjord

@jura It doesn't mess them up, rather it sometimes gets faulty results from the sources, which is a side-effect of the highly automated way Skyscraper works. For instance, if you have a relatively unknown game called "Star Blob" and a game called "Star Fox", it will quite likely get a correct hit for "Star Fox" since it's a well-known game. BUT, it will probably also return the data for "Star Fox" when it searches for "Star Blob" since that's how the source search engines work. So it doesn't mess them up, it just, sometimes, gets faulty results where it seems like the previous game has "spilled over" into the next entry, when in fact it only does so because the names are closely related. In this example by the "Star" in the name.

You can force it to not accept these by changing the minimum match percentage with the '-m' flag on command line. :) It defaults to 50%, which means that "Star Blob" and "Star Fox" gives a positive result.

I can't use a github repository for scraping, simply because I do not have the permission from github to do so. Github is not meant to be used for game media scraping and I highly doubt they would allow it if they knew people used it for that purpose.

EDIT: On a sidenote, I plan on completely rewriting the way matches are achieved, so matches will improve in a future version. No ETA though.

muldjord

A huge thank you to @chipsnblip for spreading the word about Skyscraper on reddit. I notice these things, and I appreciate it a lot! :)

chipsnblip

@muldjord it's my pleasure to help! i've been lurking in this thread since day one, and to tell the truth haven't even used skyscraper yet (but plan on it very soon-- life keeps getting in the way). i was just so impressed watching one person's hard work and dedication to a project of this nature. seeing people pour their heart into the retro gaming community is truly inspiring.

muldjord

On a related note: If any of you good people feel Skyscraper is a project you want to support, then a good way to do so is simply to spread the word about it. :) I'm not exactly a PR guru myself, so I'll concentrate on the coding part of it. When people do lend a hand it means quite a lot, and is quite motivating.

As you know Skyscraper is completely open source and free to use (I'm a huge advocate of openness in general). I do this work entirely in my spare time simply because I enjoy working on it. Part of that is watching the community actually make use of the project. So a thank you to all of you for simply using it is in order aswell. :)

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

As you know Skyscraper is completely open source and free to use (I'm a huge advocate of openness in general).

Let's drink to that. o/ᵁ

muldjord

@clyde *clink* ᵁ\o

AnalogHero

I really think skyscraper should included in Retropie-Setup. Im sure i mentioned this in a different topic.

AnalogHero

@muldjord Could it be that version 2.2.5 is a lot slower than previous versions? Just added a few whdload games and rescraped my amiga folder. I thought it crashed because the first output took sol long. Scraping with localdb takes the same amount of time. Over 10 minutes for 61 .uae files...