Versatile C++ game scraper: Skyscraper

muldjord

Yes, it's the softness/blur of the shadow. But I personally prefer a softness of 5, so just keep it at that (I might put a max of 5 on it in future releases).

muldjord

I am currently looking into an optimized algorithm for the shadow renderer. If all goes well, it should provide the same result with a huuuuuge speed improvement. So "softness=10" would become usable.

EDIT: New algorithm is almost implemented and will be in 2.2.7. The speed improvement is insane! Thousands of times faster, literally.

muldjord

The new optimized gauss shadow rendering is now fully implemented and the results are in! For 21 roms, rendering 1 shadow with a softness of 10 for each:

Old unoptimized version 2.2.6: 3 minutes and 26 seconds
New optimized version 2.2.7(still in testing): 4 SECONDS!

Now, THAT's what I call a proper optimization! :D HAHA, I love this.

Keep in mind that this is for 1 shadow with a softness of 10. The more shadows you have, the more time is saved. And also, a softness of 10 is ridiculous, so most often you'd use a softness of 2-5, which is even faster!

2.2.7 will be released as soon as I've done some testing. :)

muldjord

Skyscraper 2.2.7 released: https://github.com/muldjord/skyscraper

Optimized shadow rendering A LOT! WROOOOOOOOOM!!!
'simple mode' 'Do you want to run the script' answer now defaults to Y
Fixed bug in 'simple mode' where answering 'y' to adding brackets would actually remove them

If you took note of the drastic slowdown in 2.2.6 when using shadows, this release will bring you to nirvana. Shadow rendering has been huuuugely optimized. Like, it went from minutes of rendering, to a few seconds. Please try it out and let me know what you think.

Happy scraping! :)

AnalogHero

@muldjord Tested it, and i have to say that its really fast now. Lol ... 6 seconds for my 18 psx games. Awesome work. Thanks for it.

Newellj79

Pardon my ignorance. Is there a way to upgrade to the newest version from my current version?

AnalogHero

@newellj79 Yes, just follow the instructions on github. It will install over the old version.

hi_im_daft

Dumb question -- after finishing the scrape, there were a few games that were skipped, and it said it was added to "skipped-localdb.txt". Where is that file placed? I tried checking through the Skyscraper directory and couldn't find it anywhere.

muldjord

@hi_im_daft All Skyscraper files are located in "/home/pi/.skyscraper"

SammyBoy

@muldjord I think that Skyscraper isn't properly scraping some platforms from screenscraper.fr because I was scraping roms from the famicom disk system and none of them had a marquee however on the website there was a marquee and no games get scraped on arcade, fba, ect but I think this is due to having multiple arcade categories. Do you think you can fix it? Because I find it really annoying having to scrape from import because it is quite time consuming. Thank you

muldjord

@sammyboy Just add '--region jp' to the command line. The famicom disk system was only released in japan, and I don't check for those by default.

EDIT: I'll add it as one of the regions it tries automatically by default to 2.3.0. But please use '--region jp' until then. Version 2.3.0 won't be released for another week or so.

EDIT2: You mention 'arcade' and 'fba' aswell. Please be more specific about what you are trying to do. Give me an example filename from the platform that isn't working, so I have a chance of reproducing it.

SammyBoy

@muldjord any arcade rom for example the last blade 2

muldjord

@sammyboy Last Blade 2 scrapes just fine with the default 'arcadedb' module here.

SammyBoy

@muldjord but not with screenscraper.fr and I don't think arcade db has wheel art

muldjord

@sammyboy Then it's because screenscraper doesn't have the sha1 summed file in their database. It checksums the file, for instance "lastbld2.zip" and asks for it at screenscraper. It doesn't provide a result so it probablt doesn't exist for the arcade platform on screepscraper. If there's another platform it DOES exist for on screepscraper, please provide the platform. Then I can add it as a alias which should make it work.

SammyBoy

@muldjord Do you think it's anything to do with the way screenscraper.fr has categorised the arcade metadata for example instead of one big arcade section it is built up of cps3, Konami, ect

SammyBoy

And @muldjord it scrapes from neogeo which is where I tried to scrape from but it didn't work

muldjord

@sammyboy Yes, that is the reason. I just found the list myself, and I will add all of them as aliases in 2.3.0. But please be patient, 2.3.0 is not going to be released just yet.

SammyBoy

@muldjord OK thank you so much. And can you also find out why some marquees are not found when scraping from screenscraper.fr? Sorry to be such a bother

muldjord

@sammyboy Please read the message further up. I already answered that. It's because fds is 'jp' region only. This is also fixed in 2.3.0 but you can remedy it already by adding '--region jp' to the command line.