Versatile C++ game scraper: Skyscraper

muldjord

I'm really sorry about this guys, I do my best to test things out. Apparently not good enough. Talk about demotivation, these hangs seem to be the bane og Skyscraper even though I keep fixing them. I'll look into it...

muldjord

2.4.0 out, please test and report back. Import and localdb halts should be fixed...

AnalogHero

@muldjord Tested it. Doesnt hang anymore :)

What doesnt work for me is importing videos. In the past when i added a new game i copied a video with the same name to the import/video folder. Now i added Lotus3_v1.6_0217.lha to the amiga folder, and the video Lotus3_v1.6_0217.mp4 and it doesnt get imported. Also tried Lotus 3.mp4, Lotus III.mp4.

muldjord

@analoghero Glad the hangs are gone. I'll check the video import. I'll focus on fixes for now, so if any of you encounter any more problems, please let me know. I'll take a break from expanding new functionality. I feel like it's time to let it rest and stabilize with what it has.

So please, any problems, let me know. Thank you. :)

EDIT: Just had a thought. Did you remember to add the '--videos' option while using the import scraper for videos? Otherwise it'll ignore them just like it does with the other scraping modules. I realize this can seem a bit silly, so I'll make it auto-add '--videos' if you've chosen the import scraper. Thanks for letting me know.

muldjord

Just found another drastic error in the import scraper. I've been changing some of my overloaded functions and forgotten to cascade the changes to my inherited functions, so they basically broke (which was also the problem with the hangs eariler). It's easily fixable though, so I'll release 2.4.1 with that fixed shortly. I've also checked the remaining overloaded functions just to be sure, and they are all correct, so no more issues like this.

muldjord

2.4.1 out. This should have fixed everything with the import scraper, so please test that out if you have the time. Thanks. :)

AnalogHero

@muldjord Yes, i added --videos. Compiling new version right now, will test when it finished.

AnalogHero

@muldjord No, still not importing a correct named video. Have double checked everything. Even when running Skyscraper -p amiga -s import --videos /home/pi/RetroPie/roms/amiga/Lotus3_v1.6_0217.lha it says

pi@retropie:~ $ Skyscraper -p amiga -s import --videos /home/pi/RetroPie/roms/amiga/Lotus3_v1.6_0217.lha
------------------------------------------
Running Skyscraper v2.4.1 by Lars Muldjord
------------------------------------------
Platform:           'amiga'
Scraper module:     'import'
Input folder:       '/home/pi/RetroPie/roms/amiga'
Game list folder:   '/home/pi/RetroPie/roms/amiga'
Covers folder:      '/home/pi/RetroPie/roms/amiga/media/covers'
Screenshots folder: '/home/pi/RetroPie/roms/amiga/media/screenshots'
Wheels folder:      '/home/pi/RetroPie/roms/amiga/media/wheels'
Marquees folder:    '/home/pi/RetroPie/roms/amiga/media/marquees'
Videos folder:      '/home/pi/RetroPie/roms/amiga/media/videos'
Local db folder:    'dbs/amiga'

Reading and parsing local database, please wait...
Successfully parsed 2019 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!

Pretend set! Not changing any files, but still caching resources in local database.


Starting scraping run on 1 files using 4 threads.
Sit back, relax and let me do the work! :)

#1/1 (T1) ---- Game 'Lotus3_v1.6_0217' not found :( ----


Elapsed time: 00:00:00
Estimated time: 00:00:00

---- Scraping run completed! YAY! ----
Writing 2019 (0 new) resources to local database, please wait... Success!
---- And here are some neat stats :) ----
Total completion time: 00:00:00

Total number of games: 1
Successfully scraped games: 0
Skipped games: 1 (Filenames saved to '[homedir]/.skyscraper/skipped-import.txt')

muldjord

Ah, I know why... It's because the files are using several dots in the name, which makes it look for the file Lotus3_v1.mp4 instead of Lotus3_v1.6_0217.mp4. I'll make it use the full basename instead.

muldjord

2.4.2 out, please test.

AnalogHero

@muldjord Tested, and it works now as it should. Youre amazing. :)

AnalogHero

@muldjord Hmm just scraped amiga with localdb and now it skipped Lotus 3.

muldjord

This post is deleted!

muldjord

@analoghero Does that here aswell, looking into it...

EDIT: Found the problem, fixing...

bleuge

@muldjord Thanks very much for the fixes, going to update and test again.
@AnalogHero How do you capture your output? with all the colors and so? xD

Also, I want to share this:
https://twitter.com/WhaddupNico/status/976887812959195136

This guy has done fantastic logos for lots of games, for SNES, GBA, N64, DC, and PS1. He share the packs in mega. Logos are in vector format and in transparent PNG. Vectors are great because as you already know we can scale up to 4k or whatever.
These logos are not in any online db, but could be used with local data import of Skyscraper. Remember to rename the file as the rom you want to use it.

I still not tried the graphic composer but will have a look...

I am downloading all the packs now.

muldjord

2.4.3 out, please test... I really need to figure out a better way to put out test releases, this is getting silly. :)

EDIT: @bleuge, those logos are pretty amazing. Guy's doing a fantastic job!

EDIT2: Just updated 2.4.3 code again. Hope you didn't update in the meantime. Found a flaw with the fix I just made... And fixed the fix. Meta!

muldjord

I think I'll create rcX releases from now on. Releasing 2.4.x releases seems silly to me when new issues keep popping up.

Thank you for your help with testing @AnalogHero I appreciate it a lot.

AnalogHero

@muldjord Compiling right now. Will test when done. Maybe you can do test updates in a seperate git branch, compiled with an argument of update_skyscraper.sh.

@bleuge Three ` will display a code block. View http://commonmark.org/help/ for more information.

Edit: Looks like it worked now. Glad that i could help. Someone has to test it all out.

AnalogHero

@muldjord If i swap out my amiga collection with .lha files, will skyscraper recognise them as the same game? For example i delete Agony.uae and replace it with Agony_v1.3_0960.lha, will skyscrape use the avaiable local data from the dbs?

muldjord

@analoghero No, it will differ. If you switch to LHA's completely I would recommend deleting the entire amiga db folder as the cached resources would be of no use anymore. Skyscraper detects games from their sha1 checksums. In cases where it is a script (such as .uae) it will sha1 the filename instead of the file contents, since .uae's might change contents.

EDIT: Actually that is an interesting observation. If you change/delete your romset completely or partly, you would probably want to purge localdb of all resources not related to your roms. So in a future update, when I've had a few days for R&R, I will add the option of purging the cache of resources not related to the current roms you have in your input folder.