Versatile C++ game scraper: Skyscraper
-
@bleuge Something seems to be wrong with ArcadeDB, not sure what. I'll investigate.
EDIT: You can use -t 8 if you prefer, it'll adjust acoordingly. Be aware that you might run into problems with ScreenScraper though, as I am a bit unsure if the 4 threads limit I have on now is correct. If I were you I would provide my ScreenScraper user credentials in the config.ini so it's always used correctly for it.
EDIT2: I found the error with ArcadeDB. I had changed some stuff elsewhere that broke it... I'll release a patch shortly that will bring it back in working order.
-
2.3.9 out, please try that
-
@muldjord So fast, thanks you again!
-
@muldjord Compiling the new version right now. Couldnt resist to try out amiberrys new lha feature. Read your conversation with @HoraceAndSpider on the facebook whdload group, and edited the launch script as you said.
I know that amiberry is offtopic here but im excited. :) I think i will swap out my .uae files and gamedata with .lha files.
-
I keep getting Skyscraper hanging and don't have any idea why.
This time stalls while scraping using localdb at half of roms processed in a platform.How is this possible? If scraper is localdb i guess no network connections are being stalled or whatever, just get files from localdb and copy to whatever folder?
Has anyone got this kind of problem? Thanks
Edit: i am using mounted dbs and roms folders in an external HD, but it's working perfectly.Edit2: re-run my scripts and it hangs exactly in the same game:
Scraper: localdb
Search match: 100%
Compare title:'Seaquest'
.....
rest of data...Estimated time: 00:00:28
And it hangs there.
Any idea? Maybe i need to reescrape everything? deleting my dbs?
-
@bleuge Hangs here too. Same situation.
Edit: It hangs also when i try to import something. I put one video files in the import folder and ran skyscraper with -p amiga -s import --videos. It says Sit back, relax and let me do the work! :) forever.
My dbs folder is stored in the normal folder not on a network drive.
Edit2: Looks like it crashes when it cant find resources for a game in the dbs folder. In my amiga folder i have 74 games and 2 demos and it crashes alway when it tries to scrape a demo.
#74/76 (T1) ---- Game 'Worms Directors Cut [AGA]' found! :) ---- Scraper: localdb Search match: 100 % Compare title: 'Worms Directors Cut' Result title: 'Worms Directors Cut' (import) Platform: 'aga' (import) Release Date: '1997-01-01' (openretro) Developer: 'Team 17' (openretro) Publisher: 'Team 17' (openretro) Players: '8' (openretro) Tags: 'Artillery, Multidirectional, Puzzle, Sideways' (openretro) Rating (0-1): '' () Cover: YES (openretro) Screenshot: YES (openretro) Wheel: NO () Marquee: NO () Video: YES (import) Description: () Elapsed time: 00:01:00 Estimated time: 00:01:02 -
I'm really sorry about this guys, I do my best to test things out. Apparently not good enough. Talk about demotivation, these hangs seem to be the bane og Skyscraper even though I keep fixing them. I'll look into it...
-
2.4.0 out, please test and report back. Import and localdb halts should be fixed...
-
@muldjord Tested it. Doesnt hang anymore :)
What doesnt work for me is importing videos. In the past when i added a new game i copied a video with the same name to the import/video folder. Now i added Lotus3_v1.6_0217.lha to the amiga folder, and the video Lotus3_v1.6_0217.mp4 and it doesnt get imported. Also tried Lotus 3.mp4, Lotus III.mp4.
-
@analoghero Glad the hangs are gone. I'll check the video import. I'll focus on fixes for now, so if any of you encounter any more problems, please let me know. I'll take a break from expanding new functionality. I feel like it's time to let it rest and stabilize with what it has.
So please, any problems, let me know. Thank you. :)
EDIT: Just had a thought. Did you remember to add the '--videos' option while using the import scraper for videos? Otherwise it'll ignore them just like it does with the other scraping modules. I realize this can seem a bit silly, so I'll make it auto-add '--videos' if you've chosen the import scraper. Thanks for letting me know.
-
Just found another drastic error in the import scraper. I've been changing some of my overloaded functions and forgotten to cascade the changes to my inherited functions, so they basically broke (which was also the problem with the hangs eariler). It's easily fixable though, so I'll release 2.4.1 with that fixed shortly. I've also checked the remaining overloaded functions just to be sure, and they are all correct, so no more issues like this.
-
2.4.1 out. This should have fixed everything with the import scraper, so please test that out if you have the time. Thanks. :)
-
@muldjord Yes, i added --videos. Compiling new version right now, will test when it finished.
-
@muldjord No, still not importing a correct named video. Have double checked everything. Even when running Skyscraper -p amiga -s import --videos /home/pi/RetroPie/roms/amiga/Lotus3_v1.6_0217.lha it says
pi@retropie:~ $ Skyscraper -p amiga -s import --videos /home/pi/RetroPie/roms/amiga/Lotus3_v1.6_0217.lha ------------------------------------------ Running Skyscraper v2.4.1 by Lars Muldjord ------------------------------------------ Platform: 'amiga' Scraper module: 'import' Input folder: '/home/pi/RetroPie/roms/amiga' Game list folder: '/home/pi/RetroPie/roms/amiga' Covers folder: '/home/pi/RetroPie/roms/amiga/media/covers' Screenshots folder: '/home/pi/RetroPie/roms/amiga/media/screenshots' Wheels folder: '/home/pi/RetroPie/roms/amiga/media/wheels' Marquees folder: '/home/pi/RetroPie/roms/amiga/media/marquees' Videos folder: '/home/pi/RetroPie/roms/amiga/media/videos' Local db folder: 'dbs/amiga' Reading and parsing local database, please wait... Successfully parsed 2019 resources! Looking for optional 'priorities.xml' file in local db folder... Found! Priorities loaded successfully! Pretend set! Not changing any files, but still caching resources in local database. Starting scraping run on 1 files using 4 threads. Sit back, relax and let me do the work! :) #1/1 (T1) ---- Game 'Lotus3_v1.6_0217' not found :( ---- Elapsed time: 00:00:00 Estimated time: 00:00:00 ---- Scraping run completed! YAY! ---- Writing 2019 (0 new) resources to local database, please wait... Success! ---- And here are some neat stats :) ---- Total completion time: 00:00:00 Total number of games: 1 Successfully scraped games: 0 Skipped games: 1 (Filenames saved to '[homedir]/.skyscraper/skipped-import.txt') -
Ah, I know why... It's because the files are using several dots in the name, which makes it look for the file Lotus3_v1.mp4 instead of Lotus3_v1.6_0217.mp4. I'll make it use the full basename instead.
-
2.4.2 out, please test.
-
@muldjord Tested, and it works now as it should. Youre amazing. :)
-
@muldjord Hmm just scraped amiga with localdb and now it skipped Lotus 3.
-
This post is deleted! -
@analoghero Does that here aswell, looking into it...
EDIT: Found the problem, fixing...
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.