Versatile C++ game scraper: Skyscraper
-
@analoghero No worries I have plenty of files to test on :)
-
Just implemented '--startat' and '--endat' options that allow the user to define what files to begin and end at when scraping. In this mode it ONLY caches data, it doesn't change the game list.
This allows you to only scrape a span of files. So if you have 1200 files, but only want to scrape the middle 60, then you can do so by, alphabetically, defining the first and last file in that span. Makes sense? Useful? I know I will be using it myself at least. :D
-
@muldjord That will be very useful. It would be great for me where I'm trying to merge some files into a set, but unfortunately those files aren't in a contiguous group (i.e. some start with a, some b, some c, etc.) However, I will use it to portion out updates on some of the 6,000 ROM set systems so I can do it by letter and check rather than have it run and then go back to see where it stopped, what was missed, etc.
Thanks!
-
@timekills When i add roms to a system, i just run skyscraper and select yes to skip existing entries.
Flipside is that you can do this only one time with only one scrapermodule. So if you want to scrape with another module, you have to scrape all files.
-
Skyscraper version 2.3.7 released: https://github.com/muldjord/skyscraper
- Now checks for .lha suffix and adds spaces where appropriate to get better results
- Improved returned image data validity check (libpng errors still happen, but can be ignored)
- Rewrote the worker to main thread communication a bit
- Implemented '--startat' option that tells Skyscraper the first file to scrape
- Implemented '--endat' option that tells Skyscraper the last file to scrape
- Added thread id to terminal output
- Applied serverside artwork size limit to openretro module to avoid running out of memory
- Improved network communication class
Another "bits and pieces" release. The inclusion of '--startat' and '--endat' should make it easier to do scraping of a subset of your games. This has been requested a few times and I've made use of it myself A LOT during testing.
2.3.6 had some issues with the new openretro parser since some of the returned covers are INSANELY large. Like, 10000x10000 resolution. That in conjunction with the new alphabeticized queue system made it eat up ALL of the Pi ram really fast, which in turn made the kernel kill it off to ensure system stability. This has been fixed serverside, simply by asking for a resize before it reaches Skyscraper.
The changes to the network communicator are kind of beta. Which means that I've tested it, and haven't seen any problems. But I've removed the clearAccessCache call again since it should no longer be necessary. That call ensured that data didn't get mixed up, but was a bit of a workaround. My new code should ensure that data doesn't get mixed up without using that call. So pleeeeease, if you encounter game media getting mixed up between games, let me know! Shouldn't happen though.This was a bit of a tough one. I've been spending the last 4 days trying to figure out what the hell was causing the crashes, until I stumbled upon those insane cover artwork resolutions from openretro. Then it became quite clear that Skyscraper wasn't the problem at all...
Anyways, all my testing has gone well on my end, so please do update and try it out.
Happy scraping! :)
-
Skyscraper version 2.3.8 released: https://github.com/muldjord/skyscraper
- Implemented user credentials ('-u user:password') to set up threads for 'screenscraper' module
- Made sure artwork output gets exported, even if entry has no base artwork resource
- Changed 'verbose' to 'verbosity' to allow levels and made terminal output more useful overall
- Added '--dbstats' command line option that prints stats for the selected local dabatase cache
- Added '--purgedb' command line option that allows purging resources from localdb
- Fixed bugs in mergedb command line option
- Fixed bug in Simle Mode where 'attractmode' would not work properly (thank you Humayun)
Fixed a few bugs, added a few options, and FINALLY included the '-u' option properly. If you have a ScreenScraper user, please provide that using the format '-u user:password'. Remember that you can set this in '[homedir]/.skyscraper/config.ini' so you don't have to have it on the command line.
You can now check the stats of the local database cache with '--dbstats'. If you want to purge stuff from it, do this with '--purgedb' by adding 'm:[module]' and/or 't:[type]'. You can also have both comma-separated.
Example: 'Skyscraper -p amiga --purgedb m:thegamesdb,t:cover'. This will purge all covers from thegamesdb module from the Amiga platform's local database cache.
Another example: 'Skyscraper -p amiga --purgedb m:thegamesdb'. This will purge all resources from thegamesdb module from the Amiga platform's local database cache.
Last example: 'Skyscraper -p amiga --purgedb t:cover'. This will purge all cover resources from any module from the Amiga platform's local database cache.
Lastly, I've changed '--verbose' to '--verbosity [level]' where [level] can be 0-3. The higher the level, the more output it will give you while scraping.As always, please report bugs if you encounter any.
Happy scraping! :)
-
@muldjord Thanks for updating and new additions xD
I was going to write that don't know why but initial testings in 2.3.7 didn't work for me.This line for example:
Skyscraper -d /home/pi/RetroPie/dbs/arcade --nosubdirs --noresize --updatedb -t 8 -p arcade --videos --unattend --skipped -s arcadedb --pretend
just showed me 720.zip game not found (the first rom in the arcade folder)
print the elapsed and remaning time
and just hangs there, no more output, nothing, after 10min i hit ctrl+cI thought my roms or dbs folders are not ok, but yes, they are ok
Anyways, I updated a pair of hours ago to 2.3.7, and then ... I'll test 2.3.8 version
I remember you said generating a log for debugging purposes it was a bit difficult in the actual skyscraper, but if this could be added, debugging and spotting errors could be possible much much easier? What do you think?
Thanks you very much for so fast support!
Edit: tested 2.3.8, the same thing, I'll try another commandlines for other platforms, no idea what the hell is going wrong here.... Also i don't know why it can't find 720 in arcadedb...
The message i am getting is
#1/141 (T1) Pass 1 ---- Game '720' not found :( ----
Then the times and that's all...Edit 2: skipped arcadedb and tried
Skyscraper -d /home/pi/RetroPie/dbs/arcade --nosubdirs --noresize -t 8 -p arcade --videos --unattend --skipped -s openretro --pretend
It's working.
Btw, i use a generic bash script tailored to my platforms, i use -t 8 in all, i know this number is adjusted for specific sites, so, no problem using -t 8 in a generic way ?
-
@bleuge Something seems to be wrong with ArcadeDB, not sure what. I'll investigate.
EDIT: You can use -t 8 if you prefer, it'll adjust acoordingly. Be aware that you might run into problems with ScreenScraper though, as I am a bit unsure if the 4 threads limit I have on now is correct. If I were you I would provide my ScreenScraper user credentials in the config.ini so it's always used correctly for it.
EDIT2: I found the error with ArcadeDB. I had changed some stuff elsewhere that broke it... I'll release a patch shortly that will bring it back in working order.
-
2.3.9 out, please try that
-
@muldjord So fast, thanks you again!
-
@muldjord Compiling the new version right now. Couldnt resist to try out amiberrys new lha feature. Read your conversation with @HoraceAndSpider on the facebook whdload group, and edited the launch script as you said.
I know that amiberry is offtopic here but im excited. :) I think i will swap out my .uae files and gamedata with .lha files.
-
I keep getting Skyscraper hanging and don't have any idea why.
This time stalls while scraping using localdb at half of roms processed in a platform.How is this possible? If scraper is localdb i guess no network connections are being stalled or whatever, just get files from localdb and copy to whatever folder?
Has anyone got this kind of problem? Thanks
Edit: i am using mounted dbs and roms folders in an external HD, but it's working perfectly.Edit2: re-run my scripts and it hangs exactly in the same game:
Scraper: localdb
Search match: 100%
Compare title:'Seaquest'
.....
rest of data...Estimated time: 00:00:28
And it hangs there.
Any idea? Maybe i need to reescrape everything? deleting my dbs?
-
@bleuge Hangs here too. Same situation.
Edit: It hangs also when i try to import something. I put one video files in the import folder and ran skyscraper with -p amiga -s import --videos. It says Sit back, relax and let me do the work! :) forever.
My dbs folder is stored in the normal folder not on a network drive.
Edit2: Looks like it crashes when it cant find resources for a game in the dbs folder. In my amiga folder i have 74 games and 2 demos and it crashes alway when it tries to scrape a demo.
#74/76 (T1) ---- Game 'Worms Directors Cut [AGA]' found! :) ---- Scraper: localdb Search match: 100 % Compare title: 'Worms Directors Cut' Result title: 'Worms Directors Cut' (import) Platform: 'aga' (import) Release Date: '1997-01-01' (openretro) Developer: 'Team 17' (openretro) Publisher: 'Team 17' (openretro) Players: '8' (openretro) Tags: 'Artillery, Multidirectional, Puzzle, Sideways' (openretro) Rating (0-1): '' () Cover: YES (openretro) Screenshot: YES (openretro) Wheel: NO () Marquee: NO () Video: YES (import) Description: () Elapsed time: 00:01:00 Estimated time: 00:01:02 -
I'm really sorry about this guys, I do my best to test things out. Apparently not good enough. Talk about demotivation, these hangs seem to be the bane og Skyscraper even though I keep fixing them. I'll look into it...
-
2.4.0 out, please test and report back. Import and localdb halts should be fixed...
-
@muldjord Tested it. Doesnt hang anymore :)
What doesnt work for me is importing videos. In the past when i added a new game i copied a video with the same name to the import/video folder. Now i added Lotus3_v1.6_0217.lha to the amiga folder, and the video Lotus3_v1.6_0217.mp4 and it doesnt get imported. Also tried Lotus 3.mp4, Lotus III.mp4.
-
@analoghero Glad the hangs are gone. I'll check the video import. I'll focus on fixes for now, so if any of you encounter any more problems, please let me know. I'll take a break from expanding new functionality. I feel like it's time to let it rest and stabilize with what it has.
So please, any problems, let me know. Thank you. :)
EDIT: Just had a thought. Did you remember to add the '--videos' option while using the import scraper for videos? Otherwise it'll ignore them just like it does with the other scraping modules. I realize this can seem a bit silly, so I'll make it auto-add '--videos' if you've chosen the import scraper. Thanks for letting me know.
-
Just found another drastic error in the import scraper. I've been changing some of my overloaded functions and forgotten to cascade the changes to my inherited functions, so they basically broke (which was also the problem with the hangs eariler). It's easily fixable though, so I'll release 2.4.1 with that fixed shortly. I've also checked the remaining overloaded functions just to be sure, and they are all correct, so no more issues like this.
-
2.4.1 out. This should have fixed everything with the import scraper, so please test that out if you have the time. Thanks. :)
-
@muldjord Yes, i added --videos. Compiling new version right now, will test when it finished.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.