Versatile C++ game scraper: Skyscraper
-
Skyscraper version 2.7.0 released: https://github.com/muldjord/skyscraper
- Added scraping module specific configs in config.ini to allow different userCreds per scraping module
- Added 'igdb' scraping module
- Added '--interactive' mode which will ask user to choose the best result
- Changed release date patterns to regexps for cleaner code
- Fixed bug which caused <video> tags to disappear when skipping entries with emulationstation (thank you to ozy24 for reporting this)
- Reached commit number 1000!!! I'm gonna celebrate by coding some more
- Conformed [homedir] to be '~' everywhere
- Added several aliases to platforms to better support all scraping modules
- Added option for custom header when initiating network requests
- When generating search names, all instances of ' v.X' and ' rev.X' are now removed prior to search
- Added '--unattendskip' option to allow for always skipping existing entries without user input (thank you to ozy24 for suggesting this)
You can now set the '--interactive' flag which will cause Skyscraper to prompt for the correct entry whenever a search based scraping module is used (mobygames, thegamesdb, igdb, openretro). For 'screenscraper' and 'arcadedb' this is not enabled as they are checksum and id based, which means they are always correct, so no user input needed.
I've also added the 'igdb' module which needs a key. Request one at their site if you wish to use it. The module/api has some caveats meaning that it won't scrape any media at all as I can't guarantee that the returned media is for the requested platform. I've managed to sortof get around another caveat which means that publisher and developer is not connected to a platform either, so I only use those if only one is returned. But hey, it's another tool in the toolbox. :)
Please try this new release out and let me know if you encounter any errors or inconsistencies. I've done a fair bit of testing myself, but it's software so, you know, bugs/glitches! :)
-
@muldjord Awesome. Need to try the interactive mode asap.
-
@muldjord Tried it. The interactive mode is awesome. I wish this would somehow work for screenscraper. But i understand its crc based...
All i can say is that this scraper has to be included with retropie.
-
@analoghero Awesome! Well, I can enable it with screenscraper aswell, but I don't really see how it would be useful as there will always only be one entry to choose from since its checksum based. If you have any thoughts on this, I'd be happy to hear them.
-
Wow, I just went to check out EmuMovies. The community on there is doing some pretty amazing work!
I just downloaded the wheels pack for the Sega Master System, plopped them onto my Pi in the '~/.skyscraper/import/wheels' folder and ran "Skyscraper -p mastersystem -s import" and then "Skyscraper -p mastersystem -s localdb" (-s localdb not really necessary as it is default if it's left out) to make use of the imported data and BAM! I now have wheels for about 70% or my Mastersystem roms. That is pretty amazing. All I had to do was to make sure that my rom naming conventions match that of the pack, which is often Hyperspin.
You guys should seriously check that out.
-
@muldjord OK, here are some images that better explain my intentions and my problem.
As you can see by the Atari 2600 Adventure "gamebox" with its spine created from a "spine image" resource, the sizing is off. (I used the wrong spine color intentionally to demonstrate the slight gap in the top and bottom, which I cannot fix).
For point of reference, I've included an Adventure gamebox that was generated with a wheel resource.
An Atlantis "gamebox" shows the other problem with using a "spine image" as the resource: the side and front colors don't match up.
Finally, I'm including a mixed-art image of what I'm creating for all the systems: screenshot, wheel, gamebox, and marquee (the cartridge).
Again, I love what you've done and cannot tell you enough how much I appreciate your help. Even if you tell me it's impossible, I'll be happy to end this quest for another!
-
@muldjord Maybe there is a way to scrape not matching single roms with the screenscraper module by using a override flag and use a user provided checksum.
For example:
Skyscraper -p megadrive -s screenscraper /home/pi/RetroPie/roms/megadrive/Sonic.zip -crc AC400F6A -
@dorkvader Ah, yes, I see the problem. Let me think about this. I should be able to add some attributes to "gamebox" that will allow this to work. Great idea to use the marquee resource for cartridge btw, I really like that. Awesome!
@AnalogHero Not a bad idea, I'll give this some thought aswell and find a proper way to implement it. I'll probably just call it '--checksum [sum]' as it doesn't really matter if it's the sha1 or the md5 sum people input. Also, I'll disable it if the user tries to scrape more than 1 rom.
-
@muldjord Exactly what i thought. Just for single roms.
Then you can scrape amiga .lhas which arent in their database using a checksum from an .adf entry :)
-
@dorkvader 2.7.1 out now which can do what you are looking for. Just add the 'sidescaling="both"' attribute to the gamebox xml note, and it will autoscale the image to completely fit both width and height. That should do the trick. Please try it out and let me know if it does what you expect.
@AnalogHero Concerning the checksum option, I have decided against it for the simple reason that it is a screenscraper only option, and there's not a lot of payoff with it. For most roms you could just as easily just rename the input file, which would take just as long as getting the sha1's. With that said, for the LHA files that you suggest, I am currently investigating if I can get all of the Retroplay LHA's added to screenscraper. Then we can start linking them to the correct games. :)
-
THANK YOU SOOOOO MUCH, Muldjord! I am excited to try it out and will let you know the results ASAP!
-
@muldjord said in Versatile C++ game scraper: Skyscraper:
sidescaling="both"
Works BRILLIANTLY! Thought I'd really test it with a Vectrex box, and as you can see it worked beautifully, matching the graph lines as well as one can expect.
Thank you for your help and for your generosity in time and work. If you have some kind of "donation jar," I will gladly throw a tip your way.
-
@dorkvader No need man, but I appreciate the sentiment :) Feel free to donate something to the RetroPie project though. Or, if you insist, I have a patreon at https://www.patreon.com/muldjord :)
Looks great btw! I love that idea and I am probably going to be doing the same thing for some of my platforms. Probably just using the marquee (with the spine graphics) for the spine instead, and then using a custom artwork.xml file for those platforms that I have them for. Thank you for the inspiration.
-
@muldjord I see your point. Im also not sure if its worth the effort. But there are situations were renaming isnt a good option. (If you have already savestates or config files for some games).
Also you could not only do a checksum option but a general override option (- override {gameid OR gamename}). So it could be used for more than one scrapingmodule.
-
@analoghero I'll consider it. It's definitely not a bad idea, I just have to make sure it makes sense for Skyscraper as a whole and that the implementation in the code doesn't bloat it. :S
-
@dorkvader Those are awesome man.
Any chance you could hook me up with your templates for those? I'd love to make 3D boxart for all of the Xtras I've been making. The only template I have though makes every game look like a generic DVD case. It still looks good, but it's miles away from what you're showing here.
-
@used2berx Sure thing. I'm finishing the Vectrex spines tonight and will be happy to zip my template and or/completed images (spine and boxes) or both. The system has about 25 officially released titles and I've more than tripled that with public domain homebrews. I'm also doing that with all the home consoles from the Golden Era o' Video Games. Just email me.
-
@dorkvader Thanks man. That would be great. :)
Not sure where to find your email though?
-
Can someone help me in installing Skyscraper to RetroPie on Ubuntu? muldjord mention that install is exactly the same as is on the PI. I have done the steps listed in the GitHub and I am getting this error:
$ wget -q -O - https://raw.githubusercontent.com/muldjord/skyscraper/master/update_skyscraper.sh | bash
bash: line 3: curl: command not found
--- Fetching Skyscraper v. ---
--2018-09-03 22:34:48-- https://github.com/muldjord/skyscraper/archive/.tar.gz
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-09-03 22:34:48 ERROR 404: Not Found.I am assuming that I should also install this in the RetroPie folder?
-
@easye9inches said in Versatile C++ game scraper: Skyscraper:
I am assuming that I should also install this in the RetroPie folder?
No, you can run it from anywhere you have write access. You need to install
curland then re-run the installationsudo apt-get -y install curl
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.