Versatile C++ game scraper: Skyscraper
-
2.7.0 is shaping up really nicely. I've added an interactive mode set with '--interactive' which will ask the user instead of auto-choosing which result is best. This has been requested a few times and I really find it useful if you're struggling to get good data on a few roms.
I've also added the 'igdb' (http://igdb.com/) scraping module which is nearly finished. The api for it is pretty good, but it has some huge caveats which means I can't scrape any media for it and it also has issues with scraping publisher and developer. The module needs a user-key so each user will have to request one from them. And there's also a request limit of 100 requests per month for a free membership. You can pay for more if you like. The limit is actually 3000 for normal requests, but I have to use what they call "expanded requests" to get more than just the integer id's of the data which is limited to 100. Quite a difference there. -
@muldjord said in Versatile C++ game scraper: Skyscraper:
Does this seem sensible to you guys?
Absolutely, great idea! Having an extra tool can never hurt, and the user limit should somewhat counteract the key limit from moby.
-
First-time poster, long-time lurker ... I absolutely love Skyscraper, muldjord. After spending hours, days, weeks, months, and probably a year seeking art, creating it when there was none, and then combining multiple components for thousands of custom images, I have tossed aside that laborious process and the results for the impressive uniform look that Skyscraper provides and my all-too-eager and all-too-meager skills don't.
Praise and limited skill set aside, my question: Is there ANY WAY to remove the black outline on the "gamebox" image? I'm talking specifically about the black bar on the top and bottom of the generated gamebox front and the gamebox side. I have tried myriad ways to either stumble my way through this dilemma or work around it. (For example, I trimmed the black border almost entirely from Skyscraper's included .png images, only to have the .png front and side box return after a scraping.)
I have failed in every attempt.
My goal is to use gamebox "spine" art -- title specific when possible, generic platform template as fallback . Unfortunately, the spine art when substituted for the "wheel" art doesn't line up with the box front cover -- e.g, those Atari 2600, 5200, 7800 silver boxes with the rainbow-like bar on the front and side will not align as a single ban. The two sides come justcloseenough so that they should be together but just far enough apart that it's glaringly apparent that they don't.
Thank you, again, for you amazing contribution. And for anyone reading this who hasn't used Skyscraper, trust me when I say, if you really are into the look of your game artwork, spend the few hours or much less to learn and use Skyscraper, rather than the hours and more (much, much more) you will spend doing it yourself: amassing, creating, and then building your artwork collection by hand.
-
@dorkvader First of all, thank you very much for the nice words, I appreciate that a lot :) I am glad you find Skyscraper useful.
Concerning your question, there is currently no way of removing the black bars, and I am having a bit of trouble understanding what you mean. Could you post a screenshot where the flaw is present, so I can figure out what to do about it? That would be great :)
-
@muldjord THANKS! I'm at work at the moment but will post a screenshot tonight.
-
Skyscraper version 2.7.0 released: https://github.com/muldjord/skyscraper
- Added scraping module specific configs in config.ini to allow different userCreds per scraping module
- Added 'igdb' scraping module
- Added '--interactive' mode which will ask user to choose the best result
- Changed release date patterns to regexps for cleaner code
- Fixed bug which caused <video> tags to disappear when skipping entries with emulationstation (thank you to ozy24 for reporting this)
- Reached commit number 1000!!! I'm gonna celebrate by coding some more
- Conformed [homedir] to be '~' everywhere
- Added several aliases to platforms to better support all scraping modules
- Added option for custom header when initiating network requests
- When generating search names, all instances of ' v.X' and ' rev.X' are now removed prior to search
- Added '--unattendskip' option to allow for always skipping existing entries without user input (thank you to ozy24 for suggesting this)
You can now set the '--interactive' flag which will cause Skyscraper to prompt for the correct entry whenever a search based scraping module is used (mobygames, thegamesdb, igdb, openretro). For 'screenscraper' and 'arcadedb' this is not enabled as they are checksum and id based, which means they are always correct, so no user input needed.
I've also added the 'igdb' module which needs a key. Request one at their site if you wish to use it. The module/api has some caveats meaning that it won't scrape any media at all as I can't guarantee that the returned media is for the requested platform. I've managed to sortof get around another caveat which means that publisher and developer is not connected to a platform either, so I only use those if only one is returned. But hey, it's another tool in the toolbox. :)
Please try this new release out and let me know if you encounter any errors or inconsistencies. I've done a fair bit of testing myself, but it's software so, you know, bugs/glitches! :)
-
@muldjord Awesome. Need to try the interactive mode asap.
-
@muldjord Tried it. The interactive mode is awesome. I wish this would somehow work for screenscraper. But i understand its crc based...
All i can say is that this scraper has to be included with retropie.
-
@analoghero Awesome! Well, I can enable it with screenscraper aswell, but I don't really see how it would be useful as there will always only be one entry to choose from since its checksum based. If you have any thoughts on this, I'd be happy to hear them.
-
Wow, I just went to check out EmuMovies. The community on there is doing some pretty amazing work!
I just downloaded the wheels pack for the Sega Master System, plopped them onto my Pi in the '~/.skyscraper/import/wheels' folder and ran "Skyscraper -p mastersystem -s import" and then "Skyscraper -p mastersystem -s localdb" (-s localdb not really necessary as it is default if it's left out) to make use of the imported data and BAM! I now have wheels for about 70% or my Mastersystem roms. That is pretty amazing. All I had to do was to make sure that my rom naming conventions match that of the pack, which is often Hyperspin.
You guys should seriously check that out.
-
@muldjord OK, here are some images that better explain my intentions and my problem.
As you can see by the Atari 2600 Adventure "gamebox" with its spine created from a "spine image" resource, the sizing is off. (I used the wrong spine color intentionally to demonstrate the slight gap in the top and bottom, which I cannot fix).
For point of reference, I've included an Adventure gamebox that was generated with a wheel resource.
An Atlantis "gamebox" shows the other problem with using a "spine image" as the resource: the side and front colors don't match up.
Finally, I'm including a mixed-art image of what I'm creating for all the systems: screenshot, wheel, gamebox, and marquee (the cartridge).
Again, I love what you've done and cannot tell you enough how much I appreciate your help. Even if you tell me it's impossible, I'll be happy to end this quest for another!
-
@muldjord Maybe there is a way to scrape not matching single roms with the screenscraper module by using a override flag and use a user provided checksum.
For example:
Skyscraper -p megadrive -s screenscraper /home/pi/RetroPie/roms/megadrive/Sonic.zip -crc AC400F6A -
@dorkvader Ah, yes, I see the problem. Let me think about this. I should be able to add some attributes to "gamebox" that will allow this to work. Great idea to use the marquee resource for cartridge btw, I really like that. Awesome!
@AnalogHero Not a bad idea, I'll give this some thought aswell and find a proper way to implement it. I'll probably just call it '--checksum [sum]' as it doesn't really matter if it's the sha1 or the md5 sum people input. Also, I'll disable it if the user tries to scrape more than 1 rom.
-
@muldjord Exactly what i thought. Just for single roms.
Then you can scrape amiga .lhas which arent in their database using a checksum from an .adf entry :)
-
@dorkvader 2.7.1 out now which can do what you are looking for. Just add the 'sidescaling="both"' attribute to the gamebox xml note, and it will autoscale the image to completely fit both width and height. That should do the trick. Please try it out and let me know if it does what you expect.
@AnalogHero Concerning the checksum option, I have decided against it for the simple reason that it is a screenscraper only option, and there's not a lot of payoff with it. For most roms you could just as easily just rename the input file, which would take just as long as getting the sha1's. With that said, for the LHA files that you suggest, I am currently investigating if I can get all of the Retroplay LHA's added to screenscraper. Then we can start linking them to the correct games. :)
-
THANK YOU SOOOOO MUCH, Muldjord! I am excited to try it out and will let you know the results ASAP!
-
@muldjord said in Versatile C++ game scraper: Skyscraper:
sidescaling="both"
Works BRILLIANTLY! Thought I'd really test it with a Vectrex box, and as you can see it worked beautifully, matching the graph lines as well as one can expect.
Thank you for your help and for your generosity in time and work. If you have some kind of "donation jar," I will gladly throw a tip your way.
-
@dorkvader No need man, but I appreciate the sentiment :) Feel free to donate something to the RetroPie project though. Or, if you insist, I have a patreon at https://www.patreon.com/muldjord :)
Looks great btw! I love that idea and I am probably going to be doing the same thing for some of my platforms. Probably just using the marquee (with the spine graphics) for the spine instead, and then using a custom artwork.xml file for those platforms that I have them for. Thank you for the inspiration.
-
@muldjord I see your point. Im also not sure if its worth the effort. But there are situations were renaming isnt a good option. (If you have already savestates or config files for some games).
Also you could not only do a checksum option but a general override option (- override {gameid OR gamename}). So it could be used for more than one scrapingmodule.
-
@analoghero I'll consider it. It's definitely not a bad idea, I just have to make sure it makes sense for Skyscraper as a whole and that the implementation in the code doesn't bloat it. :S
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.