Versatile C++ game scraper: Skyscraper

AnalogHero

@muldjord Exactly what i thought. Just for single roms.

Then you can scrape amiga .lhas which arent in their database using a checksum from an .adf entry :)

muldjord

@dorkvader 2.7.1 out now which can do what you are looking for. Just add the 'sidescaling="both"' attribute to the gamebox xml note, and it will autoscale the image to completely fit both width and height. That should do the trick. Please try it out and let me know if it does what you expect.

@AnalogHero Concerning the checksum option, I have decided against it for the simple reason that it is a screenscraper only option, and there's not a lot of payoff with it. For most roms you could just as easily just rename the input file, which would take just as long as getting the sha1's. With that said, for the LHA files that you suggest, I am currently investigating if I can get all of the Retroplay LHA's added to screenscraper. Then we can start linking them to the correct games. :)

dorkvader

THANK YOU SOOOOO MUCH, Muldjord! I am excited to try it out and will let you know the results ASAP!

dorkvader

@muldjord said in Versatile C++ game scraper: Skyscraper:

sidescaling="both"

Works BRILLIANTLY! Thought I'd really test it with a Vectrex box, and as you can see it worked beautifully, matching the graph lines as well as one can expect. 0_1535491463132_Scramble Vectrex box.png

Thank you for your help and for your generosity in time and work. If you have some kind of "donation jar," I will gladly throw a tip your way.

muldjord

@dorkvader No need man, but I appreciate the sentiment :) Feel free to donate something to the RetroPie project though. Or, if you insist, I have a patreon at https://www.patreon.com/muldjord :)

Looks great btw! I love that idea and I am probably going to be doing the same thing for some of my platforms. Probably just using the marquee (with the spine graphics) for the spine instead, and then using a custom artwork.xml file for those platforms that I have them for. Thank you for the inspiration.

AnalogHero

@muldjord I see your point. Im also not sure if its worth the effort. But there are situations were renaming isnt a good option. (If you have already savestates or config files for some games).

Also you could not only do a checksum option but a general override option (- override {gameid OR gamename}). So it could be used for more than one scrapingmodule.

muldjord

@analoghero I'll consider it. It's definitely not a bad idea, I just have to make sure it makes sense for Skyscraper as a whole and that the implementation in the code doesn't bloat it. :S

Used2BeRX

@dorkvader Those are awesome man.

Any chance you could hook me up with your templates for those? I'd love to make 3D boxart for all of the Xtras I've been making. The only template I have though makes every game look like a generic DVD case. It still looks good, but it's miles away from what you're showing here.

dorkvader

@used2berx Sure thing. I'm finishing the Vectrex spines tonight and will be happy to zip my template and or/completed images (spine and boxes) or both. The system has about 25 officially released titles and I've more than tripled that with public domain homebrews. I'm also doing that with all the home consoles from the Golden Era o' Video Games. Just email me.

Used2BeRX

@dorkvader Thanks man. That would be great. :)

Not sure where to find your email though?

easye9inches

Can someone help me in installing Skyscraper to RetroPie on Ubuntu? muldjord mention that install is exactly the same as is on the PI. I have done the steps listed in the GitHub and I am getting this error:

$ wget -q -O - https://raw.githubusercontent.com/muldjord/skyscraper/master/update_skyscraper.sh | bash
bash: line 3: curl: command not found
--- Fetching Skyscraper v. ---
--2018-09-03 22:34:48-- https://github.com/muldjord/skyscraper/archive/.tar.gz
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-09-03 22:34:48 ERROR 404: Not Found.

I am assuming that I should also install this in the RetroPie folder?

mitu

@easye9inches said in Versatile C++ game scraper: Skyscraper:

I am assuming that I should also install this in the RetroPie folder?

No, you can run it from anywhere you have write access. You need to install curl and then re-run the installation

sudo apt-get -y install curl

easye9inches

@mitu said in Versatile C++ game scraper: Skyscraper:

@easye9inches said in Versatile C++ game scraper: Skyscraper:

I am assuming that I should also install this in the RetroPie folder?

No, you can run it from anywhere you have write access. You need to install curl and then re-run the installation
sudo apt-get -y install curl

Thanks! got it installed

easye9inches

Do I have to edit the config file if all my roms are on a external HDD? Because nothing is scraping because it is the default home/pi/roms setting. Id rather scrape my games on the external for backup purpose, and than transfer them over to the pi.

muldjord

Skyscraper version 2.7.3 released: https://github.com/muldjord/skyscraper

Improved image cropping to now also crop black borders, but only for screenshots (Thank you to 'chipsnblip' for suggesting this)
Made 'import' base folder configurable in config.ini
Fixed bug in 'import' scraping module that caused dummy titles to be saved to localdb when scraping media resources
Changed 'curl' to 'wget' in update_skyscraper.sh script to avoid curl requirement

muldjord

@easye9inches Please read the output of '--help' and also check out the options in '~/.skyscraper/config.ini.example'. What you want to do is easy to setup.

dorkvader

@used2berx Sorry, I thought my email was in my profile. Does it not appear?

Used2BeRX

@dorkvader I didn't see it anywhere. Just see the @ symbol and your name. If you actually see mine, you can just send me an email though. I'm not sure what my profile looks like to other people either. :)

timb

@muldjord Thanks for the awesome tool!

So, I’ve run into an issue scraping my collection with screenscraper.fr as the source. My ROMs are all compressed as individual 7z files (to save space), however Skyscraper is trying to match the SHA1/MD5s of the 7z files themselves, instead of the ROM they contain. Obviously this won’t wok, so it’s failing to match anything.

Most other tools will actually uncompress the file and get the hash of the resulting file(s), or even pull the hash out of the header for supported formats (ZIP files have CRC hashes stored in the header for each file in the archive; 7z stores MD5 hashes in the header).

Can this be implemented in Skyscraper? I belive the libarchive library has support for 7z, so it shouldn’t be hard to implement. Alternatively, Skyscraper could call the external 7z binary with the “7z h” option and read the hashes from stdin (this is a very hacky way to do it, but would be easy to implement).

Thanks for all the great work!

Edit: If this is too much work, how about adding a command line option to override the SHA1 and MD5 for individual ROMs? That way I could wrap Skyscraper in a custom Python script that would traverse a directory and for each file decompress it, generate SHA1 and MD5 hashes then call Skyscraper. Here’s some Python like pseudocode to illustrate:

#!/bin/pseudocode
for file in “/path/to/compressed/roms”:
  run(“7z e file --stdout”)
  rom = run.STDIN
  rom_md5 = hash.MD5(rom)
  rom_sha1 = hash.SHA1(rom)
  run(“Skyscraper -s screenscraper -p snes -sha1 rom_sha1 -md5 rom_md5 file”)
done

I could write a working script that does the above in basically no time. The script shouldn’t add more than 1 second per ROM to the scraping times (compared to running Skyscraper by itself on uncompressed files), which I could live with.

So yeah, if adding native compression isn’t a priority, adding hash override options would be an acceptable alternative I could live with. :)

muldjord

EDIT: I just thought about the obvious solution to the below mentioned issues. I can just decompress to ram.

@timb Yes, this has been requested quite a few times actually. If I did this, it would be implemented with an '--unpack' option that would enable the decompression functionality. And it would work on zip and 7z files. The only issue I see is the bashing I would be doing on the SD card. Unpacking each rom to a temporary file and subsequently deleting the rom again, sometimes thousands of times in a row... That's just a very bad idea if you want to keep your SD card alive for more than a few weeks. SD cards don't spread around its writes like an SSD does. So this is a real concern.

With that said, I could add it, and make sure to put up a red warning text when using it. At least people would be aware of the issue and can't blame me when their setup goes up in smoke. ;)