Versatile C++ game scraper: Skyscraper

dorkvader

@muldjord said in Versatile C++ game scraper: Skyscraper:

sidescaling="both"

Works BRILLIANTLY! Thought I'd really test it with a Vectrex box, and as you can see it worked beautifully, matching the graph lines as well as one can expect. 0_1535491463132_Scramble Vectrex box.png

Thank you for your help and for your generosity in time and work. If you have some kind of "donation jar," I will gladly throw a tip your way.

muldjord

@dorkvader No need man, but I appreciate the sentiment :) Feel free to donate something to the RetroPie project though. Or, if you insist, I have a patreon at https://www.patreon.com/muldjord :)

Looks great btw! I love that idea and I am probably going to be doing the same thing for some of my platforms. Probably just using the marquee (with the spine graphics) for the spine instead, and then using a custom artwork.xml file for those platforms that I have them for. Thank you for the inspiration.

AnalogHero

@muldjord I see your point. Im also not sure if its worth the effort. But there are situations were renaming isnt a good option. (If you have already savestates or config files for some games).

Also you could not only do a checksum option but a general override option (- override {gameid OR gamename}). So it could be used for more than one scrapingmodule.

muldjord

@analoghero I'll consider it. It's definitely not a bad idea, I just have to make sure it makes sense for Skyscraper as a whole and that the implementation in the code doesn't bloat it. :S

Used2BeRX

@dorkvader Those are awesome man.

Any chance you could hook me up with your templates for those? I'd love to make 3D boxart for all of the Xtras I've been making. The only template I have though makes every game look like a generic DVD case. It still looks good, but it's miles away from what you're showing here.

dorkvader

@used2berx Sure thing. I'm finishing the Vectrex spines tonight and will be happy to zip my template and or/completed images (spine and boxes) or both. The system has about 25 officially released titles and I've more than tripled that with public domain homebrews. I'm also doing that with all the home consoles from the Golden Era o' Video Games. Just email me.

Used2BeRX

@dorkvader Thanks man. That would be great. :)

Not sure where to find your email though?

easye9inches

Can someone help me in installing Skyscraper to RetroPie on Ubuntu? muldjord mention that install is exactly the same as is on the PI. I have done the steps listed in the GitHub and I am getting this error:

$ wget -q -O - https://raw.githubusercontent.com/muldjord/skyscraper/master/update_skyscraper.sh | bash
bash: line 3: curl: command not found
--- Fetching Skyscraper v. ---
--2018-09-03 22:34:48-- https://github.com/muldjord/skyscraper/archive/.tar.gz
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-09-03 22:34:48 ERROR 404: Not Found.

I am assuming that I should also install this in the RetroPie folder?

mitu

@easye9inches said in Versatile C++ game scraper: Skyscraper:

I am assuming that I should also install this in the RetroPie folder?

No, you can run it from anywhere you have write access. You need to install curl and then re-run the installation

sudo apt-get -y install curl

easye9inches

@mitu said in Versatile C++ game scraper: Skyscraper:

@easye9inches said in Versatile C++ game scraper: Skyscraper:

I am assuming that I should also install this in the RetroPie folder?

No, you can run it from anywhere you have write access. You need to install curl and then re-run the installation
sudo apt-get -y install curl

Thanks! got it installed

easye9inches

Do I have to edit the config file if all my roms are on a external HDD? Because nothing is scraping because it is the default home/pi/roms setting. Id rather scrape my games on the external for backup purpose, and than transfer them over to the pi.

muldjord

Skyscraper version 2.7.3 released: https://github.com/muldjord/skyscraper

Improved image cropping to now also crop black borders, but only for screenshots (Thank you to 'chipsnblip' for suggesting this)
Made 'import' base folder configurable in config.ini
Fixed bug in 'import' scraping module that caused dummy titles to be saved to localdb when scraping media resources
Changed 'curl' to 'wget' in update_skyscraper.sh script to avoid curl requirement

muldjord

@easye9inches Please read the output of '--help' and also check out the options in '~/.skyscraper/config.ini.example'. What you want to do is easy to setup.

dorkvader

@used2berx Sorry, I thought my email was in my profile. Does it not appear?

Used2BeRX

@dorkvader I didn't see it anywhere. Just see the @ symbol and your name. If you actually see mine, you can just send me an email though. I'm not sure what my profile looks like to other people either. :)

timb

@muldjord Thanks for the awesome tool!

So, I’ve run into an issue scraping my collection with screenscraper.fr as the source. My ROMs are all compressed as individual 7z files (to save space), however Skyscraper is trying to match the SHA1/MD5s of the 7z files themselves, instead of the ROM they contain. Obviously this won’t wok, so it’s failing to match anything.

Most other tools will actually uncompress the file and get the hash of the resulting file(s), or even pull the hash out of the header for supported formats (ZIP files have CRC hashes stored in the header for each file in the archive; 7z stores MD5 hashes in the header).

Can this be implemented in Skyscraper? I belive the libarchive library has support for 7z, so it shouldn’t be hard to implement. Alternatively, Skyscraper could call the external 7z binary with the “7z h” option and read the hashes from stdin (this is a very hacky way to do it, but would be easy to implement).

Thanks for all the great work!

Edit: If this is too much work, how about adding a command line option to override the SHA1 and MD5 for individual ROMs? That way I could wrap Skyscraper in a custom Python script that would traverse a directory and for each file decompress it, generate SHA1 and MD5 hashes then call Skyscraper. Here’s some Python like pseudocode to illustrate:

#!/bin/pseudocode
for file in “/path/to/compressed/roms”:
  run(“7z e file --stdout”)
  rom = run.STDIN
  rom_md5 = hash.MD5(rom)
  rom_sha1 = hash.SHA1(rom)
  run(“Skyscraper -s screenscraper -p snes -sha1 rom_sha1 -md5 rom_md5 file”)
done

I could write a working script that does the above in basically no time. The script shouldn’t add more than 1 second per ROM to the scraping times (compared to running Skyscraper by itself on uncompressed files), which I could live with.

So yeah, if adding native compression isn’t a priority, adding hash override options would be an acceptable alternative I could live with. :)

muldjord

EDIT: I just thought about the obvious solution to the below mentioned issues. I can just decompress to ram.

@timb Yes, this has been requested quite a few times actually. If I did this, it would be implemented with an '--unpack' option that would enable the decompression functionality. And it would work on zip and 7z files. The only issue I see is the bashing I would be doing on the SD card. Unpacking each rom to a temporary file and subsequently deleting the rom again, sometimes thousands of times in a row... That's just a very bad idea if you want to keep your SD card alive for more than a few weeks. SD cards don't spread around its writes like an SSD does. So this is a real concern.

With that said, I could add it, and make sure to put up a red warning text when using it. At least people would be aware of the issue and can't blame me when their setup goes up in smoke. ;)

dorkvader

@used2berx OK, my email is now visible in my profile. You just have to enable it to be seen in through profile edit. It's like the snake was at my feet ready to strike, but all be damned if I could see it. I did finish the Vectrex box art, btw.

timb

@muldjord

Yes, decompressing to ram is the way to go. That’s what the pseudocode in my last post essentially did. (It ran 7z with the —stdout option, which of course decompresses to stdout, which is read with a pipe into the pseudocode process as a data object so it never touches the SD card.) I use this method with a Python script I wrote that sorts and verifies ROMs. Obviously you won’t have to worry about piping from an external process, since you can just use libarchive (or another library) to implement the decompression natively, but decompressing into memory is the way to go.

That said, decompressing to a temp file can still work if you have to. Simply do it to /dev/shm, which is on a tmpfs (ramdisk). Or do it to /tmp with a warning that includes instructions on how to enable mounting /tmp as tmpfs (it’s a single systemctl enable command). Obviously both of these options are Linux specific, but you could always implement fall back code for other platforms if you’re worried about that sort of thing.

As a side note, I’m not super concerned about wearing out an SD card. Modern cards have wear leveling algorithms built into their controllers, just like SSDs, so they should be fairly robust. If they do wear out, they’re cheap and plentiful. I do embedded development (hardware and software) for a living and have never seen an SD card die from wear. (That said, for datalogging applications I started using the F2FS filesystem instead of ext4 last year; I’ve been so pleased with it I’ve started using it on most of the embedded Linux systems I work with, including all my personal SBCs. It’s a log-structured file system designed specifically for NAND flash storage helps reduce wear, increase speed and improved reliability, especially when it comes to a lot of small read-modify-write operations.)

muldjord

Thank you for the detailed suggestions. They all make sense to me. I think I'll go with libarchive, it's relatively simple to write a wrapper for it. I'll figure something out. My main concern with using libarchive is that it complicates the installation as it would obviously require libarchive as a dependency. Might not seem like a big deal, but I absolutely despise build systems such as autotools and cmake and try to avoid them at all costs. So I'll try to see if I can make it work in a simple manner with Qt's qmake instead. Might not be as robust, but at least I won't have to spend the next 2 weeks pulling my hair out of my scalp while fighting the unbelievably overcomplicated has-to-do-everything-to-the-point-that-only-absolute-experts-get-it-build systems. Oh yeah, we better check for that fortran compiler, that's important.

EDIT: You know what, I'll look into just using Qt's QProcess and use 7z to stdout which is returned to my QProcess.