Versatile C++ game scraper: Skyscraper

stevas

Yeah... You know what, I think it's my bad here. Now, to be fair, the no intro sets have seemingly arbitrarily decided to change the format of their filenames (Sonic is a particularly good example of how this can be easily missed, as the "The" has become "the" - which is maybe more grammatically correct - but why did we need all the v1.1s to change to Rev A etc?)... and, of course, what's exacerbated my issue here is the fact Unix can have BOTH those Sonic files existing at the same time (which meant a process I had of making sure files matched by just copying them over existing ones on my windows PC was... fundamentally flawed).

Anyway, I've decided to rescrape my megadrive stuff again (sigh), and hopefully it'll all figure itself out.

Many thanks for the swift replies though, and I love the app; I just wish there was an easy way to get the composite screenshot images I like (recalbox mix V2, with the cartridge included) so I didn't have to use Skraper on the PC at all.

muldjord

@stevas said in Versatile C++ game scraper: Skyscraper:

I just wish there was an easy way to get the composite screenshot images I like (recalbox mix V2, with the cartridge included)

You can composite that with Skyscraper quite easily. If you import the cartridges into the marquee resource in Skyscraper and then set up the artwork.xml to composite it as you want it, you can basically make it look anyway you like. The Skyscraper compositor is very powerful and can do a lot of cool things. I assume you have seen the documentation for it, but here's the link just in case: https://github.com/muldjord/skyscraper/blob/master/docs/ARTWORK.md

stevas

Yeah, I tried it... is there a way to get it to grab 3d boxart? I mean with the correct spine, not a wheel instead?

muldjord

@stevas Yeah, but it needs some tinkering. I once did it for mastersystem. I simply imported all of the mastersystem spines into the marquee resource in Skyscraper, and then used the boxart effect with the marquee on the side/spine. Looked really good. But there's no automatic way of doing it. So it requires a bit of work to import all of those.

stevas

Actually, I just realised I can use the 3d box Skraper can get at... But now I can't scrape even just screenshots from screenscraper.fr through skyscraper (FFS? FFS).

o0alucard0o

I scraped NES roms for Attract-Mode with the latest Skyscraper version.
The process correctly fills standard cache folders (covers, marquees, screenshots, wheels) but exports only snaps to Attract-Mode.

NES.cfg

romext               .zip
system               Nintendo Entertainment System
artwork    flyer           /mnt/c/Games/Attract-Mode/scraper/NES/flyer
artwork    marquee         /mnt/c/Games/Attract-Mode/scraper/NES/marquee
artwork    snap            /mnt/c/Games/Attract-Mode/scraper/NES/snap
artwork    wheel           /mnt/c/Games/Attract-Mode/scraper/NES/wheel
artwork    boxart          /mnt/c/Games/Attract-Mode/scraper/NES/boxart

Skyscraper command(s):

Skyscraper -p nes -i /mnt/c/Games/roms/nes/ -g /mnt/c/Games/Attract-Mode/romlists/ -f attractmode -e /mnt/c/Games/Skyscraper/NES.cfg -s screenscraper -u username:password --flags unattend,skipped
Skyscraper -p nes -i /mnt/c/Games/roms/nes/ -g /mnt/c/Games/Attract-Mode/romlists/ -f attractmode -e /mnt/c/Games/Skyscraper/NES.cfg --flags unattend,skipped,forcefilename

muldjord

@o0alucard0o Yeah, the default Skyscraper /home/pi/.skyscraper/artwork.xml is set up to look good with EmulationStation. For Attract-Mode you can go ahead and adjust it to fit your needs. For instance:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" width="640"/>
  <output type="cover" height="640"/>
  <output type="wheel" width="640"/>
  <output type="marquee" width="640"/>
</artwork>

When you've saved that, just regenerate the game list for Attract-Mode.

This will export all of the resources Skyscraper supports and scale them according to aspect ratio to 640 pixels height or width. This is the most basic example I can give you. But the compositor supports a lot more.

For the full documentation, go here.

o0alucard0o

@muldjord Thanks for the explanation. 👍

muldjord

@o0alucard0o You're welcome, please see my edit. I just added width and height to some of them.

o0alucard0o

...
artwork    snap            /mnt/c/Games/Attract-Mode/scraper/Sega 32X/snap
...

Hi,
are white spaces supported in path extraction from Attract Mode emulators config?
It seems that quotes, double quotes and backslash doesn't work.

muldjord

@o0alucard0o I just fixed this. Please update to 3.5.8. You can now use spaces in paths. Do not use quotes, it's not necessary and will actually break it. So just enter it like this:

artwork    snap            /mnt/c/Games/Attract-Mode/scraper/Sega 32X/snap

o0alucard0o

@muldjord Thank you

o0alucard0o

Hi,
I recently scraped PC platform with about thirty titles.
Unfortunately, the generation step assigns wrong resources to games.
Ten games with same snap and/or description.

My setup for Windows PC platform, consists of Autoit (.au3) files. This files differ only in name, but the content is the same. Could it depend on this?

For example

#2/30 (T4) ---- Game 'BlazeRush' found! :) ---- 
Scraper:        cache 
Title:          'Yoku's Island Express' (thegamesdb) 
Platform:       'PC' (thegamesdb)

#3/30 (T2) ---- Game 'Cuphead' found! :) ----
Scraper:        cache
Title:          'Yoku's Island Express' (thegamesdb)
Platform:       'PC' (thegamesdb)

muldjord

@o0alucard0o Yes, that is exactly the reason. For most files Skyscraper does a sha1 checksum to id them. But for some script files I do the sha1 checksum on the filename instead, since they either often change, of have the same contents. .au3 files are not currently on this list, so if your files have the same contents, it will have the same id in Skyscrapers cache.

I will add .au3 files to this list, so in future releases this won't be a problem. Thanks for reporting it.

AlCzervik

@muldjord Any reason when I rescraped my NES many of them now are showing "load screens" (e.g., Select 1 or 2 player screen) vs. gameplay screens?

muldjord

@AlCzervik What source is it using for the screenshots? (you can check it with Skyscraper -p nes --cache edit FILENAME. The reason is that sometimes the source has the title screens in their databases as screenshots. It's quite normal, but is a bit of a challenge when scraping from them, since I don't know one screenshot from another.

AlCzervik

@muldjord I'm using screenscraper. Is it random? As in, if I scrape again would it pull different images they've stored? Some examples I know changed were Super Mario 3, Metroid, Zelda 2.

muldjord

@AlCzervik When scraping with -s screenscraper it will first look for a screenshot of the type ss which is usually a normal screenshot. If it doesn't find that, it will look for one called sstitlewhich, obviously, is the game title screen screenshot. So in this case, my guess is that it doesn't find any of type ss and therefore uses the sstitle screenshots.

AlCzervik

@muldjord That must be it for those titles but it is strange that they changed recently. These are older NES titles so perhaps screenscraper changed for some of those titles. Not a big deal, just wanted to point it out. Thanks!

theshadowzero

@muldjord i've got a quick question about skyscraper, i've tried messing with the --cache settings but haven't been able to quite hit my use case. i scraped all my roms over a couple days with skyscraper from screenscraper, but i had videos disabled. now, i'd like to add videos, but i haven't been able to figure out how to get skyscraper to just scrape videos. because the roms already have metadata in the cache, skyscraper seems to want to either skip them entirely, or with a --cache refresh completely scrape them fresh. is there a way to tell skyscraper "the metadata is fine, i just want you to scrape the videos that are missing and not rescrape all the other fields?"

ive tried running it with just --flags videos and it asks me if i want to skip already existing game list entries. saying yes obviously skips all the roms, and saying no will show that it finds the rom, show all the metadata and media i have (with a big red NO() for video), and then do nothing and proceed to the next rom. if i scrape with --cache refresh it finds the rom and the video and will download and include it. i was under the impression skyscraper would automatically add media that is missing but that doesn't seem to be the case, or more likely i am doing something wrong. guidance would be appreciated! i don't want to spend another 2-3 days hammering screenscraper.fr to get videos when i already have 90% of the scrapes i need

edit: i also tried with --cache edit:new=videos but it errors saying it only supports certain resources. not sure that would have solved it for me anyway since i fear it would have expected manual input rather than scraping but i tried it regardless