Versatile C++ game scraper: Skyscraper

jakejm79

@muldjord
Thank you that would be great, I tend to launch it from the retropie setup menu, so a config.ini option would be great.
I was thinking a little last night, the best way to get the output I am looking for might be to take the screenshots without scanline and then super impose them over the top when creating the artwork. Maybe such an option could be added, much like adding a drop shadow or a frame. This way the scanlines would be added to the final sized screenshot and wouldn't be subject to resizing.
Thank you for all your hard work.

muldjord

@jakejm79 A "scanline" effect layer is a good idea. I'll add it to the roadmap.

jakejm79

@muldjord
Thank you, that sounds good. In the meantime I have created a overlay the same size that the screenshot gets resized to, that I put in the resources folder and edited my artwork.xml accordingly.
Much better results now. One thing I have noticed is that the black bars don't seem to be cropped out, I thought originally that black bars were cropped out and then the image would be resized to what is specified in the artwork.xml, distorting the aspect ration if need be.
alt text

muldjord

Skyscraper version 2.8.3 released: https://github.com/muldjord/skyscraper

Added "region" and "lang" options to "[platform]" section of config.ini
Added "noResize" variable to "[localDb]" config.ini section
Fixed bug where "--query" would stop working when region was autodetected

@movisman Please test the query option again, I've tested it myself and it works as expected. But it's always nice to have others test it aswell.

muldjord

@jakejm79 It might be that it isn't completely black, then it won't be cropped. Try opening it in Gimp and check if the black is actually #000000.

jakejm79

@muldjord

Thanks, that was the problem, not sure where it came from but it was a hint of green in it, it didn't used to be there, I retook the screen shot a few times and it was always there. Maybe we could have it where instead of #000000 being ignored that maybe #050505 and below is. It seems to crop out the black fine on my PSX screenshots, so I'm not sure if Picodrive is responsible or not, but I didn't notice this issue before. Anyway I manually edited the screenshot and all is good now.
alt text

While you are fulfilling requests, the last thing I'd love is a maxsize variable for resizing resources, kind of like how the themes handle things, a resource can be resized in both height and width so that they are both under the maxsize but aspect ratio is maintained, currently I can only resize one variable to maintain aspect ratio, with the wheels.png, some can do with resizing vertically others horizontally. I tried the mpixel option but it doesn't really work for extremes, tall and thin vs. short and wide.

Otherwise it now seems perfect.

jakejm79

@muldjord
I do have one other question (I searched the artwork examples, artwork.md, readme.md and wiki and didn't see it addressed). Can I have separate artwork.xmls for different game folders, i.e. can I have one for ../arcade and one for ../arcade/vertical ?

muldjord

@jakejm79 You can have one per platform, but not for subfolders inside platforms. Set them in config.ini under any "[platform]" section you need. For instance:

[amiga]
artworkXml="artwork_amiga.xml"

jakejm79

Thanks, I had seen that. I think what I will do is run skyscraper with just vertical files inside the arcade, then move those out and put the horizontal ones back in after modifying the artwork.xml. I assume skyscraper just appends to the gamelist.xml and wont delete existing entries even if the rom files aren't currently present in the roms folder.

muldjord

@jakejm79 I can't remember, you'd need to just try it. On another note, I have just finished the <scanlines> effect. It will be in 2.9.0 to be released soon. By default it will use a scanline overlay I made myself. But you can also create your own and set it with 'resource="filename.png"'. It's also possible to set the composition method such as overlay, screen, lighten and so on. Just like you can on layers. Lastly, you can set the opacity of the overlay from 0-100 with "opacity=50".

jakejm79

Sounds good I look forward to trying it out. The only issues I see with using an overlay:

If the over lay isn't black/transparent or a mix, then you get scanlines on black portions (I originally used a overlay that had a slight hint of purple/green in it vertically to try and help recreate that full crt effect, but it messed with the black areas).
Does the overlay get cropped or resized to match the screenshot size, if it gets resized then you will run into the problems with them getting distorted. Ideally you'd want to maybe start with a 1920x1080 scanline overlay and just crop a section of it to match the size of the screenshot.

muldjord

@jakejm79 said in Versatile C++ game scraper: Skyscraper:

If the over lay isn't black/transparent or a mix, then you get scanlines on black portions (I originally used a overlay that had a slight hint of purple/green in it vertically to try and help recreate that full crt effect, but it messed with the black areas).

You can use one of several different composition modes that will give you pretty good control over this. I am creating the scanline myself. I will provide 2-3 by default, and you can use your own aswell if you wish.

Does the overlay get cropped or resized to match the screenshot size, if it gets resized then you will run into the problems with them getting distorted. Ideally you'd want to maybe start with a 1920x1080 scanline overlay and just crop a section of it to match the size of the screenshot.

It is not getting resized. But I am considering adding a "scale" attribute to it, so the user can scale it up or down using nearest neighbor scaling. That should avoid the issue with the scanlines disappearing. But it will look best unscaled of course.

Basically the effect gives you almost complete control.

jakejm79

OK sounds like you have it covered. I look forward to trying it out.

movisman

@muldjord

Hey there,

Sorry for the late reply, the last couple of days have been quite busy and have not had much time in front of the PC to continue my build!

I am sure it works fine now, but I will give it a test as soon as I can (should be able to do it later) and clarify it for you.

Cheers!

muldjord

Skyscraper 2.9.0 released: https://github.com/muldjord/skyscraper

Update either from the RetroPie-Setup script or using the update procedure from the Github readme, depending on what version you are using.

Release notes:

MAJOR: Now looks up Amiga lha files in the Amiberry "whdload_db.xml" and retrieves data from "openretro.org" based on the uuid from the xml
Added search based fallback pass for Amiga when game isn't found via uuid
Added "<scanlines>" effect to compositor. Check artwork documentation for more info (Thank you to "jakejm79" for suggesting this)
"mobygames" module now uses https
Fixed bug in "openretro" module where "developer" would potentially scrape wrong under certain circumstances
Improved "description" scraping for "openretro" module
Improved bracket tag handling for Amiga lha files A LOT
Fixed minor pass 2 bug when using search based sources
Added a pass for integer to roman conversion for search based sources (eg. "4" converts to "IV")
"the" matching now uses regular expression for better precision
Added "mame-*" platforms to "mameMap" name load list

This is a big update for the Amiga whdload crowd. The best amiga database around is surely the OpenRetro database. Unfortunately the lha filenames used with whdload gave us a bit of a hard time. This is all a thing of the past now. With the help of HoraceAndTheSpider and Olly, we now have a database which maps pretty much all of the lha files directly to their data inside OpenRetro. And it all works seemlessly. Just pop those lha's in your roms/amiga folder and run "Skyscraper -p amiga -s openretro --refresh" and watch that glorious data stream in.

Lots of other stuff in this release aswell. Some rather big improvements to the filename search based scraping. If a game is called "Game Name 6" and no results are returned, an extra pass will now be tried with "Game Name VI" automatically.

Lastly, a new <scanlines> effect has been added to the compositor. It comes with two default overlays you can use. But it's also very customizable, so feel free to use your own overlay and fiddle with the compositing method to get the desired result. It's all documented here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md#scanlines-effect-node-from-v290-optional

Happy scraping!

Silent

Regarding your last changes to Roman numerals handling and other permutations of title:

Do you think it makes sense to remove ™, ® and © (also maybe others) from titles prior to scraping? "Hitman™" currently fails to scrape miserably even though it has entries in databases:
https://thegamesdb.net/game.php?id=44303

However, that's also something which will be solved by "title hints" feature you mentioned is on the roadmap at some point.

EDIT:
FYI numerals handling change fixed the way R.C. Pro-Am II is scraped (previously it'd scrape as R.C. Pro-Am, so first and second game showed the same), so good job :D

muldjord

Title hints? I don't remember what that is. :S

Silent

@muldjord said in Versatile C++ game scraper: Skyscraper:

Do you have ™, ® and © in your filenames? Or is it in the returned titles?

Filenames - as Hitman presents itself as Hitman™ and so that's how Moonlight saves it. However, I should just rename the file and be done with it - no point bothering you with something this minor.

@muldjord said in Versatile C++ game scraper: Skyscraper:

Title hints? I don't remember what that is. :S

https://github.com/muldjord/skyscraper/issues/77#issuecomment-446739139

This and your reply. At least the way I understood it is that you may consider defining title aliases for those games which are poorly matched (another use case I have for this: nothing in skyscraper detects "DiRT 3 Complete Edition", but "DiRT 3" works fine so I could define it as an alias).

EDIT:
"esgamelist" scraper sounds excellent! Will be super useful for cases where Skyscraper can't scrape the title fine but ES inbuilt scraper does! DiRT 3 issue I just mentioned is one such case, for example.

Thank you and mgerhardy!

muldjord

@Silent I don't mention any custom aliases in that. All I mention is the already implemented automatic extra pass for "Game Name II" to "Game Name 2". But you can use the --query option to create custom search queries for special cases. https://github.com/muldjord/skyscraper/releases/tag/2.7.5

But it's not a bad idea to have some kind of aliases.csv file that you could put in "DiRT 3 Complete Edition";"Dirt 3". Just one per line for instance (it's exactly what I already do for mame games). And it would then check that if you apply a "--aliases" command line option or something like that. I'll make that, should be useful.

Silent

Also call me stupid, but I don't see a way how to scrape a single rom? Just point to a specific file instead of a directory, like

Skyscraper -p nes -s screenscraper /home/pi/RetroPie/roms/nes/MyGreatGame.nes

?