Versatile C++ game scraper: Skyscraper

movisman

Cool! I will give it a try and see how it works. Sorry I didn't see your edit above already explaining that.

Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.

The reason I was thinking that, an example:
GameX (Europe).zip

Wants European boxart and logo/game title, but it isn't available on screenscraper.fr. However World is there, and US is there. World seems to me, to be more likely to have the same boxart and/or title as the European release. However on default priority, if EU cannot be found, it will go for the US release, which is more likely to have a different name, and definitely a different box.

The same goes if you have a (USA) release, if this isn't available but World and Europe are, it could go for World, and then Europe after that.

Does that make sense? I might be missing something to suggest this isn't a good idea, but I think that would possibly work better?

movisman

@muldjord

Separate question, can I scrape a single rom when using the --localdb switch? For some reason, that doesn't work for me. Running a command on a single rom, for example:

/opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/megadrive -o /home/pi/RetroPie/roms/megadrive/media --relative -p megadrive -s screenscraper -u username:password --videos "insertromnameandpathhere.zip"

Will successfully scrape a single rom to the localdb.

However, when I then run the same command as above but with the localdb as source to pull the rom into the gamelist, it doesn't do it. I believe the same happens when using the query function too.

If I then remove the single rom+quotes at the end of the above command to pull the entire localdb for that system into the gamelist.xml, it will bring the rom in successfully, but unfortunately at the expense of any manual edits I made to the other games (it will overwrite any minor changes you made).

Any idea?

Thanks

muldjord

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord
Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.
I did this originally, but it turned out many world entries have the japanese versions, so I changed it.

muldjord

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord

Separate question, can I scrape a single rom when using the --localdb switch? For some reason, that doesn't work for me. Running a command on a single rom, for example:

That is not possible and won't be possible. It would require me to inject only selected entries into the gamelist which is something I don't want to fiddle with.

movisman

@muldjord said in Versatile C++ game scraper: Skyscraper:

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord
Just thinking out loud, if you are now attempting to auto detect region, and then hit the priorities list after that, would it be sensible to have 'world' first in that list? Then eu, us, jp.
I did this originally, but it turned out many world entries have the japanese versions, so I changed it.

Ah ok, I see. That's fair enough.

muldjord

@movisman Let me elaborate a bit. I'm not saying it won't ever be able to do that (actually that is what I wrote but whatever), but it's a rather big code rewrite I would have to do for that functionality. And personally I wouldn't ever use it. I change the data in my localdb instead and rescrape. Much better imo.

movisman

@muldjord said in Versatile C++ game scraper: Skyscraper:

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord

Separate question, can I scrape a single rom when using the --localdb switch? For some reason, that doesn't work for me. Running a command on a single rom, for example:

That is not possible and won't be possible. It would require me to inject only selected entries into the gamelist which is something I don't want to fiddle with.

Cool, that's fine, I just wanted to check if it was possible or if I was doing something wrong. If I want to scrape a single game but not have it fiddle with the existing gamelist entries, is my only option at the moment either to edit the db.xml file for that system, or scrape the file to a fresh gamelist.xml and merge it into my existing one?

Thanks!

muldjord

@movisman Second option is your best option as you've already edited the gamelist manually. The first one won't work. If you've already manually edited your gamelist, it won't help to change the data in your localdb files. It will still overwrite the things you've changed upon a rescrape.

movisman

@muldjord

Ah no yeah I understood that, I haven't made loads of edits to the point where I couldn't rescrape. To be honest since you've improved the region stuff it's going to be quicker for me to rescrape anyway to pick up the correct artwork, so I could rescrape, edit db.xml after and then scrape from the DB.

movisman

@muldjord

Hiya,

Since I updated this morning, for some reason queries against SH1, CRC, MD5 are not working? It just comes back with 'game xxx not found' on pass 1. Adding the maximum verbosity on the command line shows the correct SHA1 and platform, but it never finds it. I am using literally exactly the same command as I was using previously to obtain a correct scrape for mismatched files. Tried a variety of valid checksums off screenscraper.fr.

Running without query is fine though.

Do you know what might be happening?

Thanks!

movisman

FYI,

An example command used was:
/opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/dreamcast -o /home/pi/RetroPie/roms/dreamcast/media --relative -p dreamcast -s screenscraper -u username:password --videos --nobrackets --query sha1=058b2eda664ee953caeb5ad35efbbdbcfc60d85f "/RetroPie/roms/dreamcast/Crazy Taxi (USA).chd" --verbosity 3

It made a start, then bombs out:

Starting scraping run on 1 files using 3 threads.
Sit back, relax and let me do the work! :)

#1/1 (T1) Pass 1 ---- Game 'Crazy Taxi (USA)' not found :( ----

Debug output:
Tried with: 'sha1=058b2eda664ee953caeb5ad35efbbdbcfc60d85f'
Platform: dreamcast

0/1/1

This wasn't happening before, but thought i'd share the command I used. I tried a few others, changed my file extensions around etc, but the same result.

Thanks

jakejm79

Where have my scan lines gone?
So I had hoped to take a bunch of screenshots myself that included the scanline shader.
The screenshot.png has the scanlines (I've even resized the image in GIMP to the same specs as my artwork.xml and they are still there) but after skyscraper processes the image to create my final png the scanlines have magically disappeared. Is there anything I can do to make sure that the image remains similar to the original or is it a limitation of the artwork manipulation scripts.

Also I pick up some noticeable banding (especially noticeable in the gray gradient box), is there anything I can do to help combat that?

Here are a couple of screenshots, the GIMP resized and cropped png and the artwork.xml png, you can tell the missing scanlines. And the associated artwork.ini

alt text

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" width="640" height="480">
    <layer resource="screenshot" x="20" width="520" height="390" align="center" valign="middle">
      <rounded radius="10"/>
      <stroke width="1" red="0" green="0" blue="0"/>
      <stroke width="1" red="156" green="156" blue="156"/>
      <stroke width="2" red="35" green="35" blue="35"/>
    </layer>
    <layer resource="blank.png" align="right"  valign="top">
      <rounded radius="10"/>
    <layer resource="wheel" width="161"  align="center"  valign="middle">
    </layer>
      <rounded radius="10"/>
      <stroke width="1" red="0" green="0" blue="0"/>
      <stroke width="1" red="156" green="156" blue="156"/>
      <stroke width="2" red="35" green="35" blue="35"/>  
</layer>
  </output>
</artwork>

muldjord

@jakejm79 said in Versatile C++ game scraper: Skyscraper:

Where have my scan lines gone?

If you resize an image with scanlines they will seem to disappear - or at least get smoothed over. That's not a Skyscraper issue but a resize issue in general.

muldjord

@movisman said in Versatile C++ game scraper: Skyscraper:

@muldjord

Hiya,

Since I updated this morning, for some reason queries against SH1, CRC, MD5 are not working? It just comes back with 'game xxx not found' on pass 1. Adding the maximum verbosity on the command line shows the correct SHA1 and platform, but it never finds it. I am using literally exactly the same command as I was using previously to obtain a correct scrape for mismatched files. Tried a variety of valid checksums off screenscraper.fr.

Running without query is fine though.

Do you know what might be happening?

Thanks!

Please update and try again, I fixed some stuff. I need to test the query stuff you mention to see if that has been broken. Gonna look at it sometime soon.

jakejm79

@muldjord

Well I understand what you are saying, but if you look at the example, they have both been resized to the same dimensions from the same original screenshot and the scanlines are still prominent in one.

Upon further inspection it looks like skyscraper reduces the screenshots to a 640x* resolution when importing them into the DB (maybe to save space), it appears that whatever interpolation is used with this scaling that causes the problem. Is there a flag or variable that can be change to decide what is used? or at least allow the user the option artwork to be imported to the DB at original dimensions if disk space is not an issue.

OK so it appears the database import is the issue (or if you will responsible) rather than the artwork.xml. I copied the original screenshot to the database directory (and renamed it). Ran Skyscraper and it generated the games artwork, as you can see in the example below the scanlines are visible, not to mention the overall image quality is better, doesn't really make sense to me to have Skyscraper resize the image twice (on DB import and artwork export) especially at the expense of quality.

alt text

muldjord

@jakejm79 You can use the "--noresize" option. I will also add this to the config.ini options as "noResize=true" in the "[localdb]" section in the next version (it's not currently possible to set it in config.ini).

movisman

@muldjord

Hi there,

No worries - I updated since your last commit on github though, so not sure updating again would make too much difference? I also removed the app entirely to clean everything up, put it back on and it did the same (this was yesterday late evening).

Thanks!

jakejm79

@muldjord
Thank you that would be great, I tend to launch it from the retropie setup menu, so a config.ini option would be great.
I was thinking a little last night, the best way to get the output I am looking for might be to take the screenshots without scanline and then super impose them over the top when creating the artwork. Maybe such an option could be added, much like adding a drop shadow or a frame. This way the scanlines would be added to the final sized screenshot and wouldn't be subject to resizing.
Thank you for all your hard work.

muldjord

@jakejm79 A "scanline" effect layer is a good idea. I'll add it to the roadmap.

jakejm79

@muldjord
Thank you, that sounds good. In the meantime I have created a overlay the same size that the screenshot gets resized to, that I put in the resources folder and edited my artwork.xml accordingly.
Much better results now. One thing I have noticed is that the black bars don't seem to be cropped out, I thought originally that black bars were cropped out and then the image would be resized to what is specified in the artwork.xml, distorting the aspect ration if need be.
alt text