Versatile C++ game scraper: Skyscraper

muldjord

@stoo Skyscraper looks for both .svm and .scummvm files.

It won't get rid of the dummy files, but what I can do is, for each dummy file, look that filename up in the scummvm.ini file and use the name from that file when scraping. Then you could call the dummy files whatever you like, they would be recognized regardless.

EDIT: 'players' have been fixed to always be a 1-digit or in some cases '4+' now. I currently look for the formats "1 Player", "1-3 Players", "1-3" and "1 - 2 (2 simultaneous)" and convert them to "1", "3", "3" and "2". If players exceeds 9 it handles that aswell.
If you happen upon more formats Skyscraper needs to understand, let me know in a comment. :)

hi_im_daft

@muldjord - I have seen a few 1 - 4 as well, might as well add those to cover bases, and since that's the maximum number of players (traditionally) supported by consoles.

muldjord

@hi_im_daft Yes, "1 - 4" is already covered by my current regexp. It can even do "1 - 10" without problems. I think that's it then. I will move on to improving the game result search match function in the coming week. I have plenty ideas on how to get better results from the sources that don't use checksumming. It's ok'ish right now, but I can imrove it at lot.

AnalogHero

@muldjord I went ahead and compiled 2.3.1 from master and can say it works as it should:)

muldjord

@analoghero Cool :) Glad it works. Thank you for testing it.

AnalogHero

@muldjord Strange thing is that the .hack-link something is gone now. I know its filtered now, but i rescraped my megadrive collection with exact the same command as yesterday and it worked flawless. Exeption seems amiga. When i scrape with screenscraper it guits :) after 30 fails.

muldjord

@analoghero They probably fixed their servers then. What files do you use for Amiga? If you use .uae files it won't work with screenscraper at all. If you have ADF's it might work for some of them. But I strongly recommend using openretro instead.

Skyscraper is set to quit if the first 30 consecutive entries don't return anything. It makes no sense to continue since something is obviously wrong at that point (unless you have some really rare roms as the first 30 games, which would be extremely rare).

AnalogHero

@muldjord Then its fine. Im also using .uae files with HoraceandSpiders whdload autobooter.

SammyBoy

This post is deleted!

dudleydes

@muldjord I am getting a build error when trying to compile on Linux Mint 17 Qiana Cinnamon 64-bit. Running dpkg-query -l qt5* confirms that I have qt5-default installed (version 5.2.1+dfsg-1). The error occurs when compiling src/fxshadow.cpp. Full output when running make with error message can be found at https://pastebin.com/b8805LVr.

mitu

@dudleydes Seems that the error comes from a function introduced in a newer version of Qt5 (5.3) - http://doc.qt.io/qt-5/qcolor.html#qPremultiply.

dudleydes

@mitu Thanks for the reply. Mint 17 appears to only have version 5.2.1 available in the Software Manager so I'll look to upgrade to Mint 18.

SammyBoy

@muldjord you should consider asking @herb_fargus to allow Skyscraper to be installed from inside the RetroPie menu

muldjord

@dudleydes Yes, Skyscraper requires Qt5.3. I'll add a note about that in the instructions.

@SammyBoy I would love to have Skyscraper in the setup script. I think with the new "simple mode" it would make sense. So it should just compile from source and run it in simple mode. I will look into it at some point.

SammyBoy

@muldjord Did you read my message on Reddit?

bleuge

Is there any cause why i am getting some dupes?
For example, \fba\galaga.zip, i got two exact entries in the gamelist.xml file for fba.

Also, some downloaded pngs are not valid (for example 720.png for arcade set). This could be that the downloaded png is bad (maybe skyscraper could check this), or that skyscraper saved it wrong?

Thanks everyone!

muldjord

@bleuge Concerning doubles, please create pastebin's (https://pastebin.com/) for the following and give me the links:

Paste the output from "ls -l /home/pi/RetroPie/roms/fba"
Paste the output from "cat /home/pi/RetroPie/roms/fba/gamelist.xml" (if that's where you have it)

Which version of Skyscraper are you running? I actually do check the images which is why I find it odd that you get the png errors. Is it only for 'screenscraper' you get those png errors?

bleuge

@muldjord

i connected to windows host, these are the pastebins

https://pastebin.com/fDr4CkXM

https://pastebin.com/rWXYWWnp

I use last skyscraper cloned from repo (git clone ...)

I've made a non-exhaustive check of png images, it seems only bad images in arcade folder.

Also, regarding the hack-link bug. When you fixed it, i re-scrapped everything with --updatedb option, to (as far i understood) update everything. But i'm finding still many wrong images of hack-link associated to games.
I guess this could not be fixed as they are in my localdb, and filename is not related to hack-link so skyscraper is getting these files again.

So, the solution is clear (delete) all my localdb folders and rescrape everything...?

Thanks muldjord!

muldjord

@bleuge Well, the first problem with the doubles is pretty obvious. You have the galaga.zip file in both the fba folder and the fba/samples folder. Skyscraper also scrapes subdirs. If you don't want this, please us the '--nosubdirs' option.

If you run all scraping modules with the '--updatedb' option, the hack-link entries in your localdb's should go away without the need for deleting the entire dbs/platform folder. If it doesn't maybe it's because you're skipping existing entries?

I'll check the arcadedb module to see if anything is wrong with the image check.

EDIT: Can you provide a filename for a game that returns a faulty png?

bleuge

@muldjord I've seen faulty pngs in 720 (arcade) and alien3 for example. This is not something important i think, i could update this files by hand or whatever.
The files don't start with the typical PNG header (PNG magic in the first bytes), but they end with a IEND tag (marking end of file in a png).
As i said, it's not important. But maybe if it's easy and don't slow skyscraper, checking that is a png file prior to saving it could be nice.

This is part of the script i used to rescrape the sets:

Skyscraper --updatedb -t 8 -p coleco --videos --unattend --skipped -s openretro --pretend
Skyscraper --updatedb -t 8 -p coleco --videos --unattend --skipped -s thegamesdb --pretend
Skyscraper --updatedb -t 8 -p coleco --videos --unattend --skipped -s screenscraper --pretend
Skyscraper --updatedb -t 8 -p coleco --videos --unattend --skipped -s localdb

I used --skiped because i wanted that even don't recognized files will be included in the gamelist (so i could check visually for not scraped files)