Versatile C++ game scraper: Skyscraper

bleuge

@muldjord I compiled this in Windows using Cygwin some months ago, but got many problems running it. Do you think it is too much effort to make it run ok in Windows, scrapping is lot of faster and easier that connectin to Pi, etc. Thanks!

About commandline, I'm happy with it, but I know many users will be too complicated, an interactive shell as you suggest it will ok, while a graphical gui is thought and implemented. If we could run this in Windows i'll do a gui myself if it's needed. Regards!

muldjord

@bleuge You can easily cross-compile it for Windows. Just download and install Qt 5 from the official site (open source version). When installing, put a checkmark in the latest mingw32 version, I don't support MSVC++ although it might work, not sure. When it's installed you'll find a Qt specific command prompt in your start menu. From there navigate to the Skyscraper source folder and run:
$ qmake
$ mingw32-make release

That will give you a compiled version in a subfolder called "release". Then you need to copy the needed dll's from the qt5 install folder into the release folder and start an ordinary command prompt there. Then you should be able to run it.

I thiiiink it will create the .skyscraper data folder inside "C:\Users[username]... somewhere". Just keep in mind that there are some minor Linux specific stuff in Skyscraper's "make install" command that will copy relevant files to the .skyscraper folder. So you'd be missing those. But I believe I auto-create empty ones...

So yeah, shouldn't be too hard, but it's not really something I am going to look into myself at this time. Happy hacking! :)

bleuge

Thanks for tips, I already compiled ok in Cygwin (after some work with qt5). Yesterday i updated it and recompiled it. Copied the needed DLLs and ...

Running Skyscraper v2.0.5a by Lars Muldjord

Please set a valid platform with '-p [platform]'
Check '--help' for a list of supported platforms, now exiting...

What i meant about windows, was the folder/linux structure, making it portable or something like that. I'll try with some roms to see if i can make it working normally.
Thanks!

muldjord

@bleuge I am not sure what you mean. It IS cross-platform compatible (and clearly runs just fine looking at your quoted text). It uses the home folder variable to specify where to save settings, which depends on you OS. So if you are running Windows it will save stuff in the Users folder as expected. Please elaborate.

As I mentioned the only thing that isn't cross-platform is the "make install" part which I've made more convenient and automated on Linux. But it shouldn't really be necessary for it to run properly on Windows.

bleuge

@muldjord thanks for your answers, i'll elaborate.
I have subfolders with roms in c:\roms

Skyscraper.exe -p atari2600 -i c:\roms\2600 -t 8 -d c:\roms\00skdb --nobrackets --verbose

I got

Folder '/home/myuser/.skyscraper/c:\roms\2600/images' doesn't exist, trying to create it... Failed! Please check path and permissions

I've looked for .skyscraper file and found that Skyscraper created folders in

c:\home\myuser.skyscraper\dbs
c:\home\myuser.skyscraper\import...various subs here

I don't have any problem Skyscraper creating files and folders whenever it needs it. But well, i can't make it work under Windows.

Maybe i can create a config.ini file in the folder of the executable?

Thanks for your time, this scraper is very very good! Your idea of the localdb is wonderful.

muldjord

@bleuge I see a potential problem. Please try writing your paths with forward slashes instead of backward slashes.

I do find it odd that it looks for /home/myuser though. I use the builtin user home from Qt, which should adjust for Windows and use the Users folder.

A Former User

This might be a tougher request but is it possible to get .7z support for Nintendo DS? DraStic supports .7z and most rom sets are already in that format for the platform.

Thanks.

bleuge

@muldjord I updated everything and recompiled it.

This time i ran it at my main pc at home. W10 this time.

I have the skyscrapper executable (and required dlls) in d:\ES\ (emulationstation folder)
Roms in c:\roms\

Skyscraper.exe -p atari2600 -i c:/roms/2600 -t 2 -d c:/roms/00skdb --nobrackets --verbose
Skyscraper.exe -p atari2600 -i c:\roms\2600 -t 2 -d c:\roms\00skdb --nobrackets --verbose
(both work) I changed -t 8 to -t 2, it seems it was hanging for no reason, maybe my connection is too slow. With -t 8 it processed some games and stay there, no more output, with -t 2 it keeps going until end.

Running Skyscraper v2.0.5a by Lars Muldjord
Platform : 'atari2600'
Scraper module : 'thegamesdb'
Input folder : 'c:/roms/2600'
Gamelist folder : 'c:/roms/2600'
Images folder : 'c:/roms/2600/images'
Local db folder : 'c:/roms/00skdb'

Folder '/home/bleuge/.skyscraper/c:/roms/2600/images' doesn't exist, trying to create it... Success!

I've used backslashes or forward, the result is the same. And... where are the files created? In d:\ (where skyscraper exe is and is run) Look the folder structure created by skyscraper:

D:\home\bleuge
D:\home\bleuge.skyscraper
D:\home\bleuge.skyscraper\c?
D:\home\bleuge.skyscraper\dbs
D:\home\bleuge.skyscraper\import
D:\home\bleuge.skyscraper\skipped-thegamesdb.txt
D:\home\bleuge.skyscraper\c?\roms
D:\home\bleuge.skyscraper\c?\roms\00skdb
D:\home\bleuge.skyscraper\c?\roms\2600
D:\home\bleuge.skyscraper\c?\roms\00skdb\covers
D:\home\bleuge.skyscraper\c?\roms\00skdb\db.xml
D:\home\bleuge.skyscraper\c?\roms\00skdb\screenshots
D:\home\bleuge.skyscraper\c?\roms\00skdb\videos
D:\home\bleuge.skyscraper\c?\roms\00skdb\covers\thegamesdb
D:\home\bleuge.skyscraper\c?\roms\00skdb\covers\thegamesdb\47775e7b9ff344126bc7f5ffe33e05c63bea5764.png
D:\home\bleuge.skyscraper\c?\roms\00skdb\screenshots\thegamesdb
D:\home\bleuge.skyscraper\c?\roms\00skdb\videos\thegamesdb
D:\home\bleuge.skyscraper\c?\roms\2600\gamelist.xml
D:\home\bleuge.skyscraper\c?\roms\2600\images
D:\home\bleuge.skyscraper\c?\roms\2600\images\Adventure (USA).png
....

I know this is not designed to work in Windows, but something is going very wrong with folders and file paths, etc... I am very bad at C++, i'll try to find something in the source code.

Look that funny c? ... it's exactly 0x3f char. Also i looked the skipped files list (skipped-thegamesdb.txt ):

--- The following is a list of skipped games ---
'Montezuma's Revenge - Featuring Panama Joe (USA)', Closest match was 'Montezuma's Revenge'
'Basketball (USA)', Closest match was 'BMX Air Master'
'Battlezone (USA)', Closest match was 'Basketball'
'Beamrider (USA)', Closest match was 'Battlezone'

This seems also wrong. Maybe something is going very bad in this windows compile?

Thanks for your time !!!

muldjord

@bleuge I've been thinking about this some more, and I think your problem comes from running it through cygwin. Cygwin is a POSIX env for Windows. If you want to run it on Windows, I suggest you use the Qt provided command line - both for compiling and running it.

Basically cygwin messes with A LOT of the environment stuff. For instance, the home folder for Cygwin is not the Windows home folder. This is a big problem in itself. And 'C:' is '/cygdrive/c' in Cygwin. So when you type in C: on your Skyscraper command line it is not going to work for that reason alone.

So there's a pointer for your further investigation. Simply DON'T use Cygwin and use the official tools provided by Qt instead.

muldjord

@livefastcyyoung I will add .7z support for all platforms, no problem. :) I will probably do a quick release with just this fix soon. Look out for it on the github page, and here.

muldjord

Skyscraper 2.0.5b released: https://github.com/muldjord/skyscraper

Added support for '*.7z' files for all platforms

Happy scraping! :)

bleuge

@muldjord ok, understood, thanks! I'll try to compile Windows native, thanks!

muldjord

@bleuge Good luck with it! :)

A Former User

Has anyone made a config file that would format images the same way sselph’s scraper does?

Every way I’ve tried changing this config to match, it’s still not quite right.

jura

@muldjord , your tool works amazingly great!
This good example when you are good at something and your are doing something for the community! :)

I have 700 nes and 700 snes and 1000 megadrive roms and scraping those took 30mins.
Which was amazing!

It found 80% of the covers for the roms but for c64 it found only 700 when i have 17 000 (17 thousand) roms. :)
Dunno is it possible to set the match percentage as a parameter to allow lower findings, or is it even necessary.

Any idea how I could increase the result persentage at least for C64?

dylosaur

@muldjord I'm really liking your scraper after trying a couple out. One tiny quip I was hoping I could find a solution for-- is there a way to make it so that the game name matches the file name after running several scrapers/using the localdb? For instance, my Atari 2600 "Bowling" now says "32-In-1 - 30 - Bowling" which is technically correct, but a little more cluttered than I'd like on the screen. Was going to just go in and edit the XML but figured I'd check if there was a command I could use to automate instead.

muldjord

@dylosaur Glad you like it. No, currently there is no way of forcing it to use the file name instead of the search result name.

muldjord

@jura Changing the percentage match has been possible for a looong time :). Use '-m [minimum percentage]'. Remember to read the readme and check the '--help' output. It's all in there.

dylosaur

@muldjord Thanks man. One more question: I want to just scrape screenshots with no drop shadow added. Am I correct in saying that if my config file just says the following, then I'll just get a centered, 500 x 375 screenshot? Not really sure what the X/Y tags are for, is that for position? Just want to make sure it's centered in the box.

[artwork]
screenshotEnabled="true"
screenshotWidth="500"
screenshotHeight="375"
screenshotX="47"
screenshotY="0"

EDIT: Just scanned through this thread and saw your response to somebody else regarding this. Having some trouble getting the import system to work. If I just do screenshots, it seems to work just fine, but when I add textual data on top of that, it no longer works for either the screenshot or the text data. Any solutions here?

dylosaur

@muldjord Here's what my text file looks like for MegaMania - A Space Nightmare:

<game>
<description>Megamania is an arcade action game with gameplay similar to Space Invaders. You control a spaceship at the bottom of the screen and need to defend yourself from numerous waves of attackers. But in Megamania, the attackers much more bizarre than invading aliens. There are hamburgers, diamonds, steam irons, bow ties, space dice, and more! You need to earn points by destroying as many of these strange attackers as you can before your energy supply becomes depleted.</description>
<developer>Activision</developer>
<publisher>Activision</publisher>
<players>1-2</players>
<genre>Action</genre>
<releasedate>1982</releasedate>
</game>

And the definitions file:

<game>
  <description>###DESCRIPTION###</description>
  <developer>###DEVELOPER###</developer>
  <publisher>###PUBLISHER###</publisher>
  <players>###PLAYERS###</players>
  <rating>###RATING###</rating>
  <genre>###TAGS###</genre>
  <releasedate>###RELEASEDATE###</releasedate>
</game>

Actually, I just did some tinkering around, and it seems it can't do pictures and text at the same time. Once I separated them it started working. Just wanted to point that out to you.