Versatile C++ game scraper: Skyscraper

[AnalogHero]

@muldjord Yeah, true. Scraping with crc doesnt make sense here. Each crack would have a different checksum. Maybe going by name is better. Is this possible with screenscraper?

[muldjord]

@analoghero Unfortunately not. They only support checksum search and EXACT filename match (which is also one of the parses I currently do). So you'd have to use thegamesdb.

[muldjord]

Ok, so I am looking into creating a gui for Skyscraper, and I've been looking for a lightweight gui widget framework that works on the RetroPie distro without too much fuzz. I am currently using the Qt framework for a lot of the stuff in Skyscraper, which is really useful. BUT, I can't for the life of me get Qt to work without X11, which it can, but it just freezes my Pi.

So, any suggestions for lightweight gui frameworks? I thought about doing some with ncurses or similar, but I reeeeeally would like to have a graphical one and not a text-based one.

[mitu]

@muldjord Try running some of the samples from AGAR, it seems it can run of SDL and seems to support both framebuffer and GL based contexts.

EDIT: Scratch that, It seem the project is not active anymore and the SDL version supported is 1.2.

[Deltax5]

how do I install it. WHere do I put the files? I put the commend and says it can't find gt5-defalt packege. You need to explain where to put the files.

How to install Skyscraper

Install prerequisites

Install this package:

• $ sudo apt-get install qt5-default
• [enter your 'pi' user password, default is 'raspberry']
  I skiped this since it doesn't work.

Download and compile

Create a folder for the Skyscraper source, download the latest release, compile it and install it:

• $ cd /home/pi
• $ mkdir sources
• $ cd sources
• $ wget https://github.com/muldjord/skyscraper/archive/2.0.3.tar.gz
• $ tar xvzf 2.0.3.tar.gz
• $ cd skyscraper-2.0.3

Doesn't find these

• $ qmake
• $ make
• $ sudo make install
• [enter your 'pi' user password, default is 'raspberry']

[muldjord]

You don't put the files anywhere. Just follow the guide. Type in the commands exactly as they are shown but without the '$' and it will work if you are using a RetroPie image. If not, you are doing something wrong.

[Deltax5]

@muldjord i followed it.
$ sudo apt-get install qt5-default can't find file
i did the rest got to the point where i have to put qmake and make and those can't be found. IDK whats going on. I am typing the commands in puddy with retropi is in f4.

[muldjord]

@deltax5 I can't help you then. Either your system is broken, or you are not typing in the command properly. "sudo apt-get install qt5-default" WILL work on a RetroPie image.

Note: If the prerequisites part won't work for you the rest won't either. So that's why you are having problems with the remaining commands.

[fluffypillow]

@muldjord for the gui, have you tried Qt Quick/QML?

[Deltax5]

@fluffypillow whats that sorry im new to this is that a command?

[Deltax5]

@muldjord that command works now thanks going to try the rest now

[fluffypillow]

@Deltax5 not, that's a GUI framework. It works without X11 and since Skyscraper is already based on Qt, it should be easier to integrate it with a GUI written in Qt Quick.

[muldjord]

@fluffypillow Qt Quick is just a different way of writing Qt programs. The rendering abstraction is still the same. Qt doesn't require X11, but the eglfs and linufb pipelines don't work for me, so unfortunately it isn't an option. And I don't want X11 as a requirement. But thank you for the suggestion. If I could in fact just write the UI using Qt that would be the best option, but I've spend hours trying to get it to work without X11. So far no luck.

[Deltax5]

pi@retropie:~/sources/skyscraper-2.0.3 $ qmake
Failure to open file: /home/pi/sources/skyscraper-2.0.3/Makefile
Unable to generate makefile for: /home/pi/sources/skyscraper-2.0.3/skyscraper.pr

qmake doesn't work?

[Deltax5]

@deltax5 just did it in root and it's working hope gamegear works with this, I have tried stephin and XMLscraper and those didn't find gamegear games for some reason.

[muldjord]

@deltax5 You should not do it as root, all of the files will be owned by root in the source dir if you do, and you then won't be able to recompile it later without being root, which is pretty inconvenient. Instead pay more attention that you type the commands correctly. You were missing an "o" in the skyscraper.pro file.

EDIT: Ah, no that was not the problem. The problem was probably that you had already done some of it as root. Always pay attention to when you do stuff as root and when you do stuff as a user. You will eventually run into these problems if you don't. qmake is complaining because the Makefile can't be created. This is because you have a permission problem, which is probably because you created the folder as root. I would suggest deleting the folder and starting over. But that's of course up to you.

[Deltax5]

sigh
root@retropie:/home/pi/sources/skyscraper-2.0.3# Skyscraper -p gamegear -s local db --videos

Running Skyscraper v2.0.3 by Lars Muldjord

Platform : 'gamegear'
Scraper module : 'localdb'
Input folder : '/home/pi/RetroPie/roms/gamegear'
Gamelist folder : '/home/pi/RetroPie/roms/gamegear'
Images folder : '/home/pi/RetroPie/roms/gamegear/images'
Videos folder : '/home/pi/RetroPie/roms/gamegear/videos'
Local db folder : 'dbs/gamegear'

Reading and parsing local database, please wait...
Successfully parsed 0 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!

'gamelist.xml' already exists, do you want to overwrite it? (y/n): y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries? (y/n): n

Starting scraping run on 1 files using 4 threads.
Sit back, relax and let me do the work! :)

#1/1 ---- Game 'Pac-Man (USA, Europe)' not found :( ----

Elapsed time: 00:00:00
Estimated time: 00:00:00

---- Scraping run completed! YAY! ----
Writing 0 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/gamegear/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Total number of games: 1
Successfully scraped games: 0
Skipped games: 1 (Filenames saved to 'skipped-localdb.txt')

Why can't I ever fined gamegear games. It finds them fine in the scraper in retropie but thats anoying can't scrap one by one.

[muldjord]

@deltax5 Dude, you can't just scrape with '-s localdb' first. You need to run it with '-s thegamesdb' or '-s screenscraper' first. Your 'localdb' is currently empty. Please read the readme for information on it.

Also, you are STILL running commands as root. This will give you a lot of permission problems later on. In fact, currently all of your resources will be cached in the wrong place and all of the files that are created will be owned by root making it quite difficult to work with them later.

I strongly suggest you start all over. Delete your 'sources' folder as root. Then go back to being the normal 'pi' user and follow the howto in the readme line by line exactly as it is shown. Otherwise I won't be able to help you out with the problems you encounter.

Skyscraper WILL find your Game Gear games if you use it properly.

I'm sorry if I seem a bit harsh in my comments here, but you seem to be making the situation more difficult for yourself by the minute, sorry. :S

EDIT: Now that I think about it, your gamelist.xml is already owned by 'root' because you've been running it as root, and so are the images and videos folder... So you'd have to fix that first before starting over.

[Deltax5]

deleted the source as root. did it again with pi now it works but still no luck. I don't understand. im still a noob im sorry. it don't like gamegear for some reason.
im missing something I followed the instructions in the read me.
pi@retropie:~/sources/skyscraper-2.0.3 $ Skyscraper -p gamegear -s thegamesdb --videos

Running Skyscraper v2.0.3 by Lars Muldjord

Platform : 'gamegear'
Scraper module : 'thegamesdb'
Input folder : '/home/pi/RetroPie/roms/gamegear'
Gamelist folder : '/home/pi/RetroPie/roms/gamegear'
Images folder : '/home/pi/RetroPie/roms/gamegear/images'
Videos folder : '/home/pi/RetroPie/roms/gamegear/videos'
Local db folder : 'dbs/gamegear'

Reading and parsing local database, please wait...
Successfully parsed 9 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!

'gamelist.xml' already exists, do you want to overwrite it? (y/n): y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries? (y/n): y
Parsing existing xml entries...
Resolving missing entries...
No entries to scrape...

---- Scraping run completed! YAY! ----
Writing 9 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/gamegear/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Total number of games: 0
Successfully scraped games: 0
Skipped games: 0 (Filenames saved to 'skipped-thegamesdb.txt')

pi@retropie:~/sources/skyscraper-2.0.3 $ Skyscraper -p gamegear -s screenscraper --videos

Running Skyscraper v2.0.3 by Lars Muldjord

Platform : 'gamegear'
Scraper module : 'screenscraper'
Input folder : '/home/pi/RetroPie/roms/gamegear'
Gamelist folder : '/home/pi/RetroPie/roms/gamegear'
Images folder : '/home/pi/RetroPie/roms/gamegear/images'
Videos folder : '/home/pi/RetroPie/roms/gamegear/videos'
Local db folder : 'dbs/gamegear'

Reading and parsing local database, please wait...
Successfully parsed 9 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!

'gamelist.xml' already exists, do you want to overwrite it? (y/n): y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries? (y/n): y
Parsing existing xml entries...
Resolving missing entries...
No entries to scrape...

---- Scraping run completed! YAY! ----
Writing 9 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/gamegear/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Total number of games: 0
Successfully scraped games: 0
Skipped games: 0 (Filenames saved to 'skipped-screenscraper.txt')

[muldjord]

Try answering "n" to skip existing. If your gamelist.xml somehow has the entry in it, you are telling it to not retry it by answering "y" to skip. That's why it doesn't try to scrape anything. Other than that, you seem to be doing it correctly now.