Versatile C++ game scraper: Skyscraper
-
@fluffypillow Qt Quick is just a different way of writing Qt programs. The rendering abstraction is still the same. Qt doesn't require X11, but the eglfs and linufb pipelines don't work for me, so unfortunately it isn't an option. And I don't want X11 as a requirement. But thank you for the suggestion. If I could in fact just write the UI using Qt that would be the best option, but I've spend hours trying to get it to work without X11. So far no luck.
-
pi@retropie:~/sources/skyscraper-2.0.3 $ qmake
Failure to open file: /home/pi/sources/skyscraper-2.0.3/Makefile
Unable to generate makefile for: /home/pi/sources/skyscraper-2.0.3/skyscraper.prqmake doesn't work?
-
@deltax5 just did it in root and it's working hope gamegear works with this, I have tried stephin and XMLscraper and those didn't find gamegear games for some reason.
-
@deltax5 You should not do it as root, all of the files will be owned by root in the source dir if you do, and you then won't be able to recompile it later without being root, which is pretty inconvenient. Instead pay more attention that you type the commands correctly. You were missing an "o" in the skyscraper.pro file.
EDIT: Ah, no that was not the problem. The problem was probably that you had already done some of it as root. Always pay attention to when you do stuff as root and when you do stuff as a user. You will eventually run into these problems if you don't. qmake is complaining because the Makefile can't be created. This is because you have a permission problem, which is probably because you created the folder as root. I would suggest deleting the folder and starting over. But that's of course up to you.
-
sigh
root@retropie:/home/pi/sources/skyscraper-2.0.3# Skyscraper -p gamegear -s local db --videosRunning Skyscraper v2.0.3 by Lars Muldjord
Platform : 'gamegear'
Scraper module : 'localdb'
Input folder : '/home/pi/RetroPie/roms/gamegear'
Gamelist folder : '/home/pi/RetroPie/roms/gamegear'
Images folder : '/home/pi/RetroPie/roms/gamegear/images'
Videos folder : '/home/pi/RetroPie/roms/gamegear/videos'
Local db folder : 'dbs/gamegear'Reading and parsing local database, please wait...
Successfully parsed 0 resources!Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!'gamelist.xml' already exists, do you want to overwrite it? (y/n): y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries? (y/n): nStarting scraping run on 1 files using 4 threads.
Sit back, relax and let me do the work! :)#1/1 ---- Game 'Pac-Man (USA, Europe)' not found :( ----
Elapsed time: 00:00:00
Estimated time: 00:00:00---- Scraping run completed! YAY! ----
Writing 0 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/gamegear/gamelist.xml'... Success!!!---- And here are some neat stats :) ----
Total completion time: 00:00:00Total number of games: 1
Successfully scraped games: 0
Skipped games: 1 (Filenames saved to 'skipped-localdb.txt')Why can't I ever fined gamegear games. It finds them fine in the scraper in retropie but thats anoying can't scrap one by one.
-
@deltax5 Dude, you can't just scrape with '-s localdb' first. You need to run it with '-s thegamesdb' or '-s screenscraper' first. Your 'localdb' is currently empty. Please read the readme for information on it.
Also, you are STILL running commands as root. This will give you a lot of permission problems later on. In fact, currently all of your resources will be cached in the wrong place and all of the files that are created will be owned by root making it quite difficult to work with them later.
I strongly suggest you start all over. Delete your 'sources' folder as root. Then go back to being the normal 'pi' user and follow the howto in the readme line by line exactly as it is shown. Otherwise I won't be able to help you out with the problems you encounter.
Skyscraper WILL find your Game Gear games if you use it properly.
I'm sorry if I seem a bit harsh in my comments here, but you seem to be making the situation more difficult for yourself by the minute, sorry. :S
EDIT: Now that I think about it, your gamelist.xml is already owned by 'root' because you've been running it as root, and so are the images and videos folder... So you'd have to fix that first before starting over.
-
deleted the source as root. did it again with pi now it works but still no luck. I don't understand. im still a noob im sorry. it don't like gamegear for some reason.
im missing something I followed the instructions in the read me.
pi@retropie:~/sources/skyscraper-2.0.3 $ Skyscraper -p gamegear -s thegamesdb --videosRunning Skyscraper v2.0.3 by Lars Muldjord
Platform : 'gamegear'
Scraper module : 'thegamesdb'
Input folder : '/home/pi/RetroPie/roms/gamegear'
Gamelist folder : '/home/pi/RetroPie/roms/gamegear'
Images folder : '/home/pi/RetroPie/roms/gamegear/images'
Videos folder : '/home/pi/RetroPie/roms/gamegear/videos'
Local db folder : 'dbs/gamegear'Reading and parsing local database, please wait...
Successfully parsed 9 resources!Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!'gamelist.xml' already exists, do you want to overwrite it? (y/n): y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries? (y/n): y
Parsing existing xml entries...
Resolving missing entries...
No entries to scrape...---- Scraping run completed! YAY! ----
Writing 9 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/gamegear/gamelist.xml'... Success!!!---- And here are some neat stats :) ----
Total completion time: 00:00:00Total number of games: 0
Successfully scraped games: 0
Skipped games: 0 (Filenames saved to 'skipped-thegamesdb.txt')pi@retropie:~/sources/skyscraper-2.0.3 $ Skyscraper -p gamegear -s screenscraper --videos
Running Skyscraper v2.0.3 by Lars Muldjord
Platform : 'gamegear'
Scraper module : 'screenscraper'
Input folder : '/home/pi/RetroPie/roms/gamegear'
Gamelist folder : '/home/pi/RetroPie/roms/gamegear'
Images folder : '/home/pi/RetroPie/roms/gamegear/images'
Videos folder : '/home/pi/RetroPie/roms/gamegear/videos'
Local db folder : 'dbs/gamegear'Reading and parsing local database, please wait...
Successfully parsed 9 resources!Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!'gamelist.xml' already exists, do you want to overwrite it? (y/n): y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries? (y/n): y
Parsing existing xml entries...
Resolving missing entries...
No entries to scrape...---- Scraping run completed! YAY! ----
Writing 9 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/gamegear/gamelist.xml'... Success!!!---- And here are some neat stats :) ----
Total completion time: 00:00:00Total number of games: 0
Successfully scraped games: 0
Skipped games: 0 (Filenames saved to 'skipped-screenscraper.txt') -
Try answering "n" to skip existing. If your gamelist.xml somehow has the entry in it, you are telling it to not retry it by answering "y" to skip. That's why it doesn't try to scrape anything. Other than that, you seem to be doing it correctly now.
-
you know what my problem is. I notice the 1 file it found was .zip the others where .7zip. for some reason any auto scraper doesn't recognize .7zip. unziped them now it detects them fine. the data base is getting updated right now let you know if it worked.
I got a question. Lets say I made a custom folder for famicom for example. called fc. how do I scrap them proper? -
@deltax5 You can set the rom input folder manually with '-i' followed by the folder name. Just check the readme or type "Skyscraper --help". :)
-
ok ill look it up later today can you give me an exampe
-
@deltax5 Maybe you should be a little more polite and patient when asking for anyone's help; doubly so when it's the dev of the application that you want to use. This is not your personal help desk.
-
I thank everyone who helped me. It was acting weird at first for some reason. Wasn't tring to be rude or anything. im still a noob and really don't know much about coding other then following the instructions.
-
Alright idk whats going on. The images are fine just the video's don't show up. nosing around in Winscp they actually downloaded. The thing is that it's not playing. It's in the /home/pi/skyscraper/dbs/gamegear/videos/screenscraper. The video's are labled a bunch of gibberish.
Do I have to edit the gamelist.xml or something.
Once again im sorry if I bothered anyone. This is getting annoying for me also. -
@deltax5 Run: "Skyscraper -p gamegear -s localdb --videos --unattend" and it will work.
Lastly, a couple lessons to take away from this:
- When following a howto and a command fails, double check that you wrote the command correctly before asking for help.
- If a command still fails, then don't expect the rest of the howto to work. Instead, ask for help from the community, but please remember point 1!
- Read the README's thoroughly before asking questions. It's kind of the whole point of a readme, and I spend quite a lot of time writing them. Don't expect me to answer questions explained in the readme's, it's annoying and makes me think that you basically just want me to do the work for you, not putting in the required time to try and understand it yourself.
I don't mind people asking questions, but it is common courtesy to adhere to these simple points.
Ok, so with that said, I am glad you got it to work.
-
@deltax5 said in Versatile C++ game scraper: Skyscraper:
you know what my problem is. I notice the 1 file it found was .zip the others where .7zip. for some reason any auto scraper doesn't recognize .7zip.
Because most emulators don't use 7zip so why would anyone need to scrape them? The scraping is done by MD5, SHA1, CRC and name, if you rename a file, the MD5, SHA1 and CRC don't change, this is why we use these bits of information.
On the database side where these are scraped from, someone needs to physically link the bits of information to the game, no one links 7zip files because as I said before, how many emulators actually read a 7zipped file?
-
@bladehunter it played fine on retro pi. I got my roms in 7z for some reason. Atleast I noticed it and pass it on if someone else has trouble in the future.
-
Is there a way to limit what types of resources are scraped? For example, I'm after text resources (descriptions, ratings, etc) but not images.
-
@kb1ujs Videos are off by default (turn on with '--videos'), but images are always scraped and can't be disabled.
-
@muldjord Just a small bugreport from what ive learned about videoscreensaver: Skyscrapers gamelist.xml files contain empty video tags looking like this
<video />This is a problem at the moment when using videoscreensavers. Could you make those tags optional somehow?
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.