Versatile C++ game scraper: Skyscraper
-
Just tried this out for the first time by scraping my NES collection.
Wasn't terribly pleased with how it came out so I wanted to get rid of everything and rescrape with a different scraper. However, no matter what I do, I can't get rid of the descriptions and images this downloaded.
I have deleted the gameslist and downloaded_images from /opt/retropie/configs/all/emulationstation. There is nothing in those folders but the images and descriptions still show up in EmulationStation.
I have rebooted. Is there anywhere else these are stored? Thanks!
-
@livefastcyyoung Read the readme. It's all well described in there. If you check the output of Skyscraper at the beginning, you'll also notice where the data is saved.
Concerning your trouble with descriptions you don't like, you can either delete the entire local nes database cache by deleting '~/.skyscraper/dbs/nes' and start over. I wouldn't do that though, I instead recommend simply finding out which source you prefer to have the descriptions coming from, and prioritizing that source for description in '~/.skyscraper/dbs/nes/priorities.xml', and then scrape using '-s localdb' afterwards.
Feel free to ask further questions, but please read the readmes thouroughly first. :) Thanks.
-
I have deleted the entire /.skyscraper folder and it's still displaying the Skyscraper scraped art and game info. I'm at a loss.
-
@livefastcyyoung Deleting Skyscrapers local resources won't alter your gamelist for EmulationStation. You need to delete the Emulationstation specific gamelist.xml and images folder to do that. As I mentioned, both are listed at the beginning when you scrape the platform.
-
I have done that. There is no folder for NES in the downloaded_images and the gameslist.
I guess I'll have to just keep digging. I have deleted these before with no issue to rescrape. I have never had this issue before.
-
@livefastcyyoung Those are not the right folders. Unless you set them manually when scraping, the correct folder is ~/RetroPie/roms/nes/images and ~/RetroPie/roms/nes/gamelist.xml. Doesn't it say that when you run the scraper? It should clearly state a gamelist and images folder at the very beginning.
Btw, you DON'T need to delete those to rescrape. Just choose to overwrite and NOT skip. But please, read the readme's.
-
@muldjord Thank you for your advice on scripting the scraping. I have now done so but, alas, now find my IP banned from the games database. According to https://www.gameex.info/forums/topic/10930-gamesdbase/ - Would it be possible to add emumovies as a scrape resource, not only do they promote sharing but they are partnered with the games database and I have a lifetime sub with emumovies too....
Thanks again!
-
@LocVez Yes, gamesdatabase will ban you if you overuse it.
I had a quick look at emumovies, I'll ask them for developer access to the api and let you know if I get it. Then I will include it if the API they provide fits with Skyscraper. It probably does, but I don't know atm. :)
-
@muldjord They probably will. Just as long as you can input your username and password.
Been a lifetime member there for ages.
-
Opps dobbel post..
-
@muldjord Many thanks!
-
@LocVez They banned me in the past, but after 2 weeks it worked again. Im glad it wasnt forever.
-
Wooooooo!!! Video tutorial!!!
Please spread the word! :)
Please, please, PLEASE check this out if you are new to Skyscraper. I have a feeling most people only use Skyscraper to just scrape once using the default scraper and then leave it like that. It really is too bad, since the most important feature in Skyscraper is the local db cache. This video explains why it's important.
-
@muldjord Thanks for the video. As i see now, i did it right. What bugs me a little is what happend in your video.
After the default scrape, you scraped with thegamesdb.
Then you scraped with uvlist and afterwards with localdb.
Result: 1 game skipped.
Then you scraped with screenscraper and afterwards localdb.
Result: 2 games skipped.Hope you understand what i try to say. :)
This happened on my retropie aswell. -
@analoghero Ah, I know why! It's because the search match becomes too low for one of the games because of how the title is prioritized from the different scraping modules. So one title makes the search match go below 50% which it didn't when scraping with the source originally. So basically it is /meant/ to do this, but I should still look into it. I'll figure something out for 1.8.3.
Bottom line, it's not an error, just lower the minMatch to '0' with '-m 0' when scraping with localdb and you'll get the nice result again. :) But I do want this to make sense to the user, so I'll do something about it anyways.
I do have another very minor bug I found while investigating this though, so I'm glad you brought it up. Either way, 1.8.2 is working just fine, so no need to hold your breath for 1.8.3. :)
EDIT: I've had some more time to think about this "problem". And I've decided to leave it, simply because it isn't an error at all. I can see how it seems confusing that addmore resources from more scrapers suddenly makes it skip more games than previously. But the explanation is simply that the title of a game from a certain module is prioritized above the other titles, and that title makes is go below the minimum match percentage threshold. This is easily "fixed" by lowering the threshold. And it is /supposed/ to work like this. The priorities can easily be changed inside the 'priorities.xml' file as described in the documentation on github. So in the case of the Youtube howto I made, lowering the 'title' priority for 'screenscraper' to go below that of 'mobygames' will /also/ make the "problem" go away. I am absolutely in agreement that this seems puzzling if you don't understand or make use of the priorities. It seems like it "going backwards" when you scrape with more scrapers, when in fact it just got a title from a new scraper that was of higher priority and then gave a worse percentual match to the filename. That's all it is. And it's not an error. :)
-
Every one of my gamelists are in home/pi/RetroPie/Gamelists.
That way when I re-run the script meleu has been working on and they get updated there, the symbolic links from the location that Emulation Station looks for them link to all of the files in a single folder and they automatically work on a reboot instead of having to manually move 12 to 15 files to as many locations when an update is made.
Might want to look into doing something like that yourself. Saves a ton of time, especially when you're probably updating things all the time with the testing you're doing.
-
EmuMovies has accepted my request for developer access. I'll get the details soon and start looking into an implementation. I'll keep you posted. :)
-
@muldjord Is there a way to have boxart without a drop shadow and not centered. Like just a raw boxart image?
-
@hurricanefan Yes, please read the readme on github.
-
I've noticed that out of 700 NES titles, there's 32 that your scraper can't recognize purely because the article (the, a) is already at the end in the original file name (As per GOOD tools standard), messing with the name recognition. for example:
Punisher, the
Simple name, but big problems finding it even though the default emulationstation scraper finds it no problem. for now I'll simply rename the offending roms and re-run the scraper when I have time, but I thought this might be worth mentionning. Originally there was 50 titles, but after running from multiple source it dropped down to 32.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.