Versatile C++ game scraper: Skyscraper
-
@muldjord Dont know what this is. I wanted to update my dbs as there are no wheels scraped. The command im using was :
Skyscraper -p megadrive -s screenscraper --updatedb -
@analoghero Could you post the output, please? I still don't understand what you mean. Just tested it here with screenscraper on megadrive platform and it works well.
-
pi@retropie:~ $ Skyscraper -p megadrive -s screenscraper --updatedb ------------------------------------------ Running Skyscraper v2.3.0 by Lars Muldjord ------------------------------------------ Platform : 'megadrive' Scraper module : 'screenscraper' Input folder : '/home/pi/RetroPie/roms/megadrive' Game list folder : '/home/pi/RetroPie/roms/megadrive' Covers folder : '/home/pi/RetroPie/roms/megadrive/media/covers' Screenshots folder : '/home/pi/RetroPie/roms/megadrive/media/screenshots' Wheels folder : '/home/pi/RetroPie/roms/megadrive/media/wheels' Marquees folder : '/home/pi/RetroPie/roms/megadrive/media/marquees' Local db folder : 'dbs/megadrive' Reading and parsing local database, please wait... Successfully parsed 9285 resources! Looking for optional 'priorities.xml' file in local db folder... Found! Priorities loaded successfully! 'gamelist.xml' already exists, do you want to overwrite it (y/N)? y Checking if 'gamelist.xml' is writable?... It is! :) Do you wish to skip existing entries (y/N)? n Starting scraping run on 628 files using 4 threads. Sit back, relax and let me do the work! :) #1/628 Pass 1 ---- Game 'Empire of Steel (Europe)' found! :) ---- From cache: NO Search match: 0 % Game title: '.hack-Link' () Compare title: 'Empire of Steel' Platform: 'PSP' () Release Date: '' Developer: 'Bandai Namco' () Publisher: 'CyberConnect2' () Players: '' () Tags: 'Role playing games' () Rating (0-1): '' () Cover: YES () Screenshot: YES () Wheel: NO () Marquee: NO () Description: () The first game in the .hack series for PSP (and the planned final game for the franchise), .hack//LINK logs player into a new version of its virtual landscape called The World R:X (the "R" stands for "Revision"). Set 10 years after the last .Hack, players take control of Tokio Kuryu, a second year junior-high student. Presented through manga-style visuals, the game's story promises to clear up the mysteries from past entries. Over 100 characters from past .hack games, anime, manga, and books will make an appearance. Gameplay promises to retain the basics of past titles, with players facing off in battle against enemies as they explore dungeons. The difference here is that you move around in a party of two, with the CPU-controlling the other character. The game will include 33 such CPU-controlled characters. For the PSP game, the battle system has been changed to a more action-heavy combat system. Elapsed time: 00:00:04 Estimated time: 00:48:06 #2/628 Pass 1 ---- Game 'Spirou (Europe) (En,Fr,De,Es)' found! :) ---- From cache: NO Search match: 40 % Game title: '.hack-Link' () Compare title: 'Spirou' Platform: 'PSP' () Release Date: '' Developer: 'Bandai Namco' () Publisher: 'CyberConnect2' () Players: '' () Tags: 'Role playing games' () Rating (0-1): '' () Cover: YES () Screenshot: YES () Wheel: NO () Marquee: NO () Description: () The first game in the .hack series for PSP (and the planned final game for the franchise), .hack//LINK logs player into a new version of its virtual landscape called The World R:X (the "R" stands for "Revision"). Set 10 years after the last .Hack, players take control of Tokio Kuryu, a second year junior-high student. Presented through manga-style visuals, the game's story promises to clear up the mysteries from past entries. Over 100 characters from past .hack games, anime, manga, and books will make an appearance. Gameplay promises to retain the basics of past titles, with players facing off in battle against enemies as they explore dungeons. The difference here is that you move around in a party of two, with the CPU-controlling the other character. The game will include 33 such CPU-controlled characters. For the PSP game, the battle system has been changed to a more action-heavy combat system. Elapsed time: 00:00:05 Estimated time: 00:30:12 ^C -
@analoghero Ok, that's a game called hack-LINK that gets connected to the wrong data... Would it at all be possible for you to compress your ~/.skyscraper/dbs/megadrive folder into a file and upload it somewhere where I can download it? I would be very interested in figuring out what is going on.
-
@muldjord Im zipping it right now. Note that the same thing happens for other system too. Tried amiga, same result.
-
@analoghero I think I have a lead now... I think it's related to how I handle platform detection with the screenscraper module in 2.3.0. What I don't quite understand is why it returns the "hack-LINK" game in the first place, that shouldn't happen at all!
I have to go to work now, your help is appreciated. I really want to squash this bug, so I'll be back later to ask more questions if that's ok. In the meantime, can I ask you to keep your dbs's as they are? I would hate for this problem to go away without me knowing what happened. :)
-
@muldjord Uploaded it.
-
Japanese-release only PSP title being returned for all scrapes? That's an odd one. :-|
-
@analoghero Thank you so much, I've downloaded it. Could you also do me the favor of running the command "ls -l /home/pi/RetroPie/roms/megadrive >> /home/pi/megadrive_ls.txt" and send me the megadrive_ls.txt it creates? I need as much information as I can get to reproduce this.
-
@analoghero Also, could you try running it with "Skyscraper -p megadrive -s screenscraper --updatedb --region wor" and see if that still returns the ".hack-Link" results?
-
@muldjord Ive uploaded to the same folder as before.
Edit: The command above has the same result.
-
https://retropie.org.uk/forum/topic/16059/sselph-scrapper-getting-strange-results
Sselph scraper has the same issue with scraping that .hack//link game.
-
@livefastcyyoung Thank you!!! This is really helpful, it tells me a lot about what is going on. I still don't know why it scrapes ALL games like this for AnalogHero though. I'll investigate this further and provide a fix once I figure out how to do it properly.
For now, if any of you guys are having the same issue, please let me know!
EDIT: It seems to be an error on screenscrapers part, returning faulty results, maybe because of some server issues they are having. So this problem might go away by itself. Still problematic though, as the localdb will be updated with this faulty data, so anyone haveing the issue will have to rescrape with '-s screenscraper --updatedb' once the problem is fixed on their end.
-
@muldjord After further tests on my end, i think screenscraper has issues as you said. Even when scraping a single rom it returns .hack-link :(
-
@analoghero yeah, but it doesn't seem to be a general problem. I don't get the errors here. Maybe they have 1 server that causes this. And because they use load-balancing I just happen to reach one of their servers that works fine.
-
@muldjord Not your fault ofcourse, but this problem makes 2.3.0 not very usefull atm, because screenscraper has the most images.
-
@analoghero Yes, agreed. But as you say, I can't really fix that unfortunately... I will look into the problem further when I get home though. I'm curious as to what is going on and would like to know why it returns the ".hack-Link" result in the first place... :S
EDIT: And let me just say that I find it quite unfortunate that it had to coincide with my 2.3.0 release. :D I had been testing and testing and to read about such a weird problem seemingly breaking the results for you made me think I had overlooked something really obvious... So I'm "glad" it is seemingly not a problem on my end. Sad that screenscraper is having issues of course, it really is a fantastic source when it works.
-
@screech Might be able to add some suggestion as to what is happening at Screenscraper. Unfortunately it looks like he hasn't been on the forum in the last month.
-
@muldjord many many thanks for the hardwork in this version, as i said, best scraper around.
@analoghero said in Versatile C++ game scraper: Skyscraper:
.hack link
I was getting this a lot other day,
Maybe Skyscraper simple could check if .hack link is returned from screenscraper, and if the file is not .hack link (words in filename), simple skip it?. Better to have nothing that a wrong result? -
@bleuge Agreed, I would like to filter those out so it doesn't overwrite valuable data in the localdb. Still curious as to why screenscraper returns it in the first place. I hope they know about it and fix it.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.