Versatile C++ game scraper: Skyscraper

muldjord · 3 Apr 2020, 19:45

@AlCzervik Hmm, actually that response is also shown if your credentials are wrong. It needs to look exactly like this in the config.ini:

[screenscraper]
userCreds="user:pass"

But with your own user and pass of course. If you've entered that correctly and don't have any stray configs messing with that, it will work.

AlCzervik · 3 Apr 2020, 19:48

@muldjord That's helpful! Quotes required? I know use case is important on the variable, is it important with the username/password? Perhaps I should inquire with screenscraper directly to ensure there's nothing wrong with my account? I only recently got to the point where I can download data, perhaps that's the issue.

muldjord · 4 Mar 2020, 21:19

@AlCzervik It could be. The service is currently closed for some users. I'm not sure what tiers are closed exactly.

I don't think the quotes are necessary but I know it works with them, so I would use them. But if you get that error at the beginning it either means the server is down or you entered the credentials incorrectly. I'm guessing the latter as many people seem to get it wrong.

AlCzervik · 3 Apr 2020, 20:39

@muldjord Thanks for your help. I'll have to start with them if that response is related to credentials. And I've checked and rechecked that it is spelled correctly in the config.ini file. Not sure what else to try at this point. Thanks again.

muldjord · 3 Apr 2020, 20:48

@AlCzervik If you paste your config.ini as a code-block (BE SURE TO REMOVE YOUR USER + PASS) I can take a look at it.

AlCzervik · 4 Apr 2020, 05:06

@muldjord

 # --------------------------------------------------------------------
# Skyscraper by Lars Muldjord (https://github.com/muldjord/skyscraper)
# --------------------------------------------------------------------
 
# This is an example config file for use with Skyscraper. Use it as a template for creating
#   your own. If you copy this file to 'config.ini' it will be used per default.
#   You can always copy config.ini.example to a filename of your choice and make Skyscraper
#   use it with the '-c' command line option.
 
# You can have several platform sections in one config file. Further down is an example of
#   a generic platform setup. Check '--help' for more info on supported platforms.
 
# You can have several scraping module sections in one config file. Further down is an example
#   of a generic module setup. Check '--help' for more info on supported modules.
 
# Remember that most of these settings can also be set as command line options. Doing so
#   will overrule the options seen here except for the scraping module configs which
#   overrules everything else.
# Platform specific configs overrules main configs.
 
# Uncomment the sections and lines you wish to use:
 
[main]
##Setting input, gamelist or media folder here, will automatically append '/[platform]' to path.
##If you need better control, use the same variables under a [<platform>] section instead.
#inputFolder="/home/pi/RetroPie/roms"
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#gamelistFolder="/home/pi/RetroPie/roms"
#mediaFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import"
#unpack="false"
#frontend="emulationstation"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#maxLength="10000"
#threads="2"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#maxFails="30"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#artworkXml="artwork.xml"
#relativePaths="false"
#addExtensions="*.zst"
#hints="false"
#subdirs="true"
#spaceCheck="true"
#scummIni="/full/path/to/scummvm.ini"
 
#[<PLATFORM, eg 'amiga'>]
#inputFolder="/home/pi/RetroPie/roms/amiga"
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#cacheFolder="/home/pi/.skyscraper/cache/amiga"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import/amiga"
#unpack="false"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#maxLength="10000"
#threads="2"
#startAt="filename"
#endAt="filename"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#artworkXml="artwork.xml"
#relativePaths="false"
#extensions="*.zip *.uae *.adf"
#addExtensions="*.zst *.rom"
#subdirs="true"
 
#[<FRONTEND, eg 'pegasus'>]
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#artworkXml="artwork.xml"
#emulator=""
#launch=""
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#skipped="false"
#brackets="true"
#videos="false"
#symlink="false"
#startAt="filename"
#endAt="filename"
#unattend="false"
#unattendSkip="false"
#forceFilename="false"
#verbosity="1"
#maxLength="10000"
 
[screenscraper]
userCreds="user:pass"
#threads="1"
#minMatch="0"
#maxLength="10000"
#interactive="false"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#videos="false"
#videoSizeLimit="42"

muldjord · 4 Apr 2020, 06:28

@AlCzervik That config.ini is correct.

AlCzervik · 4 Apr 2020, 16:14

@muldjord Thank you! But yeah, it's driving me crazy why it isn't working!!

muldjord · 4 Apr 2020, 18:33

@AlCzervik I assume you have gone to their website and logged on to check that your profile is active, right?

And just to give an example, this is how the entry would look if your username was mike and your pass was supersecret (not real, just came up with it):

[screenscraper]
userCreds="mike:supersecret"

AlCzervik · 6 Apr 2020, 13:53

@muldjord Thanks again. I've tried several things and nothing seems to work. So, to be clear, that error definitely means I'm not getting authenticated to ScreenScraper.fr correct? I know your program has worked with another source so I assume it is that connection to ScreenScraper. I'll try posting again in their forum for help. Any other ideas would be appreciated!

AlCzervik · 6 Apr 2020, 15:47

@muldjord A few other notes, it's correctly stating "Fetching limits for user 'User', so I know it's grabbing it properly there. Then I get the "Received invalid ScreenScraper server response..." message.

muldjord · 6 Apr 2020, 18:27

@AlCzervik Can you try and update Skyscraper and try again? I just updated it to 3.4.4 today, and it will now tell you if the credentials are rejected.

AlCzervik · 6 Apr 2020, 18:50

@muldjord Updated to 3.4.4. Similar error. First I get the "Fetching limits for user 'User', just a sec..." After that, I now get "Received invalid / empty ScreenScraper server response..."

muldjord · 4 Jun 2020, 20:02

@AlCzervik Ok, that is really, really odd. Then it's not a login error. Can you please verify that it says Skyscraper 3.4.4 when you run it?

EDIT: Ah, yes, it is. Because the error message is updated with "empty". In that case, I have no idea what is going on. I can't reproduce that error here. Can you try and change your password to something else and retry it with that? I'm thinking that maybe it is an encoding error.

AlCzervik · 4 Jun 2020, 20:09

@muldjord I'll try the password change. Here's the screenshot:

muldjord · 4 Jun 2020, 20:13

@AlCzervik I can't see that, it is too small

EDIT: Nevermind, it was processing, I can see it now.

EDIT2: It actually gives me some other information. It seems that you have no connection to screenscraper.fr at all. The "Retrying request..." only happens when it receives an empty reply. This should not happen. Are you behind some weird firewall or something? Is your wifi / network connection running at all?

EDIT3: I've just tested it, right now, on my own machine. It works just fine. I'm pretty sure this is a local issue to your machine.

AlCzervik · 6 Apr 2020, 19:12

@muldjord I tried a larger size version but still seems super small. Regardless, you are right, that error message did change to include "empty"

AlCzervik · 6 Apr 2020, 19:18

@muldjord Okay, perhaps that is it. Wifi is running fine, that's how I updated to v3.4.4. Must be something with calling that API directly and my Wifi is blocking it.

muldjord · 4 Jun 2020, 20:22

@AlCzervik It really shouldn't be an issue. It's using https (port 443) which is always open. Otherwise you couldn't visit any encrypted webpages, which is basically all webpages. Is it possible for you to put it on wired network to test it?

AlCzervik · 6 Apr 2020, 19:29

@muldjord Unfortunately, it's all built into an arcade cabinet. I'm thinking about how else I can determine why that particular call would be getting blocked.

	# --------------------------------------------------------------------
	# Skyscraper by Lars Muldjord (https://github.com/muldjord/skyscraper)
	# --------------------------------------------------------------------

	# This is an example config file for use with Skyscraper. Use it as a template for creating
	# your own. If you copy this file to 'config.ini' it will be used per default.
	# You can always copy config.ini.example to a filename of your choice and make Skyscraper
	# use it with the '-c' command line option.

	# You can have several platform sections in one config file. Further down is an example of
	# a generic platform setup. Check '--help' for more info on supported platforms.

	# You can have several scraping module sections in one config file. Further down is an example
	# of a generic module setup. Check '--help' for more info on supported modules.

	# Remember that most of these settings can also be set as command line options. Doing so
	# will overrule the options seen here except for the scraping module configs which
	# overrules everything else.
	# Platform specific configs overrules main configs.

	# Uncomment the sections and lines you wish to use:

	[main]
	##Setting input, gamelist or media folder here, will automatically append '/[platform]' to path.
	##If you need better control, use the same variables under a [<platform>] section instead.
	#inputFolder="/home/pi/RetroPie/roms"
	#excludeFiles="[BIOS]"
	#includeFiles="Super*"
	#gamelistFolder="/home/pi/RetroPie/roms"
	#mediaFolder="/home/pi/RetroPie/roms"
	#cacheFolder="/home/pi/.skyscraper/cache"
	#cacheResize="false"
	#jpgQuality="95"
	#cacheCovers="true"
	#cacheScreenshots="true"
	#cacheWheels="true"
	#cacheMarquees="true"
	#importFolder="/home/pi/.skyscraper/import"
	#unpack="false"
	#frontend="emulationstation"
	#emulator=""
	#launch=""
	#videos="false"
	#videoSizeLimit="42"
	#symlink="false"
	#brackets="true"
	#maxLength="10000"
	#threads="2"
	#pretend="false"
	#unattend="false"
	#unattendSkip="false"
	#interactive="false"
	#forceFilename="false"
	#verbosity="1"
	#skipped="false"
	#maxFails="30"
	#lang="en"
	#region="wor"
	#langPrios="en,de,es"
	#regionPrios="eu,us,ss,uk,wor,jp"
	#minMatch="0"
	#artworkXml="artwork.xml"
	#relativePaths="false"
	#addExtensions="*.zst"
	#hints="false"
	#subdirs="true"
	#spaceCheck="true"
	#scummIni="/full/path/to/scummvm.ini"

	#[<PLATFORM, eg 'amiga'>]
	#inputFolder="/home/pi/RetroPie/roms/amiga"
	#excludeFiles="[BIOS]"
	#includeFiles="Super*"
	#gamelistFolder="/home/pi/RetroPie/roms/amiga"
	#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
	#cacheFolder="/home/pi/.skyscraper/cache/amiga"
	#cacheResize="false"
	#jpgQuality="95"
	#cacheCovers="true"
	#cacheScreenshots="true"
	#cacheWheels="true"
	#cacheMarquees="true"
	#importFolder="/home/pi/.skyscraper/import/amiga"
	#unpack="false"
	#emulator=""
	#launch=""
	#videos="false"
	#videoSizeLimit="42"
	#symlink="false"
	#brackets="true"
	#lang="en"
	#region="wor"
	#langPrios="en,de,es"
	#regionPrios="eu,us,ss,uk,wor,jp"
	#minMatch="0"
	#maxLength="10000"
	#threads="2"
	#startAt="filename"
	#endAt="filename"
	#pretend="false"
	#unattend="false"
	#unattendSkip="false"
	#interactive="false"
	#forceFilename="false"
	#verbosity="1"
	#skipped="false"
	#artworkXml="artwork.xml"
	#relativePaths="false"
	#extensions=".zip .uae *.adf"
	#addExtensions=".zst .rom"
	#subdirs="true"

	#[<FRONTEND, eg 'pegasus'>]
	#excludeFiles="[BIOS]"
	#includeFiles="Super*"
	#artworkXml="artwork.xml"
	#emulator=""
	#launch=""
	#gamelistFolder="/home/pi/RetroPie/roms/amiga"
	#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
	#skipped="false"
	#brackets="true"
	#videos="false"
	#symlink="false"
	#startAt="filename"
	#endAt="filename"
	#unattend="false"
	#unattendSkip="false"
	#forceFilename="false"
	#verbosity="1"
	#maxLength="10000"

	[screenscraper]
	userCreds="user:pass"
	#threads="1"
	#minMatch="0"
	#maxLength="10000"
	#interactive="false"
	#cacheResize="false"
	#jpgQuality="95"
	#cacheCovers="true"
	#cacheScreenshots="true"
	#cacheWheels="true"
	#cacheMarquees="true"
	#videos="false"
	#videoSizeLimit="42"