RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved
    Ideas and Development
    skyscraper scraper gamelist.xml scraping github
    113
    1.6k
    1.1m
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      AlCzervik @muldjord
      last edited by mitu

      @muldjord

      # --------------------------------------------------------------------
# Skyscraper by Lars Muldjord (https://github.com/muldjord/skyscraper)
# --------------------------------------------------------------------

# This is an example config file for use with Skyscraper. Use it as a template for creating
#   your own. If you copy this file to 'config.ini' it will be used per default.
#   You can always copy config.ini.example to a filename of your choice and make Skyscraper
#   use it with the '-c' command line option.

# You can have several platform sections in one config file. Further down is an example of
#   a generic platform setup. Check '--help' for more info on supported platforms.

# You can have several scraping module sections in one config file. Further down is an example
#   of a generic module setup. Check '--help' for more info on supported modules.

# Remember that most of these settings can also be set as command line options. Doing so
#   will overrule the options seen here except for the scraping module configs which
#   overrules everything else.
# Platform specific configs overrules main configs.

# Uncomment the sections and lines you wish to use:

[main]
##Setting input, gamelist or media folder here, will automatically append '/[platform]' to path.
##If you need better control, use the same variables under a [<platform>] section instead.
#inputFolder="/home/pi/RetroPie/roms"
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#gamelistFolder="/home/pi/RetroPie/roms"
#mediaFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import"
#unpack="false"
#frontend="emulationstation"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#maxLength="10000"
#threads="2"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#maxFails="30"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#artworkXml="artwork.xml"
#relativePaths="false"
#addExtensions="*.zst"
#hints="false"
#subdirs="true"
#spaceCheck="true"
#scummIni="/full/path/to/scummvm.ini"

#[<PLATFORM, eg 'amiga'>]
#inputFolder="/home/pi/RetroPie/roms/amiga"
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#cacheFolder="/home/pi/.skyscraper/cache/amiga"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import/amiga"
#unpack="false"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#maxLength="10000"
#threads="2"
#startAt="filename"
#endAt="filename"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#artworkXml="artwork.xml"
#relativePaths="false"
#extensions="*.zip *.uae *.adf"
#addExtensions="*.zst *.rom"
#subdirs="true"

#[<FRONTEND, eg 'pegasus'>]
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#artworkXml="artwork.xml"
#emulator=""
#launch=""
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#skipped="false"
#brackets="true"
#videos="false"
#symlink="false"
#startAt="filename"
#endAt="filename"
#unattend="false"
#unattendSkip="false"
#forceFilename="false"
#verbosity="1"
#maxLength="10000"

[screenscraper]
userCreds="user:pass"
#threads="1"
#minMatch="0"
#maxLength="10000"
#interactive="false"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#videos="false"
#videoSizeLimit="42"
      muldjordM 1 Reply Last reply Reply Quote 0
      • muldjordM
        muldjord @AlCzervik
        last edited by

        @AlCzervik That config.ini is correct.

        A 1 Reply Last reply Reply Quote 0
        • A
          AlCzervik @muldjord
          last edited by

          @muldjord Thank you! But yeah, it's driving me crazy why it isn't working!!

          muldjordM 1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @AlCzervik
            last edited by muldjord

            @AlCzervik I assume you have gone to their website and logged on to check that your profile is active, right?

            And just to give an example, this is how the entry would look if your username was mike and your pass was supersecret (not real, just came up with it):

            [screenscraper]
userCreds="mike:supersecret"
            A 2 Replies Last reply Reply Quote 0
            • A
              AlCzervik @muldjord
              last edited by

              @muldjord Thanks again. I've tried several things and nothing seems to work. So, to be clear, that error definitely means I'm not getting authenticated to ScreenScraper.fr correct? I know your program has worked with another source so I assume it is that connection to ScreenScraper. I'll try posting again in their forum for help. Any other ideas would be appreciated!

              1 Reply Last reply Reply Quote 0
              • A
                AlCzervik @muldjord
                last edited by

                @muldjord A few other notes, it's correctly stating "Fetching limits for user 'User', so I know it's grabbing it properly there. Then I get the "Received invalid ScreenScraper server response..." message.

                muldjordM 1 Reply Last reply Reply Quote 0
                • muldjordM
                  muldjord @AlCzervik
                  last edited by

                  @AlCzervik Can you try and update Skyscraper and try again? I just updated it to 3.4.4 today, and it will now tell you if the credentials are rejected.

                  A 1 Reply Last reply Reply Quote 0
                  • A
                    AlCzervik @muldjord
                    last edited by

                    @muldjord Updated to 3.4.4. Similar error. First I get the "Fetching limits for user 'User', just a sec..." After that, I now get "Received invalid / empty ScreenScraper server response..."

                    muldjordM 1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord @AlCzervik
                      last edited by muldjord

                      @AlCzervik Ok, that is really, really odd. Then it's not a login error. Can you please verify that it says Skyscraper 3.4.4 when you run it?

                      EDIT: Ah, yes, it is. Because the error message is updated with "empty". In that case, I have no idea what is going on. I can't reproduce that error here. Can you try and change your password to something else and retry it with that? I'm thinking that maybe it is an encoding error.

                      A 1 Reply Last reply Reply Quote 0
                      • A
                        AlCzervik @muldjord
                        last edited by AlCzervik

                        @muldjord I'll try the password change. Here's the screenshot: IMG_0252 (2).jpeg

                        muldjordM 1 Reply Last reply Reply Quote 0
                        • muldjordM
                          muldjord @AlCzervik
                          last edited by muldjord

                          @AlCzervik I can't see that, it is too small

                          EDIT: Nevermind, it was processing, I can see it now.

                          EDIT2: It actually gives me some other information. It seems that you have no connection to screenscraper.fr at all. The "Retrying request..." only happens when it receives an empty reply. This should not happen. Are you behind some weird firewall or something? Is your wifi / network connection running at all?

                          EDIT3: I've just tested it, right now, on my own machine. It works just fine. I'm pretty sure this is a local issue to your machine.

                          A 2 Replies Last reply Reply Quote 0
                          • A
                            AlCzervik @muldjord
                            last edited by

                            @muldjord I tried a larger size version but still seems super small. Regardless, you are right, that error message did change to include "empty"

                            1 Reply Last reply Reply Quote 0
                            • A
                              AlCzervik @muldjord
                              last edited by

                              @muldjord Okay, perhaps that is it. Wifi is running fine, that's how I updated to v3.4.4. Must be something with calling that API directly and my Wifi is blocking it.

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @AlCzervik
                                last edited by muldjord

                                @AlCzervik It really shouldn't be an issue. It's using https (port 443) which is always open. Otherwise you couldn't visit any encrypted webpages, which is basically all webpages. Is it possible for you to put it on wired network to test it?

                                A 1 Reply Last reply Reply Quote 0
                                • A
                                  AlCzervik @muldjord
                                  last edited by

                                  @muldjord Unfortunately, it's all built into an arcade cabinet. I'm thinking about how else I can determine why that particular call would be getting blocked.

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @AlCzervik
                                    last edited by muldjord

                                    @AlCzervik If you at all have the chance, I would be curious to know if you could run this on any other machine. But let me know if you have further questions. Unfortunately I don't really know what else I can do at this point.

                                    EDIT: Quick idea, do you have a DNS server registered for the connection? If not, then it won't work since it won't be able to resolve screenscraper.fr. Can you do ping www.google.com and get a response?

                                    A 1 Reply Last reply Reply Quote 0
                                    • A
                                      AlCzervik @muldjord
                                      last edited by

                                      @muldjord pinging google worked just fine. Tell me more about the DNS server issue?

                                      muldjordM 2 Replies Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @AlCzervik
                                        last edited by muldjord

                                        @AlCzervik No need. If you can ping www.google.com it is not the issue. Try doing ping www.screenscraper.frjust for the heck of it. But it's not pinging on port 443 so it doesn't really tell us if that's the issue.

                                        A 1 Reply Last reply Reply Quote 0
                                        • A
                                          AlCzervik @muldjord
                                          last edited by

                                          @muldjord pinging screenscraper worked too

                                          1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord @AlCzervik
                                            last edited by muldjord

                                            @AlCzervik What happens if you do telnet www.screenscraper.fr 443 and then type test and press enter? That will connect to the server on port 443 and send testto it, which should result in a 400 Bad Request reply.

                                            A 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.