RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.5m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ?
      A Former User @muldjord
      last edited by

      @muldjord Much appreciated!

      1 Reply Last reply Reply Quote 1
      • muldjordM
        muldjord
        last edited by

        Skyscraper 3.4.3 released: https://github.com/muldjord/skyscraper

        • Implemented workaround to incorrectly formatted JSON returned from 'screenscraper' when checking user credentials
        • All arcade platforms now use 'flyer' from 'screenscraper' for cover artwork instead of 'box-2D'

        There have been issues logging in to screenscraper lately. This stems from a bug in their code which formats the returned JSON incorrectly after checking the credentials. This leads to a parse error in the Skyscraper JSON parser which then lead to a failed login. I've implemented a workaround that fixes the JSON before parsing it. I have reported the bug to them, I hope they fix it soon so I can remove this workaround again.

        User @aidy80-s suggested I use the flyer artwork from screenscraperfor the arcade platforms. This was a brilliant idea, so this has now been implemented as well.

        A 1 Reply Last reply Reply Quote 4
        • ?
          A Former User
          last edited by

          Have you ever looked into adding the LaunchBox DB as a scraping source?

          https://gamesdb.launchbox-app.com/

          muldjordM 1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @A Former User
            last edited by

            @LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

            Have you ever looked into adding the LaunchBox DB as a scraping source?

            https://gamesdb.launchbox-app.com/

            Yes, several times actually. I can't remember exactly why, but as I recall it is not an open API like other sources use.

            1 Reply Last reply Reply Quote 0
            • quicksilverQ
              quicksilver
              last edited by quicksilver

              Certain files get scraped as ZZZnotagame instead of just ignored (especially +StartDOSbox.sh) Its really annoying to have to manually rename these after scraping. Is there anyway to avoid this?

              muldjordM 1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord @quicksilver
                last edited by

                @quicksilver Hmm, I already have a filter to avoid these, but maybe some of them goes under the radar. Can you provide a specific filename for an entry that returns this?

                quicksilverQ 1 Reply Last reply Reply Quote 0
                • quicksilverQ
                  quicksilver @muldjord
                  last edited by quicksilver

                  @muldjord Ok cool, if that is the case let me try doing a cache refresh and then rescrape first to see if its fixed now and I'll report back.

                  1 Reply Last reply Reply Quote 0
                  • quicksilverQ
                    quicksilver
                    last edited by quicksilver

                    Refreshing the cache and rescraping took care of the issue. Thanks for the quick response.

                    Edit: Just as a side note to anyone else having the same issue. You'll need to purge your cache for that specific system otherwise when you rescrape it will grab the ZZZnotagame info from your cache and cause the problem all over again.

                    1 Reply Last reply Reply Quote 2
                    • A
                      AlCzervik @muldjord
                      last edited by AlCzervik

                      @muldjord I cannot figure it out. I don't have the credentials commented out, they are properly written, yet I continue to get a "Received invalid ScreenScraper server response, maybe their server is having issues, forcing 1 thread..." and then nothing but empty Json scrapes. Please help! This is with version 3.4.3 BTW.

                      muldjordM 1 Reply Last reply Reply Quote 0
                      • muldjordM
                        muldjord @AlCzervik
                        last edited by

                        @AlCzervik It works fine, just tested it. Maybe their servers were having issues (they are having a lot of issues in general because of so many people scraping at the moment), please try again.

                        A 1 Reply Last reply Reply Quote 0
                        • A
                          AlCzervik @muldjord
                          last edited by

                          @muldjord I'll take another look but are there other things I can check that could cause that issue? Is there a way I can test the API?

                          muldjordM 1 Reply Last reply Reply Quote 0
                          • muldjordM
                            muldjord @AlCzervik
                            last edited by

                            @AlCzervik Hmm, actually that response is also shown if your credentials are wrong. It needs to look exactly like this in the config.ini:

                            [screenscraper]
userCreds="user:pass"

                            But with your own user and pass of course. If you've entered that correctly and don't have any stray configs messing with that, it will work.

                            A 1 Reply Last reply Reply Quote 0
                            • A
                              AlCzervik @muldjord
                              last edited by

                              @muldjord That's helpful! Quotes required? I know use case is important on the variable, is it important with the username/password? Perhaps I should inquire with screenscraper directly to ensure there's nothing wrong with my account? I only recently got to the point where I can download data, perhaps that's the issue.

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @AlCzervik
                                last edited by muldjord

                                @AlCzervik It could be. The service is currently closed for some users. I'm not sure what tiers are closed exactly.

                                I don't think the quotes are necessary but I know it works with them, so I would use them. But if you get that error at the beginning it either means the server is down or you entered the credentials incorrectly. I'm guessing the latter as many people seem to get it wrong.

                                A 1 Reply Last reply Reply Quote 0
                                • A
                                  AlCzervik @muldjord
                                  last edited by

                                  @muldjord Thanks for your help. I'll have to start with them if that response is related to credentials. And I've checked and rechecked that it is spelled correctly in the config.ini file. Not sure what else to try at this point. Thanks again.

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @AlCzervik
                                    last edited by

                                    @AlCzervik If you paste your config.ini as a code-block (BE SURE TO REMOVE YOUR USER + PASS) I can take a look at it.

                                    A 1 Reply Last reply Reply Quote 0
                                    • A
                                      AlCzervik @muldjord
                                      last edited by mitu

                                      @muldjord

                                      # --------------------------------------------------------------------
# Skyscraper by Lars Muldjord (https://github.com/muldjord/skyscraper)
# --------------------------------------------------------------------

# This is an example config file for use with Skyscraper. Use it as a template for creating
#   your own. If you copy this file to 'config.ini' it will be used per default.
#   You can always copy config.ini.example to a filename of your choice and make Skyscraper
#   use it with the '-c' command line option.

# You can have several platform sections in one config file. Further down is an example of
#   a generic platform setup. Check '--help' for more info on supported platforms.

# You can have several scraping module sections in one config file. Further down is an example
#   of a generic module setup. Check '--help' for more info on supported modules.

# Remember that most of these settings can also be set as command line options. Doing so
#   will overrule the options seen here except for the scraping module configs which
#   overrules everything else.
# Platform specific configs overrules main configs.

# Uncomment the sections and lines you wish to use:

[main]
##Setting input, gamelist or media folder here, will automatically append '/[platform]' to path.
##If you need better control, use the same variables under a [<platform>] section instead.
#inputFolder="/home/pi/RetroPie/roms"
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#gamelistFolder="/home/pi/RetroPie/roms"
#mediaFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import"
#unpack="false"
#frontend="emulationstation"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#maxLength="10000"
#threads="2"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#maxFails="30"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#artworkXml="artwork.xml"
#relativePaths="false"
#addExtensions="*.zst"
#hints="false"
#subdirs="true"
#spaceCheck="true"
#scummIni="/full/path/to/scummvm.ini"

#[<PLATFORM, eg 'amiga'>]
#inputFolder="/home/pi/RetroPie/roms/amiga"
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#cacheFolder="/home/pi/.skyscraper/cache/amiga"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import/amiga"
#unpack="false"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#maxLength="10000"
#threads="2"
#startAt="filename"
#endAt="filename"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#artworkXml="artwork.xml"
#relativePaths="false"
#extensions="*.zip *.uae *.adf"
#addExtensions="*.zst *.rom"
#subdirs="true"

#[<FRONTEND, eg 'pegasus'>]
#excludeFiles="*[BIOS]*"
#includeFiles="Super*"
#artworkXml="artwork.xml"
#emulator=""
#launch=""
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#skipped="false"
#brackets="true"
#videos="false"
#symlink="false"
#startAt="filename"
#endAt="filename"
#unattend="false"
#unattendSkip="false"
#forceFilename="false"
#verbosity="1"
#maxLength="10000"

[screenscraper]
userCreds="user:pass"
#threads="1"
#minMatch="0"
#maxLength="10000"
#interactive="false"
#cacheResize="false"
#jpgQuality="95"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#videos="false"
#videoSizeLimit="42"
                                      muldjordM 1 Reply Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @AlCzervik
                                        last edited by

                                        @AlCzervik That config.ini is correct.

                                        A 1 Reply Last reply Reply Quote 0
                                        • A
                                          AlCzervik @muldjord
                                          last edited by

                                          @muldjord Thank you! But yeah, it's driving me crazy why it isn't working!!

                                          muldjordM 1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord @AlCzervik
                                            last edited by muldjord

                                            @AlCzervik I assume you have gone to their website and logged on to check that your profile is active, right?

                                            And just to give an example, this is how the entry would look if your username was mike and your pass was supersecret (not real, just came up with it):

                                            [screenscraper]
userCreds="mike:supersecret"
                                            A 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.