RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login
    Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

    Scraping issued with TGDB and SS

    Scheduled Pinned Locked Moved
    Help and Support
    scraper problem ssl selphs scraper network
    2
    8
    1.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H
      HipHopBlond
      last edited by HipHopBlond

      The Problem:

      I can scrape from arcade italia, MameDB and OpenVGDB, it works perfectly. I can't scrape from TGDB and SS.

      Specs:

      • Hardware: Raspberry Pi 3B+
      • Power Supply: Official
      • Build: RetroPie 4.4.12 (latest update)
      • Build From: retropie.org.uk
      • USB Devices connected: 2x Xbox One S Controllers, 2x NES Controllers, 1 wireless pocket keyboard
      • I'm using the Nespi+ Case (fan included)
      • Roms are on a Synology NAS DS216SE
      • Mount command in autostart script for Network Share from my NAS:
      sudo mount -t cifs -o vers=3.0,username=*************,password=**********,file_mode=0777,dir_mode=0777,rw,nounix,noserverino //192.168.1.103/home/RaspberryPi/RetroPie /home/pi/RetroPie

      Errors when I try to scrape with Integrated Retropie Scraper:

      lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK

      Errors when I try to scrape with Stephen's Selph Scraper:

      2019/05/13 17:33:19 arcade  srcs:
2019/05/13 17:33:19 console srcs: ss
2019/05/13 17:33:19 INFO: Checking for new hash.csv.
2019/05/13 17:33:20 INFO: hash.csv "d9eedf6e622a8ab6a1a14486bb90ffb8" up to date.
2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/007 - The World Is Not Enough.gbc
2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatiens a la Rescousse, Les.gbc
2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/10-Pin Bowling.gbc
2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatians - Puppies to the Rescue.gbc
2019/05/13 17:33:28 ERR: error processing /home/pi/RetroPie/roms/gbc/10-Pin Bowling.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
2019/05/13 17:33:28 INFO: Starting: /home/pi/RetroPie/roms/gbc/10-Pin Bowling.gbc
2019/05/13 17:33:33 ERR: error processing /home/pi/RetroPie/roms/gbc/102 Dalmatiens a la Rescousse, Les.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
2019/05/13 17:33:33 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatiens a la Rescousse, Les.gbc
2019/05/13 17:33:39 ERR: error processing /home/pi/RetroPie/roms/gbc/007 - The World Is Not Enough.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
2019/05/13 17:33:39 INFO: Starting: /home/pi/RetroPie/roms/gbc/007 - The World Is Not Enough.gbc
2019/05/13 17:33:45 ERR: error processing /home/pi/RetroPie/roms/gbc/102 Dalmatians - Puppies to the Rescue.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
2019/05/13 17:33:45 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatians - Puppies to the Rescue.gbc

      Ping to thegamesdb.net from rpi:

      ping thegamesdb.net
PING thegamesdb.net (69.30.45.135) 56(84) bytes of data.
64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=1 ttl=51 time=187 ms
64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=2 ttl=51 time=187 ms
64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=3 ttl=51 time=187 ms
64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=4 ttl=51 time=187 ms

      Errors in dmesg logs:

      brcmf_fw_map_chip_to_name: using brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
brcmf_c_preinit_dcmds: Firmware version = wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
brcmf_c_preinit_dcmds: CLM version = API: 12.2 Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09 18:56:28

      What I tried:

      • Searched on Google.

      • Checked the Forum.

      • Read the Documentation.

      • Updating Retropie and Underlying OS Packages.

      • Updating EmulationStation alone.

      • Rebooting of Rpi, NAS, Router and ONT.

      • Deleting all games list XML files and images that I had previously downloaded.

      • Deleted the json files in emulationstation/scrapers folder.

      • Reinstalled Steven Selph's Scraper.

      • Removed Network Protection from my NAS.

      • Removed CloudFlare (1.1.1.1) as my DNS provider.

      • Tried restoring to a 2 weeks old backup - it works to a certain point and then it stops working again.

      Nothing seemed to work, neither for the integrated nor Stephen Selph's scraper.

      I have no idea how to replicate the problem, as I don't know what actually caused it.

      Any idea why this might have happened, and what could have caused it? I'm starting to pull my hair out... please advise.

      1 Reply Last reply Reply Quote 0
      • mituM
        mitu Global Moderator
        last edited by

        The built in scraper has an issue with ScreenScraper (which should be solved soon), but it should work with TGDB. For Steven Selph's scraper, there's numerous reports of people not getting to work and it doesn't seem maintained at this time, but maybe someone has an idea what changed on the scraping site (ScreenScraper is sometimes busy and returns an error when it's over capacity).

        You options are either use TGDB with the internal Emulationstation scraper or use skyscraper as external scraper - accessible from the RetroPie setup as scraper - more details in the Docs.

        H 1 Reply Last reply Reply Quote 1
        • H
          HipHopBlond @mitu
          last edited by HipHopBlond

          @mitu Thanks for the reply! I would be fine with the built in scraper, but I'm getting the above mentioned ssl network errors for some reason when I'm using the integrated scraper with TGDB or Screen Scraper (doesn't matter which one). That's what's buggin me the most... it's showing the SSL errors in the logs, and it's just giving me unknown on each rom...

          Funny thing is I did set up another rpi 3B+ for a friend of mine and it worked like a charm there...

          Any advise?

          mituM 1 Reply Last reply Reply Quote 0
          • mituM
            mitu Global Moderator @HipHopBlond
            last edited by

            @HipHopBlond Are you sure you're getting the SSL errors with TGDB selected as scraping source ? Maybe you didn't switch the scraper in the settings.
            The SSL error in the log - which you posted - is from the ScreenScraper scraping module, not TGDB.

            H 1 Reply Last reply Reply Quote 1
            • H
              HipHopBlond @mitu
              last edited by HipHopBlond

              @mitu

              Thanks for the reply!

              My bad, it only happens with ScreenScraper.

              This is what happens with TGDB:

              https://photos.app.gooDOTgl/Uvfgd2vz8LaJkPc38

              I made a video for you, but it's getting flagged as spam, it's a link to the video uploaded in google photos.

              I can up it on youtube as well if you'd like.

              mituM 1 Reply Last reply Reply Quote 0
              • mituM
                mitu Global Moderator @HipHopBlond
                last edited by mitu

                @HipHopBlond TGDB has a monthly limit on scrapes, it's 1500 (? or around that) - you probably went over the limit and you're getting no results because of that.

                H 2 Replies Last reply Reply Quote 1
                • H
                  HipHopBlond @mitu
                  last edited by

                  @mitu

                  I'm gonna kill myself... you might be right... I'll look for any way I can register or pay some kind of a subscription fee as I have around 14000 roms. I'll let you know how it goes.

                  Tell you the truth it's probably the same with ScreenScraper... sorry for not including this ...small... detail.

                  1 Reply Last reply Reply Quote 0
                  • H
                    HipHopBlond @mitu
                    last edited by HipHopBlond

                    @mitu

                    Man I can't thank you enough, it's finally working! If you PM me your details, you have a few beers from me :)

                    For anyone interested, after registering at screenscraper.fr and going through all SS flags arguments, I think I finally got it right:

                    /opt/retropie/supplementary/scraper/scraper -scrape_all=true -download_images=true -download_marquees=true -download_videos=true -console_src="ss,gdb,ovgdb" -mame_src="adb,mamedb,gdb" -use_nointro_name=true -ss_user="user_name" -ss_password="password"

                    No more errors:

                    88a40a24-8e53-4c14-906c-0a9f06e443cd-image.png

                    1 Reply Last reply Reply Quote 1
                    • First post
                      Last post

                    Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                    Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.