Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

Scraping issued with TGDB and SS



  • The Problem:

    I can scrape from arcade italia, MameDB and OpenVGDB, it works perfectly. I can't scrape from TGDB and SS.

    Specs:

    • Hardware: Raspberry Pi 3B+
    • Power Supply: Official
    • Build: RetroPie 4.4.12 (latest update)
    • Build From: retropie.org.uk
    • USB Devices connected: 2x Xbox One S Controllers, 2x NES Controllers, 1 wireless pocket keyboard
    • I'm using the Nespi+ Case (fan included)
    • Roms are on a Synology NAS DS216SE
    • Mount command in autostart script for Network Share from my NAS:
    sudo mount -t cifs -o vers=3.0,username=*************,password=**********,file_mode=0777,dir_mode=0777,rw,nounix,noserverino //192.168.1.103/home/RaspberryPi/RetroPie /home/pi/RetroPie
    

    Errors when I try to scrape with Integrated Retropie Scraper:

    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    lvl0: ScraperHttpRequest network error (status: 2) - SSL peer certificate or SSH remote key was not OK
    

    Errors when I try to scrape with Stephen's Selph Scraper:

    2019/05/13 17:33:19 arcade  srcs:
    2019/05/13 17:33:19 console srcs: ss
    2019/05/13 17:33:19 INFO: Checking for new hash.csv.
    2019/05/13 17:33:20 INFO: hash.csv "d9eedf6e622a8ab6a1a14486bb90ffb8" up to date.
    2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/007 - The World Is Not Enough.gbc
    2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatiens a la Rescousse, Les.gbc
    2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/10-Pin Bowling.gbc
    2019/05/13 17:33:21 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatians - Puppies to the Rescue.gbc
    2019/05/13 17:33:28 ERR: error processing /home/pi/RetroPie/roms/gbc/10-Pin Bowling.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
    2019/05/13 17:33:28 INFO: Starting: /home/pi/RetroPie/roms/gbc/10-Pin Bowling.gbc
    2019/05/13 17:33:33 ERR: error processing /home/pi/RetroPie/roms/gbc/102 Dalmatiens a la Rescousse, Les.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
    2019/05/13 17:33:33 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatiens a la Rescousse, Les.gbc
    2019/05/13 17:33:39 ERR: error processing /home/pi/RetroPie/roms/gbc/007 - The World Is Not Enough.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
    2019/05/13 17:33:39 INFO: Starting: /home/pi/RetroPie/roms/gbc/007 - The World Is Not Enough.gbc
    2019/05/13 17:33:45 ERR: error processing /home/pi/RetroPie/roms/gbc/102 Dalmatians - Puppies to the Rescue.gbc: ss: cannot parse response: "invalid character 'V' looking for beginning of value"
    2019/05/13 17:33:45 INFO: Starting: /home/pi/RetroPie/roms/gbc/102 Dalmatians - Puppies to the Rescue.gbc
    

    Ping to thegamesdb.net from rpi:

    ping thegamesdb.net
    PING thegamesdb.net (69.30.45.135) 56(84) bytes of data.
    64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=1 ttl=51 time=187 ms
    64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=2 ttl=51 time=187 ms
    64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=3 ttl=51 time=187 ms
    64 bytes from 69.30.45.135 (69.30.45.135): icmp_seq=4 ttl=51 time=187 ms
    

    Errors in dmesg logs:

    brcmf_fw_map_chip_to_name: using brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
    brcmf_c_preinit_dcmds: Firmware version = wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
    brcmf_c_preinit_dcmds: CLM version = API: 12.2 Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09 18:56:28
    

    What I tried:

    • Searched on Google.

    • Checked the Forum.

    • Read the Documentation.

    • Updating Retropie and Underlying OS Packages.

    • Updating EmulationStation alone.

    • Rebooting of Rpi, NAS, Router and ONT.

    • Deleting all games list XML files and images that I had previously downloaded.

    • Deleted the json files in emulationstation/scrapers folder.

    • Reinstalled Steven Selph's Scraper.

    • Removed Network Protection from my NAS.

    • Removed CloudFlare (1.1.1.1) as my DNS provider.

    • Tried restoring to a 2 weeks old backup - it works to a certain point and then it stops working again.

    Nothing seemed to work, neither for the integrated nor Stephen Selph's scraper.

    I have no idea how to replicate the problem, as I don't know what actually caused it.

    Any idea why this might have happened, and what could have caused it? I'm starting to pull my hair out... please advise.


  • Global Moderator

    The built in scraper has an issue with ScreenScraper (which should be solved soon), but it should work with TGDB. For Steven Selph's scraper, there's numerous reports of people not getting to work and it doesn't seem maintained at this time, but maybe someone has an idea what changed on the scraping site (ScreenScraper is sometimes busy and returns an error when it's over capacity).

    You options are either use TGDB with the internal Emulationstation scraper or use skyscraper as external scraper - accessible from the RetroPie setup as scraper - more details in the Docs.



  • @mitu Thanks for the reply! I would be fine with the built in scraper, but I'm getting the above mentioned ssl network errors for some reason when I'm using the integrated scraper with TGDB or Screen Scraper (doesn't matter which one). That's what's buggin me the most... it's showing the SSL errors in the logs, and it's just giving me unknown on each rom...

    Funny thing is I did set up another rpi 3B+ for a friend of mine and it worked like a charm there...

    Any advise?


  • Global Moderator

    @HipHopBlond Are you sure you're getting the SSL errors with TGDB selected as scraping source ? Maybe you didn't switch the scraper in the settings.
    The SSL error in the log - which you posted - is from the ScreenScraper scraping module, not TGDB.



  • @mitu

    Thanks for the reply!

    My bad, it only happens with ScreenScraper.

    This is what happens with TGDB:

    https://photos.app.gooDOTgl/Uvfgd2vz8LaJkPc38

    I made a video for you, but it's getting flagged as spam, it's a link to the video uploaded in google photos.

    I can up it on youtube as well if you'd like.


  • Global Moderator

    @HipHopBlond TGDB has a monthly limit on scrapes, it's 1500 (? or around that) - you probably went over the limit and you're getting no results because of that.



  • @mitu

    I'm gonna kill myself... you might be right... I'll look for any way I can register or pay some kind of a subscription fee as I have around 14000 roms. I'll let you know how it goes.

    Tell you the truth it's probably the same with ScreenScraper... sorry for not including this ...small... detail.



  • @mitu

    Man I can't thank you enough, it's finally working! If you PM me your details, you have a few beers from me :)

    For anyone interested, after registering at screenscraper.fr and going through all SS flags arguments, I think I finally got it right:

    /opt/retropie/supplementary/scraper/scraper -scrape_all=true -download_images=true -download_marquees=true -download_videos=true -console_src="ss,gdb,ovgdb" -mame_src="adb,mamedb,gdb" -use_nointro_name=true -ss_user="user_name" -ss_password="password"
    

    No more errors:

    88a40a24-8e53-4c14-906c-0a9f06e443cd-image.png



Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.