• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

Scraping totally fails

Scheduled Pinned Locked Moved Help and Support
skyscraperscraperthegamesdbscreenscraper
17 Posts 9 Posters 2.8k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • B
    beppi
    last edited by beppi 10 May 2022, 00:30 4 Oct 2022, 23:24

    Good day.
    I decided to scrape some new rom, I have a Raspberry Pi 3B+ with retropie / emulationstation installed.
    Everything has always been fine and I have a lot of roms that have been correctly scraped in the past. I work with Skyscraper, that I now updated to the latest 3.7.7 version.
    Now, when I tried to scrape again, after some months of not doing it, I continuously got the same result: game not found.
    The request succeeds but no data is retrieved.

    I tried everything but no way...:

    I tried to scrape different systems, both rare and common, no game found.
    I tried to use both Skyscraper and the default scraper from inside emulationstation... no game found.
    I tried both tgdb and screenscraper... no game found.
    I tried to force scraping of a rom that I know exists on both websites and that I successfully scraped in the past, and I even tried to change its name to match letter-by-letter the correct name on the site, so there's no real reason for the message... no game found.

    ALL these commands give me the same frustrating error: game not found. I could scrape not even one single game, on any platform and from any site.

    I'm pretty sure that there are no errors on this side, I use scripts that have always worked... now I cannot do it anymore, is there something I've been missing in these months? Did those services shutdown? Did some API change so that the request fails in some way?

    I know that the problem seems stupid, but I'm really stuck, I've tried anything, thank you if you would like to give me some help.

    Addon: network works fine and I used free registered users on both tgdb and screenscraper

    M 1 Reply Last reply 5 Oct 2022, 06:52 Reply Quote 0
    • M
      max @beppi
      last edited by 5 Oct 2022, 06:52

      Not sure if it's the same / related: couple of days ago I scraped my games (I just set up everything from scratch, only had a couple of games) and it would only find a few ones out of 20; e.g. none of the ScummVM games were found.

      Fast forward, yesterday I re-ran it and found almost all of them.

      B 2 Replies Last reply 5 Oct 2022, 08:38 Reply Quote 0
      • B
        beppi @max
        last edited by beppi 10 May 2022, 09:39 5 Oct 2022, 08:38

        @max
        Thank you Max, I hope that this is what you said, a temporary problem.
        It's really strange that every single rom searched on every single site fails, even if they are present when searched manually on the web site.
        A question: as you know does exist an alternative way, instead of scraping some site, to get those mp4s, jpgs and infos? some torrent maybe, to fill up the cache without pulling the data from those central sites?

        W 1 Reply Last reply 5 Oct 2022, 11:15 Reply Quote 0
        • W
          WhiteT982 @beppi
          last edited by 5 Oct 2022, 11:15

          @beppi
          If you want to do it manually I’ve used emumovies. There’s also ARRM which is another scraper you could try. Not sure if it uses sites other than tgb or screenscraper though.

          N 1 Reply Last reply 6 Oct 2022, 15:04 Reply Quote 0
          • N
            nexusone13 @WhiteT982
            last edited by nexusone13 10 Jun 2022, 16:05 6 Oct 2022, 15:04

            Hi @WhiteT982 ,
            ARRM can scrape on this sites : Screenscraper, HFS-DB, TheGamesDb.net, LaunchBox, Steam, IGDB , GOG , MobyGames and GameTDB

            ARRM (Roms Manager, Scraper for Recalbox, Batocera, Retropie, ES...) : http://jujuvincebros.fr/telechargements2/file/10-arrm-another-recalbox-roms-manager
            Wiki : http://jujuvincebros.fr/wiki/arrm/doku.php?id=start-en
            Discord: https://discord.gg/p7QsBTS

            1 Reply Last reply Reply Quote 0
            • B
              beppi @max
              last edited by beppi 10 Jun 2022, 19:42 6 Oct 2022, 18:40

              Thanks all folks, I'm really getting crazy.
              I tried to scrape using skraper, and it works. Only problem... it runs on Windows and I need it running on the raspberry. The only ption I see now is to scrape on Windows, create a script to change manually all the paths and informations that are missing and move everything manually, a huge work!
              Furthermore, also other scrapers fail and I'm really styuck in a problem that looks to me as a sci-fi movie... I made a lot of configurations on my raspi and I consider myself able to dive in deep in linux configurations using my beloved vi, but I fail here.
              Anything else I tried, unfortunately, failed, and now I'm trying to reach the author of Skyscraper on github, but I doubt I will be able to fix it.

              Q F 2 Replies Last reply 6 Oct 2022, 22:06 Reply Quote 0
              • Q
                Quackwalks @beppi
                last edited by 6 Oct 2022, 22:06

                @beppi What I do Is back up my roms directory into a windows folder. In Skraper, you can change the settings to use the windows directory and it'll still output game lists that work for retropie. I just put my game lists in my rom directories, then a folder in each system for the snaps and box arts. After you're done scraping, transfer your game lists and scraped media to your raspberry pi where they should go.

                Sorry I can't help with the details, but I know this method works with enough tinkering.

                B C 2 Replies Last reply 7 Oct 2022, 16:38 Reply Quote 1
                • B
                  beppi @Quackwalks
                  last edited by beppi 10 Jul 2022, 17:39 7 Oct 2022, 16:38

                  @Quackwalks this is indeed the only solution i found so far, but it's not a complete solution because some information is missing, for example the information about the location of videos and screenshots, and Skyscraper was also creating a composition with the box for the screenshot that was integrated with the theme. Of course it's not a big deal but I would like to understand why Skyscraper cannot connect from raspi.
                  I feel that I could fix this problem if I only had some kind of log, monitor, or anything else, do you know if there is a log somewhere? I could not find it.

                  Y 1 Reply Last reply 8 Oct 2022, 15:36 Reply Quote 0
                  • Y
                    YFZdude @beppi
                    last edited by YFZdude 10 Sept 2022, 00:28 8 Oct 2022, 15:36

                    @beppi
                    There should be logs anytime something fails to scrape with skyscraper in the /home/pi/.skyscraper folder.

                    B 1 Reply Last reply 9 Oct 2022, 12:33 Reply Quote 0
                    • C
                      Crush @Quackwalks
                      last edited by Crush 10 Aug 2022, 19:25 8 Oct 2022, 18:25

                      @Quackwalks said in Scraping totally fails:

                      @beppi What I do Is back up my roms directory into a windows folder. In Skraper, you can change the settings to use the windows directory and it'll still output game lists that work for retropie. I just put my game lists in my rom directories, then a folder in each system for the snaps and box arts. After you're done scraping, transfer your game lists and scraped media to your raspberry pi where they should go.

                      Sorry I can't help with the details, but I know this method works with enough tinkering.

                      Just make a network drive on windows out of \\retropie\roms so you can scrape using screenscraper on windows directly instead of having to transfer back and forth.

                      Q 1 Reply Last reply 13 Oct 2022, 02:09 Reply Quote 0
                      • B
                        beppi @YFZdude
                        last edited by beppi 10 Sept 2022, 13:36 9 Oct 2022, 12:33

                        @YFZdude thank you, unfortunately there's nothing, because there's no error in Skyscraper: it just receives the information "not found", appearantly. I watched by capturing in real time written files both on .skyscraper directory, and on /var/log, and no log is being generated.

                        B 1 Reply Last reply 9 Oct 2022, 12:37 Reply Quote 0
                        • B
                          beppi @beppi
                          last edited by 9 Oct 2022, 12:37

                          Thanks all for answering me, unfortunately there's no solution right now.
                          I make you just one question: are you able to scrape with Skyscraper from inside raspberry, on screenscraper.fr or do you get the same error "game not found"? thank you

                          K 1 Reply Last reply 9 Oct 2022, 14:35 Reply Quote 0
                          • K
                            kiro @beppi
                            last edited by 9 Oct 2022, 14:35

                            @beppi trybtjis, it may work...https://github.com/zayamatias/retroscraper

                            1 Reply Last reply Reply Quote 0
                            • F
                              Floob @beppi
                              last edited by 10 Oct 2022, 16:52

                              @beppi
                              FWIW I use Skraper and scrape on windows, then just copy across the files.
                              I do that so I can copy it across to any destination quickly - so you only need to scrape once really.
                              The paths are all relative, so you dont need to change them

                              For example:
                              <image>./media/boxart2d/3-D Tic-Tac-Toe (USA).png</image> <marquee>./media/wheel/3-D Tic-Tac-Toe (USA).png</marquee> <video>./media/videos/3-D Tic-Tac-Toe (USA).mp4</video>

                              So in Skraper, the video path would be:
                              %ROMROOTFOLDER%\media\videos

                              Same as @Quackwalks really.

                              Please read the Docs before asking a new question.
                              RetroPie Help Guides: https://goo.gl/3gcNsT

                              B 1 Reply Last reply 12 Oct 2022, 12:48 Reply Quote 1
                              • B
                                beppi @Floob
                                last edited by 12 Oct 2022, 12:48

                                @Floob i fixed the problem. On retropie it still fails, but I mounted a disk in a Linux distribution and using the bind option I recreated the needed directory tree and run Skyscraper from there. It works, so it's obvious that there is a problem on my raspberry but I don't have any tool to address it. Thank you

                                K 1 Reply Last reply 12 Oct 2022, 12:52 Reply Quote 0
                                • K
                                  kiro @beppi
                                  last edited by 12 Oct 2022, 12:52

                                  @beppi could it be a permissions issue in your rpi? Pi user does not have write rights on the mount ?

                                  1 Reply Last reply Reply Quote 0
                                  • Q
                                    Quackwalks @Crush
                                    last edited by 13 Oct 2022, 02:09

                                    @Crush That is also an option, but sometimes I'd rather get everything scraped on a separate hard drive. Just makes me feel more secure in various ways. I'm probably just silly.

                                    1 Reply Last reply Reply Quote 0
                                    17 out of 17
                                    • First post
                                      17/17
                                      Last post

                                    Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                    Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                      This community forum collects and processes your personal information.
                                      consent.not_received