• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

[SOFT] New Scraper in the works

Scheduled Pinned Locked Moved Projects and Themes
scrapingscrappersoftware
253 Posts 7 Posters 59.8k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • F
    Folly @kiro
    last edited by 9 Nov 2022, 19:31

    @kiro

    If I can implement the setup somehow in my script then your retroscraper-rpie would probably be easy enough to add in my script.
    I will have to think about it some time.

    K 1 Reply Last reply 9 Nov 2022, 19:37 Reply Quote 0
    • K
      kiro @Folly
      last edited by 9 Nov 2022, 19:37

      @Folly thanks! I actually do have a script which is running (installation goes fine as well as execution) based on scraper.sh script. My only concern right now is that output is buffered, so it seems (only seems) as if the scraper freezes.
      But thanks for the offer, definitely we can talk about doing something together.

      F 2 Replies Last reply 9 Nov 2022, 19:59 Reply Quote 0
      • F
        Folly @kiro
        last edited by Folly 11 Sept 2022, 20:05 9 Nov 2022, 19:59

        @kiro

        I just did atari7800, although it also has the .png video issue, all videos and pictures are good.
        I must say if something is recognised then recognition seems to be perfect.
        I do not see that with other scraper methods.
        I also notice that the scraping is very fast, much faster that other scrape methods.
        Really nice !!!

        1 Reply Last reply Reply Quote 0
        • F
          Folly @kiro
          last edited by Folly 11 Nov 2022, 09:08 11 Nov 2022, 08:50

          @kiro

          For your information.
          I am testing your retroscraper-rpie now in my Debian11-x86_64 in VMWARE.
          Looks like the .png problem for videos doesn't occur there.
          Also the gamelist.xml contain the correct naming for the videos (.mp4)
          So this problem could be specific for the RPI/raspbian/raspbian version.

          ALL video seem to be playing correctly in emulationstation in my VM, although, for some videos, emulationstation seems to give errors like this :

          [src/libmpg123/layer3.c:2184] error: part2_3_length (1408) too large for available bit count (1368)
          

          Could it be that some videos are not that compatible or have some corruption ?

          K 1 Reply Last reply 11 Nov 2022, 16:33 Reply Quote 0
          • K
            kiro @Folly
            last edited by 11 Nov 2022, 16:33

            @Folly hi, videos are being fixed as we speak, it will need some time. PNG issue should be independent from platform (after all is running the same code) let me know if you encounter this problem again.

            F 1 Reply Last reply 11 Nov 2022, 20:33 Reply Quote 1
            • F
              Folly @kiro
              last edited by Folly 11 Nov 2022, 21:03 11 Nov 2022, 20:33

              @kiro

              I just scraped some on my RPI4 with both scrapers (normal and rpie)

              For archimedes all videos are named *.mp4, so for this system that's good and all videos seem to play now.
              However I discovered a new problem.
              The last 2 games that are recognised and for which media is downloaded are not added to the gamelist.xml.
              I checked older gamelists and they also miss these.

              For arcade(shooter category),arcade(pinball category) and atari7800 the .png problem still remains.
              Here I also checked If something was missing in atari7800.
              The last 4 games that are recognised and for which media is downloaded are not added to the gamelist.xml.
              For arcade(shooter category) the last 3 games were missing.
              For arcade(pinball category) the last 3 games were missing.

              Edit :
              For the system with videos named as .png for some videos only sound is available.

              K 1 Reply Last reply 11 Nov 2022, 20:59 Reply Quote 0
              • K
                kiro @Folly
                last edited by 11 Nov 2022, 20:59

                @Folly hi, thanks for the testing, can you give me a game name for the 7800 for which videos are downloaded as PNG? I cannot reproduce the issue.

                Just for clarity, the games that are not in the gamelisy, have the proper extension? (Recognized one)? I'm sure yes but better be safe πŸ™‚

                F 1 Reply Last reply 11 Nov 2022, 21:06 Reply Quote 0
                • F
                  Folly @kiro
                  last edited by Folly 11 Nov 2022, 21:06

                  @kiro

                  Do you need some sha1sums ?
                  Here is one :

                  sha1sum 'Choplifter (1987) (Atari).a78'
                  6457dd40aeadb9733602c4ad3498780ebd7bf0d1  Choplifter (1987) (Atari).a78
                  

                  Looks like these games are missing :

                  'Water Ski (1988) (Froggo).a78'
                  'Winter Games (1987) (Atari) [!].a78'
                  'Xenophobe (1989) (Atari) [!].a78'
                  'Xevious (1987) (Atari).a78'

                  This is the output of my videos folder for atari7800 :

                  '3D Asteroids (1987) (Atari) (Prototype) [a1].png'
                  '3D Asteroids (1987) (Atari) (Prototype).png'
                  'Ace of Aces (1988) (Atari).png'
                  'Alien Brigade (1990) (Atari) (CC2 Hack).png'
                  'Alien Brigade (1990) (Atari) [!].png'
                  'Asteroids (1987) (Atari).png'
                  'Ballblazer (1987) (Atari-Lucasfilm).png'
                  'Barnyard Blaster (1988) (Atari) [!].png'
                  'Basketbrawl (1990) (Atari).png'
                  'Beef Drop 7800 (Oct 30-Rev 08) (2004) (CC2 Hack).png'
                  'Centipede (1987) (Atari).png'
                  'Choplifter (1987) (Atari).png'
                  'Commando (1989) (Atari).png'
                  'Crack'\''ed (1988) (Atari).png'
                  'Crossbow (1988) (Atari) (CC2 Hack).png'
                  'Crossbow (1988) (Atari).png'
                  'Dark Chambers (1988) (Atari).png'
                  'Desert Falcon (1987) (Atari).mp4'
                  'Desert Falcon (2004) (Atari) (HSC Hack) [a1].mp4'
                  'Desert Falcon (2004) (Atari) (HSC Hack).mp4'
                  'Dig Dug (1987) (Atari).png'
                  'Donkey Kong (1988) (Atari).png'
                  'Donkey Kong Jr (1988) (Atari).png'
                  'Double Dragon (1989) (Activision) [!].mp4'
                  'F-18 Hornet (1988) (Absolute) [!].mp4'
                  'Fatal Run (1990) (Atari).png'
                  'Fight Night (1988) (Atari).png'
                  'Food Fight (1987) (Atari).png'
                  'Galaga (1987) (Atari).png'
                  'Gato (198x) (Atari) (Prototype) [a1].mp4'
                  'Gato (198x) (Atari) (Prototype) [a2].mp4'
                  'Gato (198x) (Atari) (Prototype).mp4'
                  'Hat Trick (1987) (Atari).png'
                  'Ikari Warriors (1989) (Atari).png'
                  'Impossible Mission (1987) (Atari) [a1].mp4'
                  'Impossible Mission (1987) (Atari) [f1].mp4'
                  'Impossible Mission (1987) (Atari) [!].mp4'
                  'Jinks (1989) (Atari).png'
                  'Joust (1987) (Atari).png'
                  'Karateka (1987) (Atari).png'
                  'Klax (1992) (Atari) (Prototype) [a1].mp4'
                  'Klax (1992) (Atari) (Prototype) [f1].mp4'
                  'Klax (1992) (Atari) (Prototype).mp4'
                  'Klax (2004) (Impossible Level 1 Hack).mp4'
                  'Klax (2004) (Impossible Level 2 Hack).mp4'
                  'Klax (2004) (Impossible Level 3 Hack).mp4'
                  'Kung Fu Master (1989) (Absolute) [!].png'
                  'Mario Brothers (1988) (Atari).png'
                  'Mat Mania Challenge (1989) (Atari).png'
                  'Mean 18 Golf (1988) (Atari) [!].png'
                  'Meltdown (1990) (Atari).png'
                  'Midnight Mutants (1990) (Atari).png'
                  'Motor Psycho (1990) (Atari).png'
                  'Ms. Pac-Man (1987) (Atari).png'
                  'Ninja Golf (1990) (Atari).png'
                  'One-on-One Basketball (1987) (Atari).png'
                  'Pete Rose Baseball (1989) (Absolute) [!].png'
                  'Pit Fighter (1992) (Atari) (Prototype) [a1].png'
                  'Pit Fighter (1992) (Atari) (Prototype) [a2].png'
                  'Pit Fighter (1992) (Atari) (Prototype).png'
                  'Planet Smashers (1990) (Atari).png'
                  'Pole Position II (1987) (Atari).png'
                  'Rampage (1989) (Activision) [!].mp4'
                  'RealSports Baseball (1988) (Atari).png'
                  'Rescue on Fractalus (1984) (Atari) (Prototype) (CC2 Hack).png'
                  'Rescue on Fractalus (1984) (Atari) (Prototype).png'
                  'Robotron 2084 (1987) (Atari).png'
                  'Scrapyard Dog (1990) (Atari).png'
                  'Sentinel (1988) (Atari) [!].png'
                  'Summer Games (1987) (Atari) [!].png'
                  'Super Huey UH-IX (1989) (Atari).png'
                  'Super Skateboardin'\'' (1989) (Absolute) [!].png'
                  'Tank Command (1988) (Froggo) [!].png'
                  'Title Match Pro Wrestling (1989) (Absolute) [!].png'
                  'Tomcat - The F-14 (1989) (Absolute) [!].png'
                  'Touchdown Football (1988) (Atari).png'
                  'Tower Toppler (1988) (Atari) [!].png'
                  'Tubes (2004) (Bruce Tomlin).png'
                  'Tubes (Nov 3) (2004) (Bruce Tomlin).png'
                  'Tubes (Nov 3-Signed) (2004) (Bruce Tomlin).png'
                  'Water Ski (1988) (Froggo).png'
                  'Winter Games (1987) (Atari) [!].png'
                  'Xenophobe (1989) (Atari) [!].png'
                  'Xevious (1987) (Atari).png'
                  

                  Edit :
                  The gamelist.xml contain the video filenames mentioned above.
                  *.png or *.mp4

                  K 1 Reply Last reply 11 Nov 2022, 21:10 Reply Quote 0
                  • K
                    kiro @Folly
                    last edited by 11 Nov 2022, 21:10

                    @Folly thanks, very helpful, will have a look tomorrow.

                    F 1 Reply Last reply 11 Nov 2022, 21:14 Reply Quote 0
                    • F
                      Folly @kiro
                      last edited by 11 Nov 2022, 21:14

                      @kiro

                      If I can I will test the same in my VMWARE tomorrow.
                      I suspect it will run fine in that VM.

                      K 1 Reply Last reply 11 Nov 2022, 21:16 Reply Quote 0
                      • K
                        kiro @Folly
                        last edited by 11 Nov 2022, 21:16

                        @Folly if it works in the VM then I'm really lost🀣...the code is the same and it is getting the information from the same backend API...so there's no reason for it to behave differently

                        F 2 Replies Last reply 11 Nov 2022, 21:18 Reply Quote 0
                        • F
                          Folly @kiro
                          last edited by Folly 11 Nov 2022, 21:19 11 Nov 2022, 21:18

                          @kiro

                          Yes, one would think so.
                          Could it be some version issue for python3 ?

                          BTW.
                          I am fully updated on the pi4.

                          1 Reply Last reply Reply Quote 0
                          • F
                            Folly @kiro
                            last edited by Folly 11 Dec 2022, 09:30 11 Nov 2022, 21:30

                            @kiro

                            Just tested atari7800 in my VM with retroscraper-rpie.
                            I see the same issues, so I replicated the problem here.
                            So that's basically a good thing.

                            I also noticed now that game-specific retroarch config files are generated in the roms directory while not using the option --bezel.
                            Adding the option --bezel will give the same results as without this option.
                            One would assume that a bezel folder is created containing the bezel images when adding this option.

                            In the VM the program doesn't always go back to the cli command prompt, after saying :
                            SCRAPPING ENDED --- Thank you for using retroscraper!!
                            (I have to press ctrl+c sometimes)

                            Multiple tries ended differently with the same romset :

                            Game : Tubes
                            Game : Tubes
                            Game : Tower Toppler
                            Game : Tubes
                            Game : Water Ski
                            Game : Xenophobe
                            SCRAPPING ENDED --- Thank you for using retroscraper!!
                            
                            Game : Tubes
                            Game : Tubes
                            Game : Tubes
                            Game : Water Ski
                            Game : Winter Games
                            SCRAPPING ENDED --- Thank you for using retroscraper!!
                            

                            To be secure I added the .local/bin path as it complained earlier with the setup.

                            pi@Bullseye-VM:~$ echo $PATH
                            /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
                            pi@Bullseye-VM:~$ PATH="$HOME/.local/bin:$PATH"
                            pi@Bullseye-VM:~$ echo $PATH
                            /home/pi/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

                            Still the same results.

                            K 1 Reply Last reply 12 Nov 2022, 08:45 Reply Quote 0
                            • K
                              kiro @Folly
                              last edited by 12 Nov 2022, 08:45

                              @Folly thanks for all the testing, will start looking at it on Monday when I'm back.

                              F 1 Reply Last reply 12 Nov 2022, 09:00 Reply Quote 0
                              • F
                                Folly @kiro
                                last edited by Folly 11 Dec 2022, 09:01 12 Nov 2022, 09:00

                                @kiro

                                That's ok, no problem.

                                Have a good weekend ;-)

                                K 1 Reply Last reply 13 Nov 2022, 13:37 Reply Quote 0
                                • K
                                  kiro @Folly
                                  last edited by 13 Nov 2022, 13:37

                                  @Folly hi folly...I've quickly checked the backend and it returns format MP4 for the videos of all the games you mentioned, it's surprising to say the least. I will have a deeper check tomorrow.

                                  F 1 Reply Last reply 13 Nov 2022, 17:33 Reply Quote 0
                                  • F
                                    Folly @kiro
                                    last edited by Folly 13 Nov 2022, 17:33

                                    @kiro

                                    I installed retroscraper-rpie on my debian buster vm 32bit x86 VM.
                                    I was thinking perhaps it will work on this VM.

                                    First I did a total update/upgrade/reboot.

                                    This is my log installing it all :
                                    https://pastebin.com/raw/gzJpdz1b

                                    I scraped the atari7800 roms and had the the same results as earlier. (same results as described earlier)
                                    (used : python3 retroscraper.py --systems atari7800 )

                                    Perhaps you can find something in my log.

                                    I'am not quite sure which dependancies are loaded by the script.
                                    If somehow the same dependancies are already installed on the OS in different paths with sudo and apt, for example.
                                    Could be that the wrong depandent programs are used from other paths (with an other version).
                                    I checked this theory by forcing only "$HOME/.local/bin" in the PATH like this :

                                    pi@VM-32bit:~/retroscraper-rpie$ PATH="$HOME/.local/bin"
                                    pi@VM-32bit:~/retroscraper-rpie$ /usr/bin/python3 retroscraper.py --systems atari7800 --cleanmedia

                                    Same result, so this doesn't seem the be the problem either.

                                    K 1 Reply Last reply 14 Nov 2022, 09:49 Reply Quote 0
                                    • K
                                      kiro @Folly
                                      last edited by 14 Nov 2022, 09:49

                                      @Folly Thank you so much for all the testing.

                                      I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

                                      I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

                                      I'll keep investigating in the meantime, and will start working on the relative path thing today.

                                      Take care!

                                      F 1 Reply Last reply 14 Nov 2022, 11:04 Reply Quote 0
                                      • F
                                        Folly @kiro
                                        last edited by Folly 14 Nov 2022, 11:04

                                        @kiro said in [SOFT] New Scraper in the works:

                                        @Folly Thank you so much for all the testing.

                                        I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

                                        Great !
                                        I did some tests.

                                        python3 retroscraper.py --systems atari7800 --cleanmedia
                                        

                                        πŸ‘ - seems to be consistent now in detected games
                                        πŸ‘ - seems to add all the detected games now in gamelist.xml
                                        πŸ‘ - no game-specific configs are generated anymore

                                        python3 retroscraper.py --systems atari7800 --bezel --cleanmedia
                                        

                                        πŸ‘ - seems also to be consistent now in detected games
                                        πŸ‘ - seems also to add all the detected games now in gamelist.xml
                                        πŸ‘ - game-specific configs are generated as it should
                                        πŸ‘ - bezel folder is created and bezel files are downloaded

                                        I suspected that the bezel files would be in /home/pi/RetroPie/roms/atari7800/bezels but the were in /home/pi/RetroPie/overlays/atari7800/bezels.
                                        Is it an idea to use this folder instead for the future ? :
                                        /home/pi/RetroPie/roms/atari7800/media/retroarch/overlays/bezels
                                        Using the same structure as I proposed for the emulationstation media :
                                        /home/pi/RetroPie/roms/atari7800/media/emulationstation/<folders>
                                        Then you can backup all in one time and with your option --cleanmedia you only have to delete 1 folder (media) which would make it much simpler.
                                        Good idea ?

                                        I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

                                        No I haven't.

                                        I'll keep investigating in the meantime, and will start working on the relative path thing today.

                                        I think that somehow string variables aren't updated correctly.
                                        Though I have too little knowledge of your script.

                                        Take care!

                                        Same !

                                        K 1 Reply Last reply 14 Nov 2022, 16:33 Reply Quote 0
                                        • K
                                          kiro @Folly
                                          last edited by 14 Nov 2022, 16:33

                                          @Folly I've updated a new version of the 'light' retroscraper with a new flag '--debug' which will create a more complete log, maybe can you run it on the system tat still creates the png so we might understand what the problem is??

                                          I've scanned my whole romset and no issues thousands of roms to be honest)

                                          Thx!

                                          F 1 Reply Last reply 14 Nov 2022, 16:55 Reply Quote 0
                                          55 out of 253
                                          • First post
                                            55/253
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received