RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] New Scraper in the works

    Scheduled Pinned Locked Moved Projects and Themes
    scrapingscrappersoftware
    253 Posts 7 Posters 59.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      Folly @kiro
      last edited by

      @kiro

      If I can implement the setup somehow in my script then your retroscraper-rpie would probably be easy enough to add in my script.
      I will have to think about it some time.

      kiroK 1 Reply Last reply Reply Quote 0
      • kiroK
        kiro @Folly
        last edited by

        @Folly thanks! I actually do have a script which is running (installation goes fine as well as execution) based on scraper.sh script. My only concern right now is that output is buffered, so it seems (only seems) as if the scraper freezes.
        But thanks for the offer, definitely we can talk about doing something together.

        F 2 Replies Last reply Reply Quote 0
        • F
          Folly @kiro
          last edited by Folly

          @kiro

          I just did atari7800, although it also has the .png video issue, all videos and pictures are good.
          I must say if something is recognised then recognition seems to be perfect.
          I do not see that with other scraper methods.
          I also notice that the scraping is very fast, much faster that other scrape methods.
          Really nice !!!

          1 Reply Last reply Reply Quote 0
          • F
            Folly @kiro
            last edited by Folly

            @kiro

            For your information.
            I am testing your retroscraper-rpie now in my Debian11-x86_64 in VMWARE.
            Looks like the .png problem for videos doesn't occur there.
            Also the gamelist.xml contain the correct naming for the videos (.mp4)
            So this problem could be specific for the RPI/raspbian/raspbian version.

            ALL video seem to be playing correctly in emulationstation in my VM, although, for some videos, emulationstation seems to give errors like this :

            [src/libmpg123/layer3.c:2184] error: part2_3_length (1408) too large for available bit count (1368)
            

            Could it be that some videos are not that compatible or have some corruption ?

            kiroK 1 Reply Last reply Reply Quote 0
            • kiroK
              kiro @Folly
              last edited by

              @Folly hi, videos are being fixed as we speak, it will need some time. PNG issue should be independent from platform (after all is running the same code) let me know if you encounter this problem again.

              F 1 Reply Last reply Reply Quote 1
              • F
                Folly @kiro
                last edited by Folly

                @kiro

                I just scraped some on my RPI4 with both scrapers (normal and rpie)

                For archimedes all videos are named *.mp4, so for this system that's good and all videos seem to play now.
                However I discovered a new problem.
                The last 2 games that are recognised and for which media is downloaded are not added to the gamelist.xml.
                I checked older gamelists and they also miss these.

                For arcade(shooter category),arcade(pinball category) and atari7800 the .png problem still remains.
                Here I also checked If something was missing in atari7800.
                The last 4 games that are recognised and for which media is downloaded are not added to the gamelist.xml.
                For arcade(shooter category) the last 3 games were missing.
                For arcade(pinball category) the last 3 games were missing.

                Edit :
                For the system with videos named as .png for some videos only sound is available.

                kiroK 1 Reply Last reply Reply Quote 0
                • kiroK
                  kiro @Folly
                  last edited by

                  @Folly hi, thanks for the testing, can you give me a game name for the 7800 for which videos are downloaded as PNG? I cannot reproduce the issue.

                  Just for clarity, the games that are not in the gamelisy, have the proper extension? (Recognized one)? I'm sure yes but better be safe πŸ™‚

                  F 1 Reply Last reply Reply Quote 0
                  • F
                    Folly @kiro
                    last edited by Folly

                    @kiro

                    Do you need some sha1sums ?
                    Here is one :

                    sha1sum 'Choplifter (1987) (Atari).a78'
                    6457dd40aeadb9733602c4ad3498780ebd7bf0d1  Choplifter (1987) (Atari).a78
                    

                    Looks like these games are missing :

                    'Water Ski (1988) (Froggo).a78'
                    'Winter Games (1987) (Atari) [!].a78'
                    'Xenophobe (1989) (Atari) [!].a78'
                    'Xevious (1987) (Atari).a78'
                    

                    This is the output of my videos folder for atari7800 :

                    '3D Asteroids (1987) (Atari) (Prototype) [a1].png'
                    '3D Asteroids (1987) (Atari) (Prototype).png'
                    'Ace of Aces (1988) (Atari).png'
                    'Alien Brigade (1990) (Atari) (CC2 Hack).png'
                    'Alien Brigade (1990) (Atari) [!].png'
                    'Asteroids (1987) (Atari).png'
                    'Ballblazer (1987) (Atari-Lucasfilm).png'
                    'Barnyard Blaster (1988) (Atari) [!].png'
                    'Basketbrawl (1990) (Atari).png'
                    'Beef Drop 7800 (Oct 30-Rev 08) (2004) (CC2 Hack).png'
                    'Centipede (1987) (Atari).png'
                    'Choplifter (1987) (Atari).png'
                    'Commando (1989) (Atari).png'
                    'Crack'\''ed (1988) (Atari).png'
                    'Crossbow (1988) (Atari) (CC2 Hack).png'
                    'Crossbow (1988) (Atari).png'
                    'Dark Chambers (1988) (Atari).png'
                    'Desert Falcon (1987) (Atari).mp4'
                    'Desert Falcon (2004) (Atari) (HSC Hack) [a1].mp4'
                    'Desert Falcon (2004) (Atari) (HSC Hack).mp4'
                    'Dig Dug (1987) (Atari).png'
                    'Donkey Kong (1988) (Atari).png'
                    'Donkey Kong Jr (1988) (Atari).png'
                    'Double Dragon (1989) (Activision) [!].mp4'
                    'F-18 Hornet (1988) (Absolute) [!].mp4'
                    'Fatal Run (1990) (Atari).png'
                    'Fight Night (1988) (Atari).png'
                    'Food Fight (1987) (Atari).png'
                    'Galaga (1987) (Atari).png'
                    'Gato (198x) (Atari) (Prototype) [a1].mp4'
                    'Gato (198x) (Atari) (Prototype) [a2].mp4'
                    'Gato (198x) (Atari) (Prototype).mp4'
                    'Hat Trick (1987) (Atari).png'
                    'Ikari Warriors (1989) (Atari).png'
                    'Impossible Mission (1987) (Atari) [a1].mp4'
                    'Impossible Mission (1987) (Atari) [f1].mp4'
                    'Impossible Mission (1987) (Atari) [!].mp4'
                    'Jinks (1989) (Atari).png'
                    'Joust (1987) (Atari).png'
                    'Karateka (1987) (Atari).png'
                    'Klax (1992) (Atari) (Prototype) [a1].mp4'
                    'Klax (1992) (Atari) (Prototype) [f1].mp4'
                    'Klax (1992) (Atari) (Prototype).mp4'
                    'Klax (2004) (Impossible Level 1 Hack).mp4'
                    'Klax (2004) (Impossible Level 2 Hack).mp4'
                    'Klax (2004) (Impossible Level 3 Hack).mp4'
                    'Kung Fu Master (1989) (Absolute) [!].png'
                    'Mario Brothers (1988) (Atari).png'
                    'Mat Mania Challenge (1989) (Atari).png'
                    'Mean 18 Golf (1988) (Atari) [!].png'
                    'Meltdown (1990) (Atari).png'
                    'Midnight Mutants (1990) (Atari).png'
                    'Motor Psycho (1990) (Atari).png'
                    'Ms. Pac-Man (1987) (Atari).png'
                    'Ninja Golf (1990) (Atari).png'
                    'One-on-One Basketball (1987) (Atari).png'
                    'Pete Rose Baseball (1989) (Absolute) [!].png'
                    'Pit Fighter (1992) (Atari) (Prototype) [a1].png'
                    'Pit Fighter (1992) (Atari) (Prototype) [a2].png'
                    'Pit Fighter (1992) (Atari) (Prototype).png'
                    'Planet Smashers (1990) (Atari).png'
                    'Pole Position II (1987) (Atari).png'
                    'Rampage (1989) (Activision) [!].mp4'
                    'RealSports Baseball (1988) (Atari).png'
                    'Rescue on Fractalus (1984) (Atari) (Prototype) (CC2 Hack).png'
                    'Rescue on Fractalus (1984) (Atari) (Prototype).png'
                    'Robotron 2084 (1987) (Atari).png'
                    'Scrapyard Dog (1990) (Atari).png'
                    'Sentinel (1988) (Atari) [!].png'
                    'Summer Games (1987) (Atari) [!].png'
                    'Super Huey UH-IX (1989) (Atari).png'
                    'Super Skateboardin'\'' (1989) (Absolute) [!].png'
                    'Tank Command (1988) (Froggo) [!].png'
                    'Title Match Pro Wrestling (1989) (Absolute) [!].png'
                    'Tomcat - The F-14 (1989) (Absolute) [!].png'
                    'Touchdown Football (1988) (Atari).png'
                    'Tower Toppler (1988) (Atari) [!].png'
                    'Tubes (2004) (Bruce Tomlin).png'
                    'Tubes (Nov 3) (2004) (Bruce Tomlin).png'
                    'Tubes (Nov 3-Signed) (2004) (Bruce Tomlin).png'
                    'Water Ski (1988) (Froggo).png'
                    'Winter Games (1987) (Atari) [!].png'
                    'Xenophobe (1989) (Atari) [!].png'
                    'Xevious (1987) (Atari).png'
                    

                    Edit :
                    The gamelist.xml contain the video filenames mentioned above.
                    *.png or *.mp4

                    kiroK 1 Reply Last reply Reply Quote 0
                    • kiroK
                      kiro @Folly
                      last edited by

                      @Folly thanks, very helpful, will have a look tomorrow.

                      F 1 Reply Last reply Reply Quote 0
                      • F
                        Folly @kiro
                        last edited by

                        @kiro

                        If I can I will test the same in my VMWARE tomorrow.
                        I suspect it will run fine in that VM.

                        kiroK 1 Reply Last reply Reply Quote 0
                        • kiroK
                          kiro @Folly
                          last edited by

                          @Folly if it works in the VM then I'm really lost🀣...the code is the same and it is getting the information from the same backend API...so there's no reason for it to behave differently

                          F 2 Replies Last reply Reply Quote 0
                          • F
                            Folly @kiro
                            last edited by Folly

                            @kiro

                            Yes, one would think so.
                            Could it be some version issue for python3 ?

                            BTW.
                            I am fully updated on the pi4.

                            1 Reply Last reply Reply Quote 0
                            • F
                              Folly @kiro
                              last edited by Folly

                              @kiro

                              Just tested atari7800 in my VM with retroscraper-rpie.
                              I see the same issues, so I replicated the problem here.
                              So that's basically a good thing.

                              I also noticed now that game-specific retroarch config files are generated in the roms directory while not using the option --bezel.
                              Adding the option --bezel will give the same results as without this option.
                              One would assume that a bezel folder is created containing the bezel images when adding this option.

                              In the VM the program doesn't always go back to the cli command prompt, after saying :
                              SCRAPPING ENDED --- Thank you for using retroscraper!!
                              (I have to press ctrl+c sometimes)

                              Multiple tries ended differently with the same romset :

                              Game : Tubes
                              Game : Tubes
                              Game : Tower Toppler
                              Game : Tubes
                              Game : Water Ski
                              Game : Xenophobe
                              SCRAPPING ENDED --- Thank you for using retroscraper!!
                              
                              Game : Tubes
                              Game : Tubes
                              Game : Tubes
                              Game : Water Ski
                              Game : Winter Games
                              SCRAPPING ENDED --- Thank you for using retroscraper!!
                              

                              To be secure I added the .local/bin path as it complained earlier with the setup.

                              pi@Bullseye-VM:~$ echo $PATH
                              /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
                              pi@Bullseye-VM:~$ PATH="$HOME/.local/bin:$PATH"
                              pi@Bullseye-VM:~$ echo $PATH
                              /home/pi/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
                              

                              Still the same results.

                              kiroK 1 Reply Last reply Reply Quote 0
                              • kiroK
                                kiro @Folly
                                last edited by

                                @Folly thanks for all the testing, will start looking at it on Monday when I'm back.

                                F 1 Reply Last reply Reply Quote 0
                                • F
                                  Folly @kiro
                                  last edited by Folly

                                  @kiro

                                  That's ok, no problem.

                                  Have a good weekend ;-)

                                  kiroK 1 Reply Last reply Reply Quote 0
                                  • kiroK
                                    kiro @Folly
                                    last edited by

                                    @Folly hi folly...I've quickly checked the backend and it returns format MP4 for the videos of all the games you mentioned, it's surprising to say the least. I will have a deeper check tomorrow.

                                    F 1 Reply Last reply Reply Quote 0
                                    • F
                                      Folly @kiro
                                      last edited by Folly

                                      @kiro

                                      I installed retroscraper-rpie on my debian buster vm 32bit x86 VM.
                                      I was thinking perhaps it will work on this VM.

                                      First I did a total update/upgrade/reboot.

                                      This is my log installing it all :
                                      https://pastebin.com/raw/gzJpdz1b

                                      I scraped the atari7800 roms and had the the same results as earlier. (same results as described earlier)
                                      (used : python3 retroscraper.py --systems atari7800 )

                                      Perhaps you can find something in my log.

                                      I'am not quite sure which dependancies are loaded by the script.
                                      If somehow the same dependancies are already installed on the OS in different paths with sudo and apt, for example.
                                      Could be that the wrong depandent programs are used from other paths (with an other version).
                                      I checked this theory by forcing only "$HOME/.local/bin" in the PATH like this :

                                      pi@VM-32bit:~/retroscraper-rpie$ PATH="$HOME/.local/bin"
                                      pi@VM-32bit:~/retroscraper-rpie$ /usr/bin/python3 retroscraper.py --systems atari7800 --cleanmedia
                                      

                                      Same result, so this doesn't seem the be the problem either.

                                      kiroK 1 Reply Last reply Reply Quote 0
                                      • kiroK
                                        kiro @Folly
                                        last edited by

                                        @Folly Thank you so much for all the testing.

                                        I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

                                        I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

                                        I'll keep investigating in the meantime, and will start working on the relative path thing today.

                                        Take care!

                                        F 1 Reply Last reply Reply Quote 0
                                        • F
                                          Folly @kiro
                                          last edited by Folly

                                          @kiro said in [SOFT] New Scraper in the works:

                                          @Folly Thank you so much for all the testing.

                                          I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

                                          Great !
                                          I did some tests.

                                          python3 retroscraper.py --systems atari7800 --cleanmedia
                                          

                                          πŸ‘ - seems to be consistent now in detected games
                                          πŸ‘ - seems to add all the detected games now in gamelist.xml
                                          πŸ‘ - no game-specific configs are generated anymore

                                          python3 retroscraper.py --systems atari7800 --bezel --cleanmedia
                                          

                                          πŸ‘ - seems also to be consistent now in detected games
                                          πŸ‘ - seems also to add all the detected games now in gamelist.xml
                                          πŸ‘ - game-specific configs are generated as it should
                                          πŸ‘ - bezel folder is created and bezel files are downloaded

                                          I suspected that the bezel files would be in /home/pi/RetroPie/roms/atari7800/bezels but the were in /home/pi/RetroPie/overlays/atari7800/bezels.
                                          Is it an idea to use this folder instead for the future ? :
                                          /home/pi/RetroPie/roms/atari7800/media/retroarch/overlays/bezels
                                          Using the same structure as I proposed for the emulationstation media :
                                          /home/pi/RetroPie/roms/atari7800/media/emulationstation/<folders>
                                          Then you can backup all in one time and with your option --cleanmedia you only have to delete 1 folder (media) which would make it much simpler.
                                          Good idea ?

                                          I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

                                          No I haven't.

                                          I'll keep investigating in the meantime, and will start working on the relative path thing today.

                                          I think that somehow string variables aren't updated correctly.
                                          Though I have too little knowledge of your script.

                                          Take care!

                                          Same !

                                          kiroK 1 Reply Last reply Reply Quote 0
                                          • kiroK
                                            kiro @Folly
                                            last edited by

                                            @Folly I've updated a new version of the 'light' retroscraper with a new flag '--debug' which will create a more complete log, maybe can you run it on the system tat still creates the png so we might understand what the problem is??

                                            I've scanned my whole romset and no issues thousands of roms to be honest)

                                            Thx!

                                            F 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.