RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] New Scraper in the works

    Scheduled Pinned Locked Moved Projects and Themes
    scrapingscrappersoftware
    253 Posts 7 Posters 62.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • kiroK
      kiro @Folly
      last edited by

      @Folly hi, thanks for the testing, can you give me a game name for the 7800 for which videos are downloaded as PNG? I cannot reproduce the issue.

      Just for clarity, the games that are not in the gamelisy, have the proper extension? (Recognized one)? I'm sure yes but better be safe πŸ™‚

      F 1 Reply Last reply Reply Quote 0
      • F
        Folly @kiro
        last edited by Folly

        @kiro

        Do you need some sha1sums ?
        Here is one :

        sha1sum 'Choplifter (1987) (Atari).a78'
        6457dd40aeadb9733602c4ad3498780ebd7bf0d1  Choplifter (1987) (Atari).a78
        

        Looks like these games are missing :

        'Water Ski (1988) (Froggo).a78'
        'Winter Games (1987) (Atari) [!].a78'
        'Xenophobe (1989) (Atari) [!].a78'
        'Xevious (1987) (Atari).a78'
        

        This is the output of my videos folder for atari7800 :

        '3D Asteroids (1987) (Atari) (Prototype) [a1].png'
        '3D Asteroids (1987) (Atari) (Prototype).png'
        'Ace of Aces (1988) (Atari).png'
        'Alien Brigade (1990) (Atari) (CC2 Hack).png'
        'Alien Brigade (1990) (Atari) [!].png'
        'Asteroids (1987) (Atari).png'
        'Ballblazer (1987) (Atari-Lucasfilm).png'
        'Barnyard Blaster (1988) (Atari) [!].png'
        'Basketbrawl (1990) (Atari).png'
        'Beef Drop 7800 (Oct 30-Rev 08) (2004) (CC2 Hack).png'
        'Centipede (1987) (Atari).png'
        'Choplifter (1987) (Atari).png'
        'Commando (1989) (Atari).png'
        'Crack'\''ed (1988) (Atari).png'
        'Crossbow (1988) (Atari) (CC2 Hack).png'
        'Crossbow (1988) (Atari).png'
        'Dark Chambers (1988) (Atari).png'
        'Desert Falcon (1987) (Atari).mp4'
        'Desert Falcon (2004) (Atari) (HSC Hack) [a1].mp4'
        'Desert Falcon (2004) (Atari) (HSC Hack).mp4'
        'Dig Dug (1987) (Atari).png'
        'Donkey Kong (1988) (Atari).png'
        'Donkey Kong Jr (1988) (Atari).png'
        'Double Dragon (1989) (Activision) [!].mp4'
        'F-18 Hornet (1988) (Absolute) [!].mp4'
        'Fatal Run (1990) (Atari).png'
        'Fight Night (1988) (Atari).png'
        'Food Fight (1987) (Atari).png'
        'Galaga (1987) (Atari).png'
        'Gato (198x) (Atari) (Prototype) [a1].mp4'
        'Gato (198x) (Atari) (Prototype) [a2].mp4'
        'Gato (198x) (Atari) (Prototype).mp4'
        'Hat Trick (1987) (Atari).png'
        'Ikari Warriors (1989) (Atari).png'
        'Impossible Mission (1987) (Atari) [a1].mp4'
        'Impossible Mission (1987) (Atari) [f1].mp4'
        'Impossible Mission (1987) (Atari) [!].mp4'
        'Jinks (1989) (Atari).png'
        'Joust (1987) (Atari).png'
        'Karateka (1987) (Atari).png'
        'Klax (1992) (Atari) (Prototype) [a1].mp4'
        'Klax (1992) (Atari) (Prototype) [f1].mp4'
        'Klax (1992) (Atari) (Prototype).mp4'
        'Klax (2004) (Impossible Level 1 Hack).mp4'
        'Klax (2004) (Impossible Level 2 Hack).mp4'
        'Klax (2004) (Impossible Level 3 Hack).mp4'
        'Kung Fu Master (1989) (Absolute) [!].png'
        'Mario Brothers (1988) (Atari).png'
        'Mat Mania Challenge (1989) (Atari).png'
        'Mean 18 Golf (1988) (Atari) [!].png'
        'Meltdown (1990) (Atari).png'
        'Midnight Mutants (1990) (Atari).png'
        'Motor Psycho (1990) (Atari).png'
        'Ms. Pac-Man (1987) (Atari).png'
        'Ninja Golf (1990) (Atari).png'
        'One-on-One Basketball (1987) (Atari).png'
        'Pete Rose Baseball (1989) (Absolute) [!].png'
        'Pit Fighter (1992) (Atari) (Prototype) [a1].png'
        'Pit Fighter (1992) (Atari) (Prototype) [a2].png'
        'Pit Fighter (1992) (Atari) (Prototype).png'
        'Planet Smashers (1990) (Atari).png'
        'Pole Position II (1987) (Atari).png'
        'Rampage (1989) (Activision) [!].mp4'
        'RealSports Baseball (1988) (Atari).png'
        'Rescue on Fractalus (1984) (Atari) (Prototype) (CC2 Hack).png'
        'Rescue on Fractalus (1984) (Atari) (Prototype).png'
        'Robotron 2084 (1987) (Atari).png'
        'Scrapyard Dog (1990) (Atari).png'
        'Sentinel (1988) (Atari) [!].png'
        'Summer Games (1987) (Atari) [!].png'
        'Super Huey UH-IX (1989) (Atari).png'
        'Super Skateboardin'\'' (1989) (Absolute) [!].png'
        'Tank Command (1988) (Froggo) [!].png'
        'Title Match Pro Wrestling (1989) (Absolute) [!].png'
        'Tomcat - The F-14 (1989) (Absolute) [!].png'
        'Touchdown Football (1988) (Atari).png'
        'Tower Toppler (1988) (Atari) [!].png'
        'Tubes (2004) (Bruce Tomlin).png'
        'Tubes (Nov 3) (2004) (Bruce Tomlin).png'
        'Tubes (Nov 3-Signed) (2004) (Bruce Tomlin).png'
        'Water Ski (1988) (Froggo).png'
        'Winter Games (1987) (Atari) [!].png'
        'Xenophobe (1989) (Atari) [!].png'
        'Xevious (1987) (Atari).png'
        

        Edit :
        The gamelist.xml contain the video filenames mentioned above.
        *.png or *.mp4

        kiroK 1 Reply Last reply Reply Quote 0
        • kiroK
          kiro @Folly
          last edited by

          @Folly thanks, very helpful, will have a look tomorrow.

          F 1 Reply Last reply Reply Quote 0
          • F
            Folly @kiro
            last edited by

            @kiro

            If I can I will test the same in my VMWARE tomorrow.
            I suspect it will run fine in that VM.

            kiroK 1 Reply Last reply Reply Quote 0
            • kiroK
              kiro @Folly
              last edited by

              @Folly if it works in the VM then I'm really lost🀣...the code is the same and it is getting the information from the same backend API...so there's no reason for it to behave differently

              F 2 Replies Last reply Reply Quote 0
              • F
                Folly @kiro
                last edited by Folly

                @kiro

                Yes, one would think so.
                Could it be some version issue for python3 ?

                BTW.
                I am fully updated on the pi4.

                1 Reply Last reply Reply Quote 0
                • F
                  Folly @kiro
                  last edited by Folly

                  @kiro

                  Just tested atari7800 in my VM with retroscraper-rpie.
                  I see the same issues, so I replicated the problem here.
                  So that's basically a good thing.

                  I also noticed now that game-specific retroarch config files are generated in the roms directory while not using the option --bezel.
                  Adding the option --bezel will give the same results as without this option.
                  One would assume that a bezel folder is created containing the bezel images when adding this option.

                  In the VM the program doesn't always go back to the cli command prompt, after saying :
                  SCRAPPING ENDED --- Thank you for using retroscraper!!
                  (I have to press ctrl+c sometimes)

                  Multiple tries ended differently with the same romset :

                  Game : Tubes
                  Game : Tubes
                  Game : Tower Toppler
                  Game : Tubes
                  Game : Water Ski
                  Game : Xenophobe
                  SCRAPPING ENDED --- Thank you for using retroscraper!!
                  
                  Game : Tubes
                  Game : Tubes
                  Game : Tubes
                  Game : Water Ski
                  Game : Winter Games
                  SCRAPPING ENDED --- Thank you for using retroscraper!!
                  

                  To be secure I added the .local/bin path as it complained earlier with the setup.

                  pi@Bullseye-VM:~$ echo $PATH
                  /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
                  pi@Bullseye-VM:~$ PATH="$HOME/.local/bin:$PATH"
                  pi@Bullseye-VM:~$ echo $PATH
                  /home/pi/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
                  

                  Still the same results.

                  kiroK 1 Reply Last reply Reply Quote 0
                  • kiroK
                    kiro @Folly
                    last edited by

                    @Folly thanks for all the testing, will start looking at it on Monday when I'm back.

                    F 1 Reply Last reply Reply Quote 0
                    • F
                      Folly @kiro
                      last edited by Folly

                      @kiro

                      That's ok, no problem.

                      Have a good weekend ;-)

                      kiroK 1 Reply Last reply Reply Quote 0
                      • kiroK
                        kiro @Folly
                        last edited by

                        @Folly hi folly...I've quickly checked the backend and it returns format MP4 for the videos of all the games you mentioned, it's surprising to say the least. I will have a deeper check tomorrow.

                        F 1 Reply Last reply Reply Quote 0
                        • F
                          Folly @kiro
                          last edited by Folly

                          @kiro

                          I installed retroscraper-rpie on my debian buster vm 32bit x86 VM.
                          I was thinking perhaps it will work on this VM.

                          First I did a total update/upgrade/reboot.

                          This is my log installing it all :
                          https://pastebin.com/raw/gzJpdz1b

                          I scraped the atari7800 roms and had the the same results as earlier. (same results as described earlier)
                          (used : python3 retroscraper.py --systems atari7800 )

                          Perhaps you can find something in my log.

                          I'am not quite sure which dependancies are loaded by the script.
                          If somehow the same dependancies are already installed on the OS in different paths with sudo and apt, for example.
                          Could be that the wrong depandent programs are used from other paths (with an other version).
                          I checked this theory by forcing only "$HOME/.local/bin" in the PATH like this :

                          pi@VM-32bit:~/retroscraper-rpie$ PATH="$HOME/.local/bin"
                          pi@VM-32bit:~/retroscraper-rpie$ /usr/bin/python3 retroscraper.py --systems atari7800 --cleanmedia
                          

                          Same result, so this doesn't seem the be the problem either.

                          kiroK 1 Reply Last reply Reply Quote 0
                          • kiroK
                            kiro @Folly
                            last edited by

                            @Folly Thank you so much for all the testing.

                            I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

                            I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

                            I'll keep investigating in the meantime, and will start working on the relative path thing today.

                            Take care!

                            F 1 Reply Last reply Reply Quote 0
                            • F
                              Folly @kiro
                              last edited by Folly

                              @kiro said in [SOFT] New Scraper in the works:

                              @Folly Thank you so much for all the testing.

                              I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

                              Great !
                              I did some tests.

                              python3 retroscraper.py --systems atari7800 --cleanmedia
                              

                              πŸ‘ - seems to be consistent now in detected games
                              πŸ‘ - seems to add all the detected games now in gamelist.xml
                              πŸ‘ - no game-specific configs are generated anymore

                              python3 retroscraper.py --systems atari7800 --bezel --cleanmedia
                              

                              πŸ‘ - seems also to be consistent now in detected games
                              πŸ‘ - seems also to add all the detected games now in gamelist.xml
                              πŸ‘ - game-specific configs are generated as it should
                              πŸ‘ - bezel folder is created and bezel files are downloaded

                              I suspected that the bezel files would be in /home/pi/RetroPie/roms/atari7800/bezels but the were in /home/pi/RetroPie/overlays/atari7800/bezels.
                              Is it an idea to use this folder instead for the future ? :
                              /home/pi/RetroPie/roms/atari7800/media/retroarch/overlays/bezels
                              Using the same structure as I proposed for the emulationstation media :
                              /home/pi/RetroPie/roms/atari7800/media/emulationstation/<folders>
                              Then you can backup all in one time and with your option --cleanmedia you only have to delete 1 folder (media) which would make it much simpler.
                              Good idea ?

                              I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

                              No I haven't.

                              I'll keep investigating in the meantime, and will start working on the relative path thing today.

                              I think that somehow string variables aren't updated correctly.
                              Though I have too little knowledge of your script.

                              Take care!

                              Same !

                              kiroK 1 Reply Last reply Reply Quote 0
                              • kiroK
                                kiro @Folly
                                last edited by

                                @Folly I've updated a new version of the 'light' retroscraper with a new flag '--debug' which will create a more complete log, maybe can you run it on the system tat still creates the png so we might understand what the problem is??

                                I've scanned my whole romset and no issues thousands of roms to be honest)

                                Thx!

                                F 1 Reply Last reply Reply Quote 0
                                • F
                                  Folly @kiro
                                  last edited by Folly

                                  @kiro

                                  OK, I will do.

                                  Found the log, but it's too big for pastebin.
                                  Probably paste it on the google-drive.

                                  kiroK 1 Reply Last reply Reply Quote 0
                                  • kiroK
                                    kiro @Folly
                                    last edited by

                                    @Folly thanks!

                                    F 1 Reply Last reply Reply Quote 0
                                    • F
                                      Folly @kiro
                                      last edited by Folly

                                      @kiro

                                      retroscraper.log removed from google-drive.

                                      Looks like the format variable is not updated correctly.

                                      kiroK 1 Reply Last reply Reply Quote 0
                                      • kiroK
                                        kiro @Folly
                                        last edited by

                                        @Folly Thanks, will have a look right now.

                                        On the other hand, I've updated both versions to cater for systems with 'non-standard names' such as 'shooters' :-) The only drawback is that the scraper will not be able to do a name search, due to the fact that it does not understand which is the actual system, but if checksums are in the DB they'll show up properly.

                                        F 2 Replies Last reply Reply Quote 0
                                        • F
                                          Folly @kiro
                                          last edited by Folly

                                          @kiro

                                          Cool will give it a go.

                                          Yea, it works really nice !

                                          Seems a bit slower.
                                          Does it have to search longer in the database with an unknown system ?

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            Folly @kiro
                                            last edited by

                                            @kiro

                                            I am getting this error, nothing works anymore :

                                            pi@raspberrypi:~/retroscraper-rpie $ python3 retroscraper.py --systems arcade
                                            Loading RetroScraper config File
                                            Traceback (most recent call last):
                                              File "/home/pi/retroscraper-rpie/retroscraper.py", line 150, in <module>
                                                trans = complete['en']
                                            TypeError: list indices must be integers or slices, not str
                                            
                                            kiroK 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.