RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] New Scraper in the works

    Scheduled Pinned Locked Moved Projects and Themes
    scrapingscrappersoftware
    253 Posts 7 Posters 71.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • FollyF
      Folly @kiro
      last edited by Folly

      @kiro

      I did try to scrape all_in1 but the images are "empty".
      Or do we have to wait until others try it too ?
      Any news about the videos ?

      kiroK 1 Reply Last reply Reply Quote 0
      • kiroK
        kiro @Folly
        last edited by

        @Folly hey sorry, I'm away from home at the moment, I'll be back on Wednesday and check by then! Sorry!

        FollyF 1 Reply Last reply Reply Quote 0
        • FollyF
          Folly @kiro
          last edited by

          @kiro

          Aha, no problem.
          Have good days, see you later ;-)

          kiroK 1 Reply Last reply Reply Quote 0
          • kiroK
            kiro @Folly
            last edited by

            @Folly I've just tried again the rztiger and it does work for me, the only difference is that I've got the .7z roms, can you point me (to my private email to avoid issues) to the .zip version of these roms??

            c52c2949-f38e-4e43-be47-1fb905d65e7c-image.png

            and the gamelist:

            <?xml version='1.0' encoding='utf-8'?><gameList>
            	<game>
            		<rating></rating>
            		<name>R-Zone: Batman Forever</name>
            		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzbatfor-marquee.png</marquee>
            		<image>/home/pi/RetroPie/roms/tigerrz/images/rzbatfor-image.png</image>
            		<publisher>Unknown</publisher>
            		<releasedate></releasedate>
            		<players></players>
            		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzbatfor-video.mp4</video>
            		<genre></genre>
            		<path>/home/pi/RetroPie/roms/tigerrz/rzbatfor.7z</path>
            		<developer>Unknown</developer>
            		<thumbnail/>
            		<desc>Batman Forever (handheld)</desc>
            		<playcount>0</playcount>
            		<lastplayed>0</lastplayed>
            	</game>
            	<game>
            		<rating></rating>
            		<name>R-Zone: Indy 500</name>
            		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzindy500-marquee.png</marquee>
            		<image>/home/pi/RetroPie/roms/tigerrz/images/rzindy500-image.png</image>
            		<publisher>Unknown</publisher>
            		<releasedate></releasedate>
            		<players></players>
            		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzindy500-video.mp4</video>
            		<genre></genre>
            		<path>/home/pi/RetroPie/roms/tigerrz/rzindy500.7z</path>
            		<developer>Unknown</developer>
            		<thumbnail/>
            		<desc>Indy 500 (handheld)</desc>
            		<playcount>0</playcount>
            		<lastplayed>0</lastplayed>
            	</game>
            	<game>
            		<rating></rating>
            		<name>R-Zone: Battle Arena Toshinden</name>
            		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rztoshden-marquee.png</marquee>
            		<image>/home/pi/RetroPie/roms/tigerrz/images/rztoshden-image.png</image>
            		<publisher>Unknown</publisher>
            		<releasedate></releasedate>
            		<players></players>
            		<video>/home/pi/RetroPie/roms/tigerrz/videos/rztoshden-video.mp4</video>
            		<genre></genre>
            		<path>/home/pi/RetroPie/roms/tigerrz/rztoshden.7z</path>
            		<developer>Unknown</developer>
            		<thumbnail/>
            		<desc>Battle Arena (handheld)</desc>
            		<playcount>0</playcount>
            		<lastplayed>0</lastplayed>
            	</game>
            </gameList>
            
            FollyF 1 Reply Last reply Reply Quote 0
            • FollyF
              Folly @kiro
              last edited by Folly

              @kiro

              Hi again.

              I tested .7z and .zip files and both worked now.
              I think the .7z files came from the mame-0.231-merged pack but this is not available anymore.

              You've got mail

              kiroK 1 Reply Last reply Reply Quote 0
              • kiroK
                kiro @Folly
                last edited by

                @Folly Cool, I'll test with the romset you mentioned.

                1 Reply Last reply Reply Quote 0
                • kiroK
                  kiro
                  last edited by

                  I've uploaded the latest version, adding the --sort flag, it works like this:

                  --sort ORIGDIR DESTDIR
                  

                  Where origdir is the place where all your roms are stored (they can all be mixed and in one single directoryt) and it will identify them and copy (not move) them over to your destdir/SYSTEM folder, where SYSTEM is the name of the system the roms belong to.

                  Have fun!

                  FollyF 1 Reply Last reply Reply Quote 0
                  • FollyF
                    Folly @kiro
                    last edited by

                    @kiro

                    I am experimenting with running retroscraper "online".

                    The advantage would be that the script is always up to date and that the basic script doesn't need to be stored anywhere.

                    I want to look if this is a good way of adding it to my script.
                    That way I only have to install the regular depends.

                    I already use a python script called gdrivedl in my module-script.
                    This is an example of running gdrivedl.py online using only the help option :

                    curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - -h
                    

                    So I wanted to see if I could do the same with your "retroscraper-rpie".
                    Basically, with the same help option and it actually works as expected :

                    curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h
                    

                    However your script uses more dependant scripts.
                    So when it comes to the actual scraping, it will miss these dependant scripts. (apicalls, checksums, scrapfunctions)

                    So I have a question :
                    Is there a way to make one script including these dependant scripts so it can be run online ?

                    kiroK 1 Reply Last reply Reply Quote 0
                    • kiroK
                      kiro @Folly
                      last edited by kiro

                      @Folly said in [SOFT] New Scraper in the works:

                      curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

                      Hi Folly, I guess it would be possible to include these files in the main python file, but it will add complexity to the maintenance of the code (at least for me :-) ). Let me check if something can be done through the command line or any other alternative. will keep you posted.

                      EDIT: Just discovered the httpimport module which actually allows you to import dependencies from the web. I'm testing it right now. It may have an impact on the initialization of the script though.

                      kiroK 1 Reply Last reply Reply Quote 0
                      • kiroK
                        kiro @kiro
                        last edited by

                        @Folly , some update. Try this:

                        curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3
                        

                        It has 2 consequences:

                        1-There's a new dependency (httpimport module, to be installed with pip)
                        2-Startup is slower since it grabs everything form the repository directly.

                        Let me know if it works for you.

                        FollyF 1 Reply Last reply Reply Quote 0
                        • FollyF
                          Folly @kiro
                          last edited by

                          @kiro

                          Thanks !
                          I will try it when I can and let you know how it goes.

                          FollyF 1 Reply Last reply Reply Quote 0
                          • FollyF
                            Folly @Folly
                            last edited by Folly

                            @Folly

                            Ok, I tested it out and it seems the idea could work.

                            But I have a few problems to overcome.

                            I have added the dependency installs in the depends function of my module script .
                            However I have to find a way to skip it properly so it will not try to when it's not needed.
                            Or I have to separate so it can be installed in a menu option.

                            Was hoping that it would run ok the first time.
                            It did work in the terminal
                            But from the script it didn't work.
                            I had to change the cmd, in my module script, and run it with su $user -c because the dependencies are installed under the normal user and therefor the script has also be run with the normal user.
                            So that one is solved.

                            To my surprise the waiting time of getting the script and it's dependencies remotely doesn't take that much time
                            Of-course you have to wait a little while but it's acceptable in my opinion and the scraping speed it just as normal.

                            Sadly I ran into an issue.
                            The crc and the sha1sum are saved in the name in the gamelist.xml.
                            This is an example :

                            <name>Boulder Dash,149604,0C0F7FF29CBE1CAA5A22AA0CF291BF5560DDD358</name>
                            

                            Do you have an idea what it could be ?


                            Edit :
                            For archiving and looking up purposes I will add here what code I added.

                            Getting the retroscraper-remote depends in my module-script :

                                local pip=$( su $user -c "python3 -c 'import pip'" 2>&1 )
                                local succ="ModuleNotFoundError"
                                echo "$pip"
                                if [[ $pip == *"$succ"* ]]; then
                                    su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
                                    su $user -c "python3 /tmp/get-pip.py"
                                fi
                                su $user -c "python3 -m pip install --user --upgrade pip wheel setuptools"
                                su $user -c "python3 -m pip install --user googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0"
                            

                            Test command from the terminal :

                            curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup
                            

                            Test command from the module-script :

                            su $user -c "curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup"
                            
                            kiroK 1 Reply Last reply Reply Quote 0
                            • kiroK
                              kiro @Folly
                              last edited by

                              @Folly Argh! Sorry, I uploaded a test version (it's been a while since I didn't touch the code, sorry) I've updated it! Should be fine now..

                              FollyF 1 Reply Last reply Reply Quote 0
                              • FollyF
                                Folly @kiro
                                last edited by

                                @kiro

                                Nice, so there is probably no issue.
                                Will do test again when I can.

                                I hope it's the correct route for me.
                                Nevertheless it's already quite interesting that we managed to run it this way.

                                kiroK 1 Reply Last reply Reply Quote 0
                                • kiroK
                                  kiro @Folly
                                  last edited by

                                  @Folly indeed, I never thought about that possibility to be honest. Let me know how it goes!

                                  FollyF 1 Reply Last reply Reply Quote 0
                                  • FollyF
                                    Folly @kiro
                                    last edited by

                                    @kiro

                                    Nice, it's working ;-)

                                    kiroK 1 Reply Last reply Reply Quote 0
                                    • kiroK
                                      kiro @Folly
                                      last edited by

                                      @Folly Cool... I'm still in the process of importing all mess roms...it's a pain in the .... anyway, as soon as that is done, I'll start adding the images from your playlists..

                                      FollyF 1 Reply Last reply Reply Quote 0
                                      • FollyF
                                        Folly @kiro
                                        last edited by Folly

                                        @kiro

                                        A pain ?
                                        I can imagine.
                                        Just be patient there is no hurry.

                                        Hopefully we can manage that the outputs are be becoming a good alternative for the predefined gamelists,
                                        because the scraping speed is much higher than downloading speed of the predefined ones.
                                        So all in all that would really be great.

                                        kiroK 1 Reply Last reply Reply Quote 0
                                        • FollyF Folly referenced this topic on
                                        • kiroK
                                          kiro @Folly
                                          last edited by kiro

                                          Hi @Folly! Finally all roms from mame-sl are in the backend and I've added the pictures found in your gamelists where missing, phew... hope this helps!

                                          Supported systems with figures here: http://77.68.23.83/

                                          FollyF 2 Replies Last reply Reply Quote 1
                                          • FollyF
                                            Folly @kiro
                                            last edited by

                                            @kiro

                                            Very cool that you managed all of this.
                                            I can only imagine how much work it must be.
                                            Thumbs up ^^ :-)

                                            I will do some scraping when I can ;-)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.