RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

    Scheduled Pinned Locked Moved Projects and Themes
    scrapescraperuxs
    728 Posts 111 Posters 749.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • FlyingTomahawkF
      FlyingTomahawk @Dochartaigh
      last edited by FlyingTomahawk

      @Dochartaigh

      I personally installed EmulationStation on my PC and created the exact same folder structure as on my RPi.
      All the roms are the same on my PC as they are on my RPi.
      This is not necessary but it makes things a bit easier to keep overview later.
      I then define the paths for roms, images and gamelists under Configuration -> Path configuration.
      I scrape one system after the other so I redefine the paths every time for each system before scrapping.

      You can create any folder structure and define the paths for roms, images and gamelist to whatever you like and helps you to keep overview.

      1 Reply Last reply Reply Quote 1
      • P
        paradadf @Dochartaigh
        last edited by

        @Dochartaigh isn't there a profil called "roms path" or something? IMO that's the best location for the gamelist and images folder, next to your roms. That's also the location of the data when you use sselph scraper.

        1 Reply Last reply Reply Quote 0
        • screechS
          screech
          last edited by

          Hi guys, sorry, I were a little bit busy these last time... Not so much Time on UXS...

          @Jeck11 : It's unfortunatly false positive. The language I use (Autoit) is mainly detected as malicious. I had contactated many AV editor to say it's a false positive. Lot's of them have answer and it's ok... But some haven't answer (yet ?) like Microsoft :S .

          @Dukafleed Yep, it's in a very long todo list ^^ you can try : https://github.com/Universal-Rom-Tools/Universal-XML-Editor but it's "dedicated" to recalbox, maybe it works on retropie (but it's an early Beta ^^)

          @BORIStheBLADE1 and others, sorry for the "lost connection" to the DB... When server is very "high" on CPU Use, it temporary "cut" the access to none register user... Try to Register to Screenscraper ;) maybe it can help .

          @hooperre You are right... Must add a save function to the original XML.
          For the Donkey Kong File. I think there is a probleme with the ' in the name... Need to check that...

          @FlyingTomahawk near 7h for 820 roms is pretty...long... Normally it can handle much more roms (someone scrape more than 6000 Amiga roms with success ;) ) Try to register and submit some proposition on www.screenscraper.fr to get some extra thread ;) it will be faster ;)

          @Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)
          (FYI : Thread are simultaneous scrape engine number... The main EXE send to X Engine the API return to manage it and download picture, create the mix, generate a part of the final XML. So normally, except if the server is very loaded, it must go pretty fast...)

          @Paradadf thank you very much taking the UXS support when I'm not here ^^. And you are right... Prefere the "Roms path" profile when you scrape on you PC... It put the downloaded_image folder and gamelist.xml directly in the roms folder ;)

          E D 2 Replies Last reply Reply Quote 1
          • E
            EctoOne @screech
            last edited by

            @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

            @hooperre You are right... Must add a save function to the original XML.
            For the Donkey Kong File. I think there is a probleme with the ' in the name... Need to check that...

            I noticed that last night too. Tried to scrape Chip's Challenge and whatever i tried to name it, it couldn't find it. Also there was some problem with names that have multiple . in them. Like B.A.T. or P. P. Hammer.. I got those working by accidentally renaming them to NAME.ipf.zip

            1 Reply Last reply Reply Quote 0
            • screechS
              screech
              last edited by

              Normally, the "good" process is to identify your rom with CRC... Filename is just a "fallback" feature to scrape as much as it can ;) And so it's not very "reliable"...

              E 1 Reply Last reply Reply Quote 0
              • FlyingTomahawkF
                FlyingTomahawk
                last edited by FlyingTomahawk

                @screech
                Thanks for your reply.
                Today it was faster, lot faster than yesterday. I have an account at screenscraper.fr but I haven't contributed ($$) anything yet.
                Is it possible to run the scrape again but let it only scrape the roms from the missing list?

                1 Reply Last reply Reply Quote 0
                • screechS
                  screech
                  last edited by screech

                  Very nice ;)
                  FYI : $$ donation aren't necessary ;) just contribut to the DB is enough to grant you several Thread ;)
                  (We are missing lot's of sinopsys or media you can submit ;) )

                  And of course you can just "add" the missing rom when you crape ;) in General menu, look at the scrape mode and select "update" ;)

                  1 Reply Last reply Reply Quote 1
                  • E
                    EctoOne @screech
                    last edited by

                    @screech yeah i know, but I was scraping custom amiga uae files, so no hashes for me :)

                    1 Reply Last reply Reply Quote 0
                    • screechS
                      screech
                      last edited by

                      @EctoOne try this :
                      https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/Rom-Missing-on-Screenscraper

                      Pass your uae file in F-CRC and send them to me ;) if they are well named, they will be autoassociated to there legitimate game ;) so you can scrape directly with CRC ;)

                      E 1 Reply Last reply Reply Quote 0
                      • hooperreH
                        hooperre
                        last edited by

                        @screech Thanks for this man. It's an amazing tool and you've enhanced my RetroPie experience greatly. So thank you.

                        Would creating a file directly from a URL to the screenscraper.fr database be difficult to incorporate for ROMs that don't pull easily? Sorry if this is a... novice question.

                        4B ~ RPi PSU 5.1V / 3.0A ~ 32GB SanDisk microSD ~ 128GB USB

                        1 Reply Last reply Reply Quote 0
                        • screechS
                          screech
                          last edited by

                          @hooperre sorry, my english must be too bad ^^ I don't understand your question :p

                          hooperreH 1 Reply Last reply Reply Quote 0
                          • hooperreH
                            hooperre @screech
                            last edited by hooperre

                            @screech I was wondering if you could add something that will make a MIX image simply by pasting a link to the ROM in the DB.

                            For example, for Donkey Kong 3, I would just paste "https://www.screenscraper.fr/gameinfos.php?plateforme=4&gameid=2181" into the XML Scraper and it would output the MIX image for that game.

                            4B ~ RPi PSU 5.1V / 3.0A ~ 32GB SanDisk microSD ~ 128GB USB

                            1 Reply Last reply Reply Quote 0
                            • screechS
                              screech
                              last edited by

                              unfortunatly, it's actually impossible...

                              To create a MIX, I need to catch the Media.
                              To Catch the Media, I need to have an API response from the DB.
                              To have an API response from the DB, I need to ask the API with a CRC32, MD5, SHA1, or Filename...

                              I can't with the Game ID...

                              But I can ask the Screenscraper boss to add this to the API ;) but I think it's not for tomorrow ;) (he is pretty busy right now ^^)

                              T 1 Reply Last reply Reply Quote 0
                              • E
                                EctoOne @screech
                                last edited by

                                @screech I don't think that would help in this case. UAE are just text config files and even with the automatic ways to create them i think they would be different from system to system. I mean it uses a template to create them and after that it already makes automatic changes to the controller name when you start them.. It was easier to batch rename them.

                                1 Reply Last reply Reply Quote 0
                                • screechS
                                  screech
                                  last edited by

                                  Arg.. ok ^^ CRC on this is useless ^^ (not very in Amiga ^^)

                                  1 Reply Last reply Reply Quote 0
                                  • E
                                    EctoOne
                                    last edited by

                                    Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.

                                    1 Reply Last reply Reply Quote 1
                                    • T
                                      tmak @screech
                                      last edited by

                                      @screech

                                      Hi Screech,

                                      I am new tot he retropie community and love the way the universal xml 3 image art looks. My only problem is I have a MAC computer, and i can use the universal xml scraper. Do you have any suggestions on another way to quickly scrap this great image art to my retropie?

                                      Thanks,

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        Dochartaigh @screech
                                        last edited by Dochartaigh

                                        @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                                        @Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)

                                        So I gave up on NES, trying Genesis now. 1,707 ROMs from the No-Intro romset (totally standard). Turned CRC off and just going by filename to speed things up. AND it's reading directly off my hard drive. AND I joined the screenscraper site and logged in on the program. AND I contributed unique content (still only have 1 thread). AND the scraper site's server load is only 52% right now...

                                        ...AND it's still going to take me 14 hours to scrape Sega Genesis/Mega Drive....

                                        Sorry, but if this entire program hinged on a slow flaky website (can't register from a Mac running Google Chrome either FYI...have to use Safari) to scape from it's completely flawed I'm sorry to say.

                                        P.S. AND the program just crashed like 3 times in a row... lol

                                        FlyingTomahawkF P 2 Replies Last reply Reply Quote 0
                                        • FlyingTomahawkF
                                          FlyingTomahawk @Dochartaigh
                                          last edited by

                                          @Dochartaigh

                                          Many ANDs in there, lol
                                          Just start it before going to bed and next day all is done. The time will change. Yesterday when I was scraping my NES roms (750+) the time constantly changed and got shorter and shorter.

                                          1 Reply Last reply Reply Quote 1
                                          • P
                                            paradadf @Dochartaigh
                                            last edited by

                                            @Dochartaigh usung a filename isn't faster than the crc for small files.
                                            I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
                                            It can also be that your firewall is causing problems, don't know.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.