Navigation

    RetroPie forum home
    • Register
    • Login
    • Search
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

    [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

    Projects and Themes
    scrape scraper uxs
    111
    728
    336031
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • screech
      screech last edited by screech

      Hi,

      I'm waiting the V2 is near stable to introduce it to the retropie comunity. And it still in Beta, but I think really near to be in stable ;)
      So I need Retropie user to test it ;)

      This is the presentation :

      alt text

      Universal XML Scraper V2 is an easy to use and configure scraper.
      It works in English, French, German, Spanish, Portugaise (and all other language you can translate it if you want ;) )
      It's work only on windows (sorry) but maybe you can try to test it on WINE.
      It's open source ( The Github )
      And based on Screenscraper Database, I really think it's the best DB you can found right now... It's just 1 year old but better (in quality and content) than any other DB I found... And it's just the begining ;)

      What you can do with Universal XML Scraper (UXS for the friend ;) ) :

      Easy configuration :
      Wizard and configuration menu are easy to handle.
      And the great things is the "Autoconfiguration", select your profil depending on the Systeme you use (Retropie or Recalbox for now... but it's really open) Then choose your "root" directory where all your Roms folder are... Then you just hace to choose the directory you want to scrape all is autoconfigurated.

      alt text

      Evolutive :
      All the configuration are in XML File... So if you are not pleased with the "pre configured" things you can do what you want.
      And so it's open to lot's configuration without touching the code.

      Lot's of language :
      I already say that the software can handle lot's of language... But do you know the DB too ?
      If you want your synopsis in Spanish, deutsh, French, portugaise, and of course english, it's possible.
      Depending on your language selection for UXS, the data you scrape will match.
      (And it's possible in menu to change that).
      It of course have a fallback function... so if no synopsis in deutsh is found, it will take the English one (for exemple)

      Fast :
      The default configuration use only one thread (roms were scrapped one by one). But if you want to help us to filling the hole in the Screenscraper Database, you will be granted more thread...
      with only one participation, you already have 2 threads... And scrape 2X faster ;)
      Don't forget to put your Screenscraper information in the general menu to check how many Threads you can have.

      alt text

      Acurate :
      The scrape is done by 2 way : your romfile hash and if not found your romfile name.
      it's not taking the Game name, only the rom name to match a rom in the Screenscraper Database.
      Why ? Because, rom are link to a country. And media and some informations are also link to the rom country.
      So when you scrape a Japan rom, you will have the Japan Box and the Japan Name of the game.
      If you scrape a US one ;) it's the same, you will have media and information corresponding.
      And no mistake can happen with 2 games with "near" the same rom name... Because Hash aren't the same ;)

      alt text
      alt text

      Nice :
      Emulationstation can handle only one dynamique picture... So you have to choose : Screenshot ? 3D Box ? 2D Box ?....
      With the MIX profil, you can "create" very nice picture to make your front end beautiful ...

      Some MIX exemple :

      alt text
      alt text

      With the appropriate theme, you can have full screen dynamique picture :

      alt text

      Teasing : A new MIX template with Emulationstation theme in preparation :

      alt text

      You can found the last version here : Github

      don't hesitate to test, and tell me if all is alright ;)

      Rion 1 Reply Last reply Reply Quote 11
      • Rion
        Rion @screech last edited by

        @screech I can highly recommend this to anyone. Been using it for quite some time now and it's working great!

        Screenscraper.fr is also a great community with friendly and helpful people.

        Don't hesitate to join the site and contribute to the database!

        1 Reply Last reply Reply Quote 1
        • screech
          screech last edited by

          2.0.0.6 Beta is out...

          Lot's of change :

          • Corrected :
          • French, German, spanish and english translation correction.
          • First rom sometime doesn't be scrapped corrected.
          • Include/Exclude label size corrected
          • Registering link with Digimirror corrected.
          • Some roms don't be scraped because of "not allowed" characteres corrected.
          • Added :
          • New XML Include
          • Bypass research function (to make dedicated profil for some system like scummvm)
          • New Mix Profil to Test SCUMMVM on Recalbox V4 (MIX)
          • Rewrite of path option to add some tag :
            • %SystemDir% = Full path to the system directory
            • %System% = Name of the System directory
            • %Source_RootPath% = Path to the local Root
          • Rewrite of Retropie Profil to match Retropie Path (to test)
          • Merge with verybadsoldier fork :
          • Added a new function to scrape IN zip files when zip file rom isn't found. (THANKS A LOT verybadsoldier)
          sonicsheppard 1 Reply Last reply Reply Quote 1
          • sonicsheppard
            sonicsheppard last edited by

            Loving this! Thank you!

            1 Reply Last reply Reply Quote 1
            • vbs
              vbs last edited by

              Big thanks for the scraper! I just discovered it some days ago and I like it alot!

              1 Reply Last reply Reply Quote 1
              • windale
                windale last edited by windale

                I just did a scrape of my MAME collection and the end result left a "Mame_missing" text file saying it could not find 6 roms. I just want the gamelist to link these last 6 roms to a screenshot. I checked the gamelist and theres no sign of them. So even if you have the roms (and screenshots) it doesn't add an entry to the gamelist and they won't show up after scraping. People will be seeing a lot less roms in Retropie than they actually have.

                So to summarize, if the rom is not found in your database can we have an option to add it to the gamelist so it will show up and display the screenshot (that we manually added) ? (Steven Selph's scraper has an option to "add not found" ).

                I tried the option "Add missing roms in XML" but i'm not sure what the option does and it didn't help.

                paradadf 1 Reply Last reply Reply Quote 0
                • paradadf
                  paradadf @windale last edited by

                  @windale that option should do exactly what sselph's scraper does. But even without selecting it, ES should be able, as long a the rom exists, to show the game.

                  1 Reply Last reply Reply Quote 0
                  • sonicsheppard
                    sonicsheppard @screech last edited by

                    @screech

                    I've had a far higher success rate at scraping roms using the new .6 beta, however I have noticed it is now having trouble with roms containing apostrophes ('). An example would be Crack'Ed on Atari7800.

                    This wasn't an issue on the .5 beta.

                    If you need any more info please let me know.

                    Thanks so much!

                    1 Reply Last reply Reply Quote 0
                    • screech
                      screech last edited by

                      @windale I have an issue open on the "Add missing roms in XML" doesn't work well.. I need to check... But like @paradadf says normally even if the rom isn't scraped, it will show in the ES Game list (as a non scraped one)

                      @sonicsheppard Strange, I add a "correction" normally on the rom with a ' in the name and it works on my try... Need to test it again :S sorry...

                      windale 1 Reply Last reply Reply Quote 0
                      • windale
                        windale @screech last edited by windale

                        @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                        @windale I have an issue open on the "Add missing roms in XML" doesn't work well.. I need to check... But like @paradadf says normally even if the rom isn't scraped, it will show in the ES Game list (as a non scraped one)

                        Yes, but I like to have the option "parse gamelists only" on in Retropie so that it doesn't show the non-scraped Bios files and multi disk games etc. I manually added the 6 not found to my gamelist. I'll wait to see if the option "Add missing roms in XML" gets fixed.

                        1 Reply Last reply Reply Quote 0
                        • screech
                          screech last edited by

                          @windale I'm on it ;)

                          1 Reply Last reply Reply Quote 1
                          • screech
                            screech last edited by

                            I just release what I think is the first stable version ;)

                            some improvement for Retropie on it ;)

                            (autoconfiguration path are normalized with Reptropie Path and Scraped extension file are restricted to the Retropie used extension)

                            @windale the Add missings roms in XML is corrected ;)
                            @sonicsheppard the ' problem is normmaly resolved ;)

                            UXS V2 Stable here

                            sonicsheppard 1 Reply Last reply Reply Quote 2
                            • sonicsheppard
                              sonicsheppard @screech last edited by

                              @screech

                              Hi, happy to report the issue about roms with apostrophes being missed is verified fixed in the latest version, as is the fixed pathing for RetroPie.

                              Couple of questions though, in this new V2 Stable version I've noticed that the rom titles in the gamelist.xml now contain the region, e.g. Aliens vs Predator (USA). This wasn't happening in the previous versions, is there an option to disable that? Also, for roms with a World region, they are being displayed as the French (Monde).

                              Thanks for sorting this though, it's coming along great! If you need any more info just let me know.

                              Cheers!

                              vbs windale 2 Replies Last reply Reply Quote 0
                              • vbs
                                vbs @sonicsheppard last edited by

                                @sonicsheppard
                                You can configure in UXS how you want your game names composed. I currently cannot check for the exact menu though. I think its in "options".

                                sonicsheppard 1 Reply Last reply Reply Quote 1
                                • sonicsheppard
                                  sonicsheppard @vbs last edited by

                                  @vbs

                                  Wicked, found it in Configuration -> Options configuration -> Item Choice/Name Format.

                                  Works a treat. Thanks!

                                  1 Reply Last reply Reply Quote 0
                                  • windale
                                    windale @sonicsheppard last edited by windale

                                    @sonicsheppard said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                                    @screech
                                    Also, for roms with a World region, they are being displayed as the French (Monde).

                                    Yeah, I reported this earlier today (before this version was released). 'Japan' was also misspelt as 'Japon', and 'Platform' was 'Plateform'. I think they said that 'Plateform' was fixed. Can't check at the moment.

                                    1 Reply Last reply Reply Quote 0
                                    • screech
                                      screech last edited by

                                      Hello all,

                                      I release some new version, with little correction ;) -> 2.0.0.3

                                      For the wrong english translation, it's due to the DB.

                                      Screenscraper (at start) was made for french people :S with more and more people we add (quickly) other language... But it's not so well added ;)

                                      So a Screenscraper V2 is under developpement to manage every language ;) (for data AND media ;) )

                                      Be patient it will come ;)

                                      vbs 1 Reply Last reply Reply Quote 3
                                      • vbs
                                        vbs @screech last edited by

                                        @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                                        Screenscraper (at start) was made for french people :S with more and more people we add (quickly) other language... But it's not so well added ;)

                                        So a Screenscraper V2 is under developpement to manage every language ;) (for data AND media ;) )

                                        Be patient it will come ;)

                                        Thats really cool to hear! Will the forum be in english too?

                                        1 Reply Last reply Reply Quote 0
                                        • screech
                                          screech last edited by

                                          I will ask ^^

                                          It's obvious it's not easy for foreign user to speak with us ^^

                                          But if you have problem, don't hesitate to visit us on our IRC Channel, we will be please to answer any question you have ;)

                                          https://kiwiirc.com/client/irc.freenode.net/?#recalbox

                                          Squilenator 1 Reply Last reply Reply Quote 0
                                          • Dochartaigh
                                            Dochartaigh last edited by Dochartaigh

                                            This looks great, but I'm afraid I'll have to wait until this can be run on the Pi itself, and not a 2nd system which has network drive access to the PI. (plus I'm on a Mac so I feel leftout....and Wine has never worked well for me in the past :(

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.