• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

Scheduled Pinned Locked Moved Projects and Themes
scrapescraperuxs
728 Posts 111 Posters 740.0k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    sonicsheppard
    last edited by 22 Nov 2016, 18:37

    Loving this! Thank you!

    1 Reply Last reply Reply Quote 1
    • V
      vbs
      last edited by 22 Nov 2016, 20:34

      Big thanks for the scraper! I just discovered it some days ago and I like it alot!

      1 Reply Last reply Reply Quote 1
      • W
        windale
        last edited by windale 23 Nov 2016, 13:01

        I just did a scrape of my MAME collection and the end result left a "Mame_missing" text file saying it could not find 6 roms. I just want the gamelist to link these last 6 roms to a screenshot. I checked the gamelist and theres no sign of them. So even if you have the roms (and screenshots) it doesn't add an entry to the gamelist and they won't show up after scraping. People will be seeing a lot less roms in Retropie than they actually have.

        So to summarize, if the rom is not found in your database can we have an option to add it to the gamelist so it will show up and display the screenshot (that we manually added) ? (Steven Selph's scraper has an option to "add not found" ).

        I tried the option "Add missing roms in XML" but i'm not sure what the option does and it didn't help.

        P 1 Reply Last reply 23 Nov 2016, 13:14 Reply Quote 0
        • P
          paradadf @windale
          last edited by 23 Nov 2016, 13:14

          @windale that option should do exactly what sselph's scraper does. But even without selecting it, ES should be able, as long a the rom exists, to show the game.

          1 Reply Last reply Reply Quote 0
          • S
            sonicsheppard @screech
            last edited by 23 Nov 2016, 13:24

            @screech

            I've had a far higher success rate at scraping roms using the new .6 beta, however I have noticed it is now having trouble with roms containing apostrophes ('). An example would be Crack'Ed on Atari7800.

            This wasn't an issue on the .5 beta.

            If you need any more info please let me know.

            Thanks so much!

            1 Reply Last reply Reply Quote 0
            • S
              screech
              last edited by 23 Nov 2016, 13:27

              @windale I have an issue open on the "Add missing roms in XML" doesn't work well.. I need to check... But like @paradadf says normally even if the rom isn't scraped, it will show in the ES Game list (as a non scraped one)

              @sonicsheppard Strange, I add a "correction" normally on the rom with a ' in the name and it works on my try... Need to test it again :S sorry...

              W 1 Reply Last reply 23 Nov 2016, 13:50 Reply Quote 0
              • W
                windale @screech
                last edited by windale 23 Nov 2016, 13:50

                @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                @windale I have an issue open on the "Add missing roms in XML" doesn't work well.. I need to check... But like @paradadf says normally even if the rom isn't scraped, it will show in the ES Game list (as a non scraped one)

                Yes, but I like to have the option "parse gamelists only" on in Retropie so that it doesn't show the non-scraped Bios files and multi disk games etc. I manually added the 6 not found to my gamelist. I'll wait to see if the option "Add missing roms in XML" gets fixed.

                1 Reply Last reply Reply Quote 0
                • S
                  screech
                  last edited by 23 Nov 2016, 13:56

                  @windale I'm on it ;)

                  1 Reply Last reply Reply Quote 1
                  • S
                    screech
                    last edited by 23 Nov 2016, 20:56

                    I just release what I think is the first stable version ;)

                    some improvement for Retropie on it ;)

                    (autoconfiguration path are normalized with Reptropie Path and Scraped extension file are restricted to the Retropie used extension)

                    @windale the Add missings roms in XML is corrected ;)
                    @sonicsheppard the ' problem is normmaly resolved ;)

                    UXS V2 Stable here

                    S 1 Reply Last reply 23 Nov 2016, 22:56 Reply Quote 2
                    • S
                      sonicsheppard @screech
                      last edited by 23 Nov 2016, 22:56

                      @screech

                      Hi, happy to report the issue about roms with apostrophes being missed is verified fixed in the latest version, as is the fixed pathing for RetroPie.

                      Couple of questions though, in this new V2 Stable version I've noticed that the rom titles in the gamelist.xml now contain the region, e.g. Aliens vs Predator (USA). This wasn't happening in the previous versions, is there an option to disable that? Also, for roms with a World region, they are being displayed as the French (Monde).

                      Thanks for sorting this though, it's coming along great! If you need any more info just let me know.

                      Cheers!

                      V W 2 Replies Last reply 23 Nov 2016, 22:59 Reply Quote 0
                      • V
                        vbs @sonicsheppard
                        last edited by 23 Nov 2016, 22:59

                        @sonicsheppard
                        You can configure in UXS how you want your game names composed. I currently cannot check for the exact menu though. I think its in "options".

                        S 1 Reply Last reply 23 Nov 2016, 23:18 Reply Quote 1
                        • S
                          sonicsheppard @vbs
                          last edited by 23 Nov 2016, 23:18

                          @vbs

                          Wicked, found it in Configuration -> Options configuration -> Item Choice/Name Format.

                          Works a treat. Thanks!

                          1 Reply Last reply Reply Quote 0
                          • W
                            windale @sonicsheppard
                            last edited by windale 23 Nov 2016, 23:19

                            @sonicsheppard said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                            @screech
                            Also, for roms with a World region, they are being displayed as the French (Monde).

                            Yeah, I reported this earlier today (before this version was released). 'Japan' was also misspelt as 'Japon', and 'Platform' was 'Plateform'. I think they said that 'Plateform' was fixed. Can't check at the moment.

                            1 Reply Last reply Reply Quote 0
                            • S
                              screech
                              last edited by 25 Nov 2016, 15:21

                              Hello all,

                              I release some new version, with little correction ;) -> 2.0.0.3

                              For the wrong english translation, it's due to the DB.

                              Screenscraper (at start) was made for french people :S with more and more people we add (quickly) other language... But it's not so well added ;)

                              So a Screenscraper V2 is under developpement to manage every language ;) (for data AND media ;) )

                              Be patient it will come ;)

                              V 1 Reply Last reply 25 Nov 2016, 15:24 Reply Quote 3
                              • V
                                vbs @screech
                                last edited by 25 Nov 2016, 15:24

                                @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                                Screenscraper (at start) was made for french people :S with more and more people we add (quickly) other language... But it's not so well added ;)

                                So a Screenscraper V2 is under developpement to manage every language ;) (for data AND media ;) )

                                Be patient it will come ;)

                                Thats really cool to hear! Will the forum be in english too?

                                1 Reply Last reply Reply Quote 0
                                • S
                                  screech
                                  last edited by 25 Nov 2016, 15:27

                                  I will ask ^^

                                  It's obvious it's not easy for foreign user to speak with us ^^

                                  But if you have problem, don't hesitate to visit us on our IRC Channel, we will be please to answer any question you have ;)

                                  https://kiwiirc.com/client/irc.freenode.net/?#recalbox

                                  S 1 Reply Last reply 6 Feb 2017, 18:29 Reply Quote 0
                                  • D
                                    Dochartaigh
                                    last edited by Dochartaigh 25 Nov 2016, 18:32

                                    This looks great, but I'm afraid I'll have to wait until this can be run on the Pi itself, and not a 2nd system which has network drive access to the PI. (plus I'm on a Mac so I feel leftout....and Wine has never worked well for me in the past :(

                                    1 Reply Last reply Reply Quote 0
                                    • V
                                      vbs
                                      last edited by 25 Nov 2016, 23:59

                                      Has anyone an idea how to scrape MAME sets with parents and clones so that you get sensible game titles in the end?

                                      Lets say the files are named:

                                      airbustrj.zip
                                      airbustrb.zip
                                      airbustr.zip
                                      

                                      Which should map to something like "Air Buster (Japan)", "Air Buster (bootleg)" and "Air Buster (World)". But what whatever filename option I use in UXS I either get the plain filename (cryptic) or for all three the same just "Air Buster".

                                      R 1 Reply Last reply 26 Nov 2016, 00:37 Reply Quote 0
                                      • R
                                        Rion @vbs
                                        last edited by 26 Nov 2016, 00:37

                                        @vbs I have been talking about this with @screech on the official irc channel for screenscraper.fr. There have been talks about implementation of Arcade Italia's datbase. What i last heard is that it's planned for v2 of the website.

                                        Mamedb is down as we all know and have/had not been updated as of Mame 0.15X

                                        Right now when you scrape games for Mame/fba you get the same names for the parrent rom and the clones with some few deviation from roms that have (World) or (Japan) in them.

                                        Hopefully this will see the light of day soon.

                                        FBNeo rom filtering
                                        Mame2003 Arcade Bezels
                                        Fba Arcade Bezels
                                        Fba NeoGeo Bezels

                                        1 Reply Last reply Reply Quote 0
                                        • H
                                          hansolo77
                                          last edited by 26 Nov 2016, 02:17

                                          I hate to drudge up a possible issue since I'm a new user to this. I was not aware a tool like this existed prior to it being mentioned here:

                                          https://retropie.org.uk/forum/topic/5498/scraper-by-sselph-can-t-scrape-chosen-systems/9

                                          I have SAMBA shares enabled on my Pi, connected to the same network. I can browse to the Pi using the \\RetroPie\ address, and can see the various folders (bios, configs, roms, splashscreens). I can also SFTP to the Pi and can browse to the real location of those files (/home/pi/retropie/).

                                          Before I left for work today, I configured the Scraper to use the RetroPie configuration, and went through the process of telling it to use the 4-way MIX view (I like that, the other scrapers just get cover art!). I then started the process off. The first system it started scraping for me was Atari 2600, which is correct. But I waited and watched as it does it's thing. It appeared to be doing something, but I checked the location the files should be saved at via SFTP and didn't find them. I figured it was because it was still scraping, and maybe the program grabs everything FIRST, then uploads them to the Pi rather than each file ASAP. So I went off to work, hoping it would be done scraping everything when I get home. Well, I'm home now and it's still scraping. I SFTP'd again to see if it had sent any images or anything to the Pi, and it hasn't. If I browse to the path /home/pi/.emulationstation (which actually goes to /opt/retropie/configs/all/emulationstation) I see the 2 folders "downloaded_images" and "gamelists", but both folders are empty. This made me think "Do I need to register and PAY to use this program or something?"

                                          Further checking in the logs shows it is indeed sending stuff to the Pi, but I can't find it. According to the logs, there is this line:

                                          [11:05:09] - > $vTarget_ImagePath = /home/pi/.emulationstation/downloaded_images/atari2600/

                                          When I SFTP to that folder, nothing is there. It is currently scraping for NES now, so the Atari sets should be done. Does it not send the images/etc to the Pi until the ENTIRE process is completed?


                                          EDIT (APPEND)

                                          Ok, so I just checked a bit further. I noticed also in the log that it says:

                                          [11:05:09] - > $vSource_ImagePath = \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600

                                          So to me, this means it's checking first to see if images exist somewhere in the \roms path. That can't be right. I don't have any of my other scrapers configured to store images and gamelists next to the roms. Nevertheless, I checked that path, and there still isn't any artwork in /home/pi/retropie/roms/atari2600, just an added text file about what's missing.

                                          Interestingly, there is a NEW folder in the /home/pi/retropie/roms path called "configs" which wasn't there before. When I go in there, there are a few more new nested folders... so now there is a path line going clear to /home/pi/retropie/roms/configs/all/emulationstation. Inside there, I see 2 folders "downloaded_images" and "gamelists". Ah-hah... checking THOSE folders does indeed reveal the uploaded artwork.

                                          Why in the world is this program scraping artwork into that path? That's not the right path at all. Once this is all said and done, I'm pretty sure EmulationStation isn't going to be able to find those folders to display them. What a complete waste of time! Am I correct in seeing all this? As I said, I'm still a new user, just downloaded and installed the latest version from the github today. Specifically, it was THIS version. Should I be using a different version, or is this all correct in it's function?

                                          Who's Scruffy Looking?

                                          V 1 Reply Last reply 26 Nov 2016, 09:21 Reply Quote 0
                                          23 out of 728
                                          • First post
                                            23/728
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received