RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

    Scheduled Pinned Locked Moved Projects and Themes
    scrapescraperuxs
    728 Posts 111 Posters 747.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • hansolo77H
      hansolo77
      last edited by

      I hate to drudge up a possible issue since I'm a new user to this. I was not aware a tool like this existed prior to it being mentioned here:

      https://retropie.org.uk/forum/topic/5498/scraper-by-sselph-can-t-scrape-chosen-systems/9

      I have SAMBA shares enabled on my Pi, connected to the same network. I can browse to the Pi using the \\RetroPie\ address, and can see the various folders (bios, configs, roms, splashscreens). I can also SFTP to the Pi and can browse to the real location of those files (/home/pi/retropie/).

      Before I left for work today, I configured the Scraper to use the RetroPie configuration, and went through the process of telling it to use the 4-way MIX view (I like that, the other scrapers just get cover art!). I then started the process off. The first system it started scraping for me was Atari 2600, which is correct. But I waited and watched as it does it's thing. It appeared to be doing something, but I checked the location the files should be saved at via SFTP and didn't find them. I figured it was because it was still scraping, and maybe the program grabs everything FIRST, then uploads them to the Pi rather than each file ASAP. So I went off to work, hoping it would be done scraping everything when I get home. Well, I'm home now and it's still scraping. I SFTP'd again to see if it had sent any images or anything to the Pi, and it hasn't. If I browse to the path /home/pi/.emulationstation (which actually goes to /opt/retropie/configs/all/emulationstation) I see the 2 folders "downloaded_images" and "gamelists", but both folders are empty. This made me think "Do I need to register and PAY to use this program or something?"

      Further checking in the logs shows it is indeed sending stuff to the Pi, but I can't find it. According to the logs, there is this line:

      [11:05:09] - > $vTarget_ImagePath = /home/pi/.emulationstation/downloaded_images/atari2600/

      When I SFTP to that folder, nothing is there. It is currently scraping for NES now, so the Atari sets should be done. Does it not send the images/etc to the Pi until the ENTIRE process is completed?


      EDIT (APPEND)

      Ok, so I just checked a bit further. I noticed also in the log that it says:

      [11:05:09] - > $vSource_ImagePath = \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600

      So to me, this means it's checking first to see if images exist somewhere in the \roms path. That can't be right. I don't have any of my other scrapers configured to store images and gamelists next to the roms. Nevertheless, I checked that path, and there still isn't any artwork in /home/pi/retropie/roms/atari2600, just an added text file about what's missing.

      Interestingly, there is a NEW folder in the /home/pi/retropie/roms path called "configs" which wasn't there before. When I go in there, there are a few more new nested folders... so now there is a path line going clear to /home/pi/retropie/roms/configs/all/emulationstation. Inside there, I see 2 folders "downloaded_images" and "gamelists". Ah-hah... checking THOSE folders does indeed reveal the uploaded artwork.

      Why in the world is this program scraping artwork into that path? That's not the right path at all. Once this is all said and done, I'm pretty sure EmulationStation isn't going to be able to find those folders to display them. What a complete waste of time! Am I correct in seeing all this? As I said, I'm still a new user, just downloaded and installed the latest version from the github today. Specifically, it was THIS version. Should I be using a different version, or is this all correct in it's function?

      Who's Scruffy Looking?

      vbsV 1 Reply Last reply Reply Quote 0
      • vbsV
        vbs @hansolo77
        last edited by

        @hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

        [11:05:09] - > $vSource_ImagePath = \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600

        So to me, this means it's checking first to see if images exist somewhere in the \roms path. That can't be right. I don't have any of my other scrapers configured to store images and gamelists next to the roms. Nevertheless, I checked that path, and there still isn't any artwork in /home/pi/retropie/roms/atari2600, just an added text file about what's missing.

        Interestingly, there is a NEW folder in the /home/pi/retropie/roms path called "configs" which wasn't there before. When I go in there, there are a few more new nested folders... so now there is a path line going clear to /home/pi/retropie/roms/configs/all/emulationstation. Inside there, I see 2 folders "downloaded_images" and "gamelists". Ah-hah... checking THOSE folders does indeed reveal the uploaded artwork.

        The ..means to go back one directory, so \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600would result in \\retropie\configs\all\emulationstation\downloaded_images\atari2600. But I also stumbled about it and I think it does not work as intended. I made a ticket about my problem here:
        https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/issues/51

        For me it worked after I replaced in XML file name and Folder Name where pictures will be downloaded (Autoconfig) the %Source_RootPath%\..\configsdirectly with \\mypi\configs.

        But this path still did not get used after changing it and it sticked to the old path when scraping. I found the old path also in UXS-config.ini. After fixing it there manually too it worked for me...

        1 Reply Last reply Reply Quote 0
        • screechS
          screech
          last edited by screech

          sorry to answer so late...

          This is how it work :

          In Profil XML you have the "autoconfiguration path" standard for retropie :

          	<AutoConf>
          		<Source_RootPath>\\retropie\roms</Source_RootPath>
          		<Target_XMLName>%Source_RootPath%\..\configs\all\emulationstation\gamelists\%System%\gamelist.xml</Target_XMLName>
          		<Target_RomPath>/home/pi/RetroPie/roms/%system%/</Target_RomPath>
          		<Source_ImagePath>%Source_RootPath%\..\configs\all\emulationstation\downloaded_images\%System%</Source_ImagePath>
          		<Target_ImagePath>/home/pi/.emulationstation/downloaded_images/%system%/</Target_ImagePath>
          	</AutoConf>
          

          Explaination :
          Source_RootPath is where UXS Autoconfig look for system folder
          Target_XMLName is where UXS put the final gamelist.xml
          Target_RomPath is where emulationstation will found the rom (UXS put this path in gamelist.xml)
          Source_ImagePath is where UXS will download picture (mix or not)
          Target_ImagePath is where emulationstation will found the picture (UXS put this in gamelist.xml)

          I added some "tag" that will be replace automatically :
          %Source_RootPath% will be replaced by the path in <Source_RootPath>
          %system% will be replaced by the folder of the system you selected in autoconf
          %SystemDir% will be a merge of %Source_RootPath%+%system%

          Autoconf will replace path and put it in UXS-config.ini, when you select a system in the menu "scrape/select a system"
          And the scrape use path in UXS-config.ini

          So now if we examin the standard path :
          Source_RootPath = \retropie\roms
          Target_XMLName = %Source_RootPath%..\configs\all\emulationstation\gamelists\%System%\gamelist.xml
          -> If we select nes as system :
          \retropie\roms\..\configs\all\emulationstation\gamelists\nes\gamelist.xml (in UXS-config.ini)
          So :
          \retropie\configs\all\emulationstation\gamelists\nes\gamelist.xml (is the reel path /../ meaning 1 step behind)
          Target_RomPath = /home/pi/RetroPie/roms/%system%/
          -> If we select nes as system : /home/pi/RetroPie/roms/nes/ (in UXS-config.ini)
          Source_ImagePath = %Source_RootPath%\..\configs\all\emulationstation\downloaded_images\%System%
          -> If we select nes as system :
          \retropie\roms..\configs\all\emulationstation\downloaded_images\nes (in UXS-config.ini)
          So :
          \retropie\configs\all\emulationstation\downloaded_images\nes (is the reel path /../ meaning 1 step behind)
          Target_ImagePath = /home/pi/.emulationstation/downloaded_images/%system%/
          -> If we select nes has system : /home/pi/.emulationstation/downloaded_images/nes/ (in UXS-config.ini)

          It's a little bit complexe... Sorry, but it's needed to "respect Retropie Path" and keep a simple way to configure path with changing only the Source_RootPath.
          So if you want to scrape a USB Key having the "same" type of path, you juste have to change the \retropie\roms By E:\roms for exemple, and all other path will adapte themself...

          Last but not least, the easiest way still to use Path like Sselph's scraper (or the path used in Recalbox) where all gamelist and downloaded picture are directly in the rom system folder.

          That's why I create a new profil in the last version with [Sselph Path] : much much more easier ;)

          hansolo77H 1 Reply Last reply Reply Quote 1
          • hansolo77H
            hansolo77 @screech
            last edited by

            @screech Yes, I see what the idea is supposed to be, but it doesn't work that way. It's not saving the information (images/gamelist) in the /home/pi/.emulationstation/downloaded_images/nes/ path. It saves it in the /home/pi/retropie/roms/configs/all/emulationstation/downloaded_images/nes/ path.

            Who's Scruffy Looking?

            1 Reply Last reply Reply Quote 0
            • screechS
              screech
              last edited by

              That's weird :S

              I just retry, and it works for me :S

              I haven't retropie, so I make a network directory on my nas named Retropie, and connect a netdrive on it... (just to simulate the retropie).

              And I try to scrape

              Do I have this : http://hpics.li/841835b

              So for me it's ok :S (from my point of view :S)

              Can you try delete the UXS folder, and download the very last version ?
              (I ask to delete the folder to be sure the "old" profil will be recreated)

              1 Reply Last reply Reply Quote 1
              • vbsV
                vbs
                last edited by

                Just tested it again and I have to say for me really ".." does not work. A workaround for now is to go to Autoconfiguration path configuration and edit XML file name and Folder name where pictures will be downloaded. Replace %Source_RootPath%\..\configs with \\<host>\configs. (enter the hostname instead of <host> ofc :)

                Important: After changing that you have to re-select the system in the menu "Select your system"!

                After that it worked for me and UXS used the correct folders.

                1 Reply Last reply Reply Quote 0
                • hansolo77H
                  hansolo77
                  last edited by

                  I think the problem is that you don't have a RetroPie to test this on. According to @vbs the problem is with the \..\ inside the path. So I'm trying his approach now.


                  Fresh Install.. Open the program, go to Configuation, go to Path Configuration.... Get this screen:
                  alt text

                  Changed XML file path to \\retropie\configs\all\emulationstation\gamelists\amstradcpc\gamelist.xml
                  Changed Directory where pictures will be downlo to \\retropie\configs\all\emulationstation\downloaded_images\amstradcpc

                  Will this only save for THAT system?


                  Fresh install still, go to Configuration, Autoconfiguration path configuration.. Get this screen:
                  alt text

                  Changed XML file name to \\retropie\configs\all\emulationstation\gamelists\%System%\gamelist.xml
                  Changed Folder Name where pictures will be do to \\retropie\configs\all\emulationstation\downloaded_images\%System%

                  Who's Scruffy Looking?

                  1 Reply Last reply Reply Quote 0
                  • screechS
                    screech
                    last edited by

                    I added a %host% tag.... So now, it will work for standard Retropie Path and if you change these path to a usb stick for exemple, only Root system directory need to be changed ;)

                    And some other correction : V2.0.0.4

                    1 Reply Last reply Reply Quote 1
                    • pommeP
                      pomme
                      last edited by pomme

                      Maybe a silly question. I just started using Universal XML Scraper and it's downloading the game screenshots by default. I was hoping to get boxart or maybe try out 3D boxart and see how it looks. Is there a place to change this in the configuration?

                      1 Reply Last reply Reply Quote 0
                      • screechS
                        screech
                        last edited by

                        of course you can @pomme ;)

                        just go to the option configuration menu. Select "picture" and wich picture you want ;)
                        (you can also select "alt picture" if the first one isn't found ;) )

                        1 Reply Last reply Reply Quote 1
                        • hansolo77H
                          hansolo77
                          last edited by

                          Here's a question:
                          I just scanned my collection of Sega CD games. I don't have a lot because of the limited storage space, only like 6 games. The scraper came back and said it found 6 files, and 5 out of 6 roms. I looked in the correct downloaded_images path and discovered the one it missed was Loadstar. Apparently the scraper doesn't like my hashes, and tried to search by filename and came up short. When I looked on the website, there are indeed MANY Loadstar games listed, but none of them match my filename 100%. So what do I do in this case? Do I go into my files and rename them to match one that matches? Or do I try to submit a new ROM? If I need to submit, how do I go about doing that?

                          Who's Scruffy Looking?

                          1 Reply Last reply Reply Quote 1
                          • D
                            derebo
                            last edited by

                            @screech you have developed a great tool. So congratulations for your effort! Is there an option to configure UXS to check databases other than screenscraper.fr ? Because there are great 8-bit homecomputing systems with rich databases already populated that would be a fantastic alternative to enrich our RetroPie personal installs.
                            Thank you once more for your great tool.

                            1 Reply Last reply Reply Quote 0
                            • screechS
                              screech
                              last edited by

                              @hansolo77 for only 1 rom, the fastest way is to rename your rom with a matching name ;)
                              But if you want to add some rom to the db, just send me your missing.txt files ;)
                              I can add them, but they need to be associated after that to the matching game ;)

                              @derebo sorry, for now it is not planned to add a new DB. But if you want, you are welcome to participate to Screenscraper, adding data and media missing in 8bit computer...

                              1 Reply Last reply Reply Quote 1
                              • hansolo77H
                                hansolo77
                                last edited by

                                @screech Thank you, I'll just rename the one ROM. :)

                                Here's one more question.. Is there any way for the software to see that it's missing certain bits of artwork, and then update them with a future pass? For instance, I'm using the 4-way MIX. When browsing through my systems I notice some ROMs are missing things like their Boxart, and some are missing screenshots. So if I come back in a week or whatever and try to scrape again, will it replace those images with newer ones if they're available at screenscrapper.fr?

                                Who's Scruffy Looking?

                                vbsV RionR 2 Replies Last reply Reply Quote 0
                                • vbsV
                                  vbs @hansolo77
                                  last edited by

                                  @hansolo77
                                  Not possible in that way AFAIK. You can either make a full rescan, meaning it will rebuild everything from scratch or you can do an update but it will only create missing entries (not update existing ones).

                                  But there is a ToDo for it on GitHub:
                                  https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/issues/23

                                  1 Reply Last reply Reply Quote 1
                                  • RionR
                                    Rion @hansolo77
                                    last edited by

                                    @hansolo77 You can also help out and add missing boxart, screenshot, metadata and so on.

                                    FBNeo rom filtering
                                    Mame2003 Arcade Bezels
                                    Fba Arcade Bezels
                                    Fba NeoGeo Bezels

                                    1 Reply Last reply Reply Quote 0
                                    • daeksD
                                      daeks
                                      last edited by daeks

                                      Nice tool, tried it on my machine and found couple of roms not associated. How can I upload these missing rom hashs to the website?

                                      Improvement suggestion: If you query the database for a rom and it can not be found, why not adding the hash and filename automatically to the db?

                                      1 Reply Last reply Reply Quote 0
                                      • screechS
                                        screech
                                        last edited by

                                        Hi @daeks , Sorry, you can't directly add roms hash to the DB, but you can send me your missing.txt file, and I'll check what I can do.

                                        For the auto add to database, it's already existe ;) but with limitation : the missing rom must be scraped from 5 differents user to be send to moderation, then modo can associate it to the good game.

                                        The aim is to avoid lot's of bad file (if you scrape your familly photo folder by mistake, we don't want to add them to the DB ;) )

                                        RionR 1 Reply Last reply Reply Quote 0
                                        • RionR
                                          Rion @screech
                                          last edited by

                                          @screech LoL

                                          FBNeo rom filtering
                                          Mame2003 Arcade Bezels
                                          Fba Arcade Bezels
                                          Fba NeoGeo Bezels

                                          1 Reply Last reply Reply Quote 0
                                          • L
                                            Lok
                                            last edited by Lok

                                            Hello and many thanks for this scrapper who look very usefull.

                                            At first, excuse my english.

                                            I'm using Retropie within kodi on a OSMC configuration with that https://discourse.osmc.tv/t/howto-retrosmc-retrogaming-on-osmc/6671 and everthing works fine. Except when I try Universal XLM scrapper.

                                            I think that the problem come from bad path because my Pi is not "/home/pi/RetroPie..." but / OSMC/osmc/RetroPie..." so I have changed the autoconfiguration path with something like :

                                            alt text

                                            I kill the emulstation process (after changing the SSH parameters), the scrapping launch and finish well but when I came back to RetroPie no information on my roms.

                                            A folder named "configs" is created on my Pi (\OSMC\osmc\configs) with lots of info in it but RetroPie won't get theses informations from there.

                                            I must have done womething wrong with the paths I guess but what ?

                                            Thanks for your help.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.