RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

    Scheduled Pinned Locked Moved Projects and Themes
    scrapescraperuxs
    728 Posts 111 Posters 749.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      EctoOne
      last edited by EctoOne

      I'm going to answer my own question about scraping ScummVM games. It doesn't work with RetroPie. At least not in a way that would allow for easy scraping and playing. To make it work, screenscraper.fr would need all the hashes from the different .svm files RetroPie uses to be able to start ScummVM games straight from Emulation Station.

      At least i figured out how to use the .scummvm files from this thread over at the Recalbox forum, to get Mix Images.

      If you want to get the images/videos:

      • Get the RAR file from the thread mentioned
      • Put all .scummvm files into a folder named scummvm in your rom folder.
      • Open the profil xml you're using (e.g. Retropie (MIX).xml) in a texteditor.
      • Add *.scummvm;into the <Research> tag.
      • Save the file and open UXS to scrape.

      What i did was to scrape inside Emulation Station (need to manually insert the right game name for every game) after ScummVM created all the .svm files. Then i replaced the scraped images with the Mix images from UXS.

      Sidenote:
      I also added *.uae; to my <research> tag and it worked right away only using scraping for names without hashes.
      If anyone is interested, here is my profil.xml
      It has support for uae and scummvm files enabled. It also downloads the 2D boxart, screenshot, marquee and (if available) fanart and video. BUT it doesn't add those to the gamelist! It only adds the selected Mix image and the normal Metadata. I also removed the <thumbnail> tag and the gameid/screenscraper name from the <game> tag.
      The only issue with this is, that videos are saved as png files and so far i don't know how to fix it. Although since i don't use video view i don't care.

      P 1 Reply Last reply Reply Quote 0
      • P
        paradadf @EctoOne
        last edited by

        @EctoOne I'm not sure how the .svm files look inside, but if they are empty, you can just do this:
        https://retropie.org.uk/forum/topic/4142/test-a-new-data-source-for-sselph-scraper/17
        It has a link to the samd recalbox tutorial. Just ñut the official long name inside the .svm file and you are ready to scrape wirhout too much hassle.

        E 1 Reply Last reply Reply Quote 0
        • E
          EctoOne @paradadf
          last edited by

          @paradadf They are empty but whatever. I just finished scraping all images, so i won't test it. :P
          In the end it was easier to scrape the whole scummvm folder from the rar and replace *.scummvm with *.svm inside the gamelist.xml UXS creates with the ones the GUI creates.

          1 Reply Last reply Reply Quote 0
          • J
            Jeck11
            last edited by

            New Feature - It would be cool to be able to use your combination engine to create an image offline. It would be able to take 3 images that we supply and combine them into the single image then write to the Pi.

            1 Reply Last reply Reply Quote 0
            • S
              stet
              last edited by

              I have been getting the following error when trying to use "Scrap all your system":

              Line 22140 (File "C:\winapps\Universal Scraper\Universal_XML_Scraper64.exe"}:

              Error: Variable used without being declared.

              Because of this error I have had to scrape each system separately instead of just letting it run overnight. Is anyone else getting a similar error?

              patastinkyP 1 Reply Last reply Reply Quote 0
              • screechS
                screech
                last edited by

                Yep :( sorry.... Scrape all is broken right now :( I need to correct it...

                1 Reply Last reply Reply Quote 0
                • denisuuD
                  denisuu
                  last edited by

                  How can I scrape PORTS? I can use the ES scraper to find DOOM, DUKENUKEM3d, Quake... But not Outrun etc. Can I use U XML for this? Or any way to manually do this? I already tried editing the gamelist.xml under /home/pi/RetroPie/roms/ports but that doesn't work.

                  I tried another app (Emulationstation XML maker) but that results in the same and editing /opt/retropie/configs/all/emulationstation/gamelists/ports/gamelist.xml also doesn't work. :(

                  1 Reply Last reply Reply Quote 0
                  • patastinkyP
                    patastinky @stet
                    last edited by

                    @stet said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                    I have been getting the following error when trying to use "Scrap all your system":

                    Line 22140 (File "C:\winapps\Universal Scraper\Universal_XML_Scraper64.exe"}:

                    Error: Variable used without being declared.

                    Because of this error I have had to scrape each system separately instead of just letting it run overnight. Is anyone else getting a similar error?

                    I'm having the same issue with NES and N64. SNES was able to scrap just fine. I tried SCRAPE ALL and got that problem, hence why I was scraping them individually all day today. Just got down to N64 and NES when the "LINE 22140" error appeared.

                    1 Reply Last reply Reply Quote 0
                    • FlyingTomahawkF
                      FlyingTomahawk
                      last edited by FlyingTomahawk

                      @screech

                      Is it possible to create a custom Mix image profile with video and marquee similar to the Standard (2img) but instead of a screenshot have a video?

                      mattrixkM 1 Reply Last reply Reply Quote 1
                      • mattrixkM
                        mattrixk @FlyingTomahawk
                        last edited by

                        @FlyingTomahawk: There are 2 different Profile types:

                        1. Scrape Profile
                        2. MIX Profile

                        The Scrape Profile is the one that grabs the metadata, and downloads the single images (and videos) and creates the gamelist using them.

                        The MIX Profile gets the selected images and combines them together in the positions you want them.

                        You tell the Scrape Profile to create a MIX image, and then you tell the UXS program which MIX Profile you want to use to create that image.

                        eg:

                        You have 3 tags to fill: <video>, <image> and <marquee>.

                        • "md_video" uses "md_image" as a backup if the video is missing, so a Screenshot is best for that.
                        • "md_marquee" sits over the video/screenshot (in the z-index), so you might want the game Logo and 3D Box-art, so you want to make a MIX image for that.

                        You create a MIX Profile that combines the Logo and the 3D Box-art into a single image, and call the profile "Logo Box-art MIX".

                        You then create a Scrape Profile called "Scrape [video][image][marquee]" that:

                        • Downloads the Video and puts it into the <video> tag of the gamelist.xml.
                        • Downloads the Screenshot and puts it into the <image> tag of the gamelist.xml.
                        • Downloads the selected MIX Profile image and puts it into the <marquee> tag of the gamelist.xml.

                        In UXS, select the "Scrape [video][image][marquee]" Scrape profile. (You'll have to set up the autoconfig to point to your Roms, and also make sure the gamelist.xml has all the paths set correctly. You might need to play with that a little to get it right.)

                        Now set the MIX Profile to "Logo Box-art MIX" and scrape your Roms.

                        I suggest testing it first on a System that only has 2 or 3 Roms in it. It's annoying to scrape the data/images for a few hundred Roms only to discover the MIX Image doesn't look how you want it to, or you've missed something in the code so the Screenshot didn't download, forcing you to do it all again.

                        If you need examples of the code let me know, but you can usually suss it out from the Profiles provided with UXS.

                        My ES themes: MetaPixel | Spare | Io | Indent

                        FlyingTomahawkF 1 Reply Last reply Reply Quote 1
                        • FlyingTomahawkF
                          FlyingTomahawk @mattrixk
                          last edited by

                          @mattrixk

                          I knew that it is possible to create mix profiles I just didn't know how to add the video part.
                          I wanted to take the Standard 2img profile and just replace the screenshot code with the video code.
                          You provided your code which will help me do that. Thanks.

                          mattrixkM 2 Replies Last reply Reply Quote 0
                          • mattrixkM
                            mattrixk @FlyingTomahawk
                            last edited by

                            @FlyingTomahawk:
                            This Scrape Profile should do what you want it to do.

                            Just add it to the "ProfilsFiles" folder within UXS, then select it from the Scrape Profiles list. You'll also need to set up the autoconfig so UXS sees your Roms, and points the paths where they need to go. You'll then need to select the Standard (2img) MIX Profile and you should be good to go.

                            My ES themes: MetaPixel | Spare | Io | Indent

                            FlyingTomahawkF 1 Reply Last reply Reply Quote 1
                            • FlyingTomahawkF
                              FlyingTomahawk @mattrixk
                              last edited by

                              @mattrixk

                              Awesome! Thx.
                              I test it as soon I get back from work.

                              1 Reply Last reply Reply Quote 0
                              • mattrixkM
                                mattrixk @FlyingTomahawk
                                last edited by

                                @FlyingTomahawk I had to update that xml file as it still had some unneeded parts in it, but it's fixed now.

                                My ES themes: MetaPixel | Spare | Io | Indent

                                FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                                • FlyingTomahawkF
                                  FlyingTomahawk @mattrixk
                                  last edited by FlyingTomahawk

                                  @mattrixk

                                  I had a chance to test the code of yours and it does give me the correct entries inside the gamelist xml file but the videos are downloaded as "gamename"-video.png
                                  I can see that the entry for video in your file is correct and says mp4 as file format but I think the Standard (2img) code is saying to save it as png. I tried to change that to match a video file and mp4 format but then I get all kinds of errors.
                                  Right now I am scraping the roms twice, once for marquee and once for video. At the same time I set it to update the XML file to get the right entries.
                                  That works good for me right now, I just thought I could reduce 1 scraping step and have both at once.

                                  E mattrixkM 2 Replies Last reply Reply Quote 0
                                  • E
                                    EctoOne @FlyingTomahawk
                                    last edited by

                                    @FlyingTomahawk Can you not just batch rename the *-video.png files? I'm using a similar scrape profil as @mattrixk has and it also downloads the video files as png. My script downloads all files (2D Box, marquee, screenshot, fanart, video), creates a Mix Image but only put the mix image in the xml (because i don't care about video, but i want to have the other images). So that's why i deleted all my *-video.png files after scraping without testing if those where just falsly named mp4 files.

                                    FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                                    • FlyingTomahawkF
                                      FlyingTomahawk @EctoOne
                                      last edited by FlyingTomahawk

                                      @EctoOne

                                      Yes, I did that. I batch renamed them and I can do that for all the others too.
                                      Still it does not reduce my work, I still need to do 2 steps either scraping, scraping or scraping, renaming.
                                      But then again I don't think I will be doing a lot scraping once everything is setup, so a 2 step solution is not bad after all.

                                      E mattrixkM 2 Replies Last reply Reply Quote 0
                                      • E
                                        EctoOne @FlyingTomahawk
                                        last edited by

                                        @FlyingTomahawk
                                        OH right, I forgot that you have to edit the xml files as well if you rename the files. I probably purged the memory from my brain about all the work I had with my setup. :D

                                        1 Reply Last reply Reply Quote 1
                                        • mattrixkM
                                          mattrixk @FlyingTomahawk
                                          last edited by

                                          @FlyingTomahawk that's really odd. I didn't have a problem scraping videos, but I don't think I used that exact profile (I have lots of them scattered around my hard drive in various states of completion).

                                          I'll hopefully get a chance to have a look at it tomorrow.

                                          My ES themes: MetaPixel | Spare | Io | Indent

                                          1 Reply Last reply Reply Quote 0
                                          • mattrixkM
                                            mattrixk @FlyingTomahawk
                                            last edited by

                                            @FlyingTomahawk okay, I just had a quick look and this is what it should scrape:

                                            <image>gamename-screenshot.jpg</image>
                                            <marquee>gamename-mix-video.png</marquee>
                                            <video>gamename-video.mp4</video>
                                            

                                            Are you maybe getting confused about the MIX image being called "-mix-video.png"?

                                            My ES themes: MetaPixel | Spare | Io | Indent

                                            FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.