RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

    Scheduled Pinned Locked Moved Projects and Themes
    scrapescraperuxs
    728 Posts 111 Posters 745.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mattrixkM
      mattrixk @FlyingTomahawk
      last edited by

      @FlyingTomahawk: There are 2 different Profile types:

      1. Scrape Profile
      2. MIX Profile

      The Scrape Profile is the one that grabs the metadata, and downloads the single images (and videos) and creates the gamelist using them.

      The MIX Profile gets the selected images and combines them together in the positions you want them.

      You tell the Scrape Profile to create a MIX image, and then you tell the UXS program which MIX Profile you want to use to create that image.

      eg:

      You have 3 tags to fill: <video>, <image> and <marquee>.

      • "md_video" uses "md_image" as a backup if the video is missing, so a Screenshot is best for that.
      • "md_marquee" sits over the video/screenshot (in the z-index), so you might want the game Logo and 3D Box-art, so you want to make a MIX image for that.

      You create a MIX Profile that combines the Logo and the 3D Box-art into a single image, and call the profile "Logo Box-art MIX".

      You then create a Scrape Profile called "Scrape [video][image][marquee]" that:

      • Downloads the Video and puts it into the <video> tag of the gamelist.xml.
      • Downloads the Screenshot and puts it into the <image> tag of the gamelist.xml.
      • Downloads the selected MIX Profile image and puts it into the <marquee> tag of the gamelist.xml.

      In UXS, select the "Scrape [video][image][marquee]" Scrape profile. (You'll have to set up the autoconfig to point to your Roms, and also make sure the gamelist.xml has all the paths set correctly. You might need to play with that a little to get it right.)

      Now set the MIX Profile to "Logo Box-art MIX" and scrape your Roms.

      I suggest testing it first on a System that only has 2 or 3 Roms in it. It's annoying to scrape the data/images for a few hundred Roms only to discover the MIX Image doesn't look how you want it to, or you've missed something in the code so the Screenshot didn't download, forcing you to do it all again.

      If you need examples of the code let me know, but you can usually suss it out from the Profiles provided with UXS.

      My ES themes: MetaPixel | Spare | Io | Indent

      FlyingTomahawkF 1 Reply Last reply Reply Quote 1
      • FlyingTomahawkF
        FlyingTomahawk @mattrixk
        last edited by

        @mattrixk

        I knew that it is possible to create mix profiles I just didn't know how to add the video part.
        I wanted to take the Standard 2img profile and just replace the screenshot code with the video code.
        You provided your code which will help me do that. Thanks.

        mattrixkM 2 Replies Last reply Reply Quote 0
        • mattrixkM
          mattrixk @FlyingTomahawk
          last edited by

          @FlyingTomahawk:
          This Scrape Profile should do what you want it to do.

          Just add it to the "ProfilsFiles" folder within UXS, then select it from the Scrape Profiles list. You'll also need to set up the autoconfig so UXS sees your Roms, and points the paths where they need to go. You'll then need to select the Standard (2img) MIX Profile and you should be good to go.

          My ES themes: MetaPixel | Spare | Io | Indent

          FlyingTomahawkF 1 Reply Last reply Reply Quote 1
          • FlyingTomahawkF
            FlyingTomahawk @mattrixk
            last edited by

            @mattrixk

            Awesome! Thx.
            I test it as soon I get back from work.

            1 Reply Last reply Reply Quote 0
            • mattrixkM
              mattrixk @FlyingTomahawk
              last edited by

              @FlyingTomahawk I had to update that xml file as it still had some unneeded parts in it, but it's fixed now.

              My ES themes: MetaPixel | Spare | Io | Indent

              FlyingTomahawkF 1 Reply Last reply Reply Quote 0
              • FlyingTomahawkF
                FlyingTomahawk @mattrixk
                last edited by FlyingTomahawk

                @mattrixk

                I had a chance to test the code of yours and it does give me the correct entries inside the gamelist xml file but the videos are downloaded as "gamename"-video.png
                I can see that the entry for video in your file is correct and says mp4 as file format but I think the Standard (2img) code is saying to save it as png. I tried to change that to match a video file and mp4 format but then I get all kinds of errors.
                Right now I am scraping the roms twice, once for marquee and once for video. At the same time I set it to update the XML file to get the right entries.
                That works good for me right now, I just thought I could reduce 1 scraping step and have both at once.

                E mattrixkM 2 Replies Last reply Reply Quote 0
                • E
                  EctoOne @FlyingTomahawk
                  last edited by

                  @FlyingTomahawk Can you not just batch rename the *-video.png files? I'm using a similar scrape profil as @mattrixk has and it also downloads the video files as png. My script downloads all files (2D Box, marquee, screenshot, fanart, video), creates a Mix Image but only put the mix image in the xml (because i don't care about video, but i want to have the other images). So that's why i deleted all my *-video.png files after scraping without testing if those where just falsly named mp4 files.

                  FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                  • FlyingTomahawkF
                    FlyingTomahawk @EctoOne
                    last edited by FlyingTomahawk

                    @EctoOne

                    Yes, I did that. I batch renamed them and I can do that for all the others too.
                    Still it does not reduce my work, I still need to do 2 steps either scraping, scraping or scraping, renaming.
                    But then again I don't think I will be doing a lot scraping once everything is setup, so a 2 step solution is not bad after all.

                    E mattrixkM 2 Replies Last reply Reply Quote 0
                    • E
                      EctoOne @FlyingTomahawk
                      last edited by

                      @FlyingTomahawk
                      OH right, I forgot that you have to edit the xml files as well if you rename the files. I probably purged the memory from my brain about all the work I had with my setup. :D

                      1 Reply Last reply Reply Quote 1
                      • mattrixkM
                        mattrixk @FlyingTomahawk
                        last edited by

                        @FlyingTomahawk that's really odd. I didn't have a problem scraping videos, but I don't think I used that exact profile (I have lots of them scattered around my hard drive in various states of completion).

                        I'll hopefully get a chance to have a look at it tomorrow.

                        My ES themes: MetaPixel | Spare | Io | Indent

                        1 Reply Last reply Reply Quote 0
                        • mattrixkM
                          mattrixk @FlyingTomahawk
                          last edited by

                          @FlyingTomahawk okay, I just had a quick look and this is what it should scrape:

                          <image>gamename-screenshot.jpg</image>
                          <marquee>gamename-mix-video.png</marquee>
                          <video>gamename-video.mp4</video>
                          

                          Are you maybe getting confused about the MIX image being called "-mix-video.png"?

                          My ES themes: MetaPixel | Spare | Io | Indent

                          FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                          • FlyingTomahawkF
                            FlyingTomahawk @mattrixk
                            last edited by

                            @mattrixk

                            hmm...nope not confused....I think.
                            I had the video files as png format and when I renamed them to mp4 all was good.
                            I am fiddling for 3 hours now and at the same time cleaning out my roms since I try to reduce it to max 50 roms per system. Let's be honest no one really needs 700+ roms per system, there is no way to play all the games anyway. That will reduce my work with this video stuff significantly.
                            So I might have a burnout at the moment and lost a bit overview and concentration.

                            E mattrixkM 2 Replies Last reply Reply Quote 0
                            • E
                              EctoOne @FlyingTomahawk
                              last edited by

                              @FlyingTomahawk
                              Tell me about it. I probably spend a whole week just to get perfect Metadata. At the very beginning I've made the decision to use the standard 3 img mix, but I changed it to use 2D boxart. So I was able to simply create/fix images if they had missing elements or none at all (for Romhacks). Then go through the created xml with the Metadata Editor to check if everything is set correctly. It was such a tedious task, I won't even imagine to do that with complete romsets.

                              1 Reply Last reply Reply Quote 1
                              • mattrixkM
                                mattrixk @FlyingTomahawk
                                last edited by

                                @FlyingTomahawk okay, that's really strange and I have no idea why that's happening. I'll have to do some testing.

                                My ES themes: MetaPixel | Spare | Io | Indent

                                FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                                • FlyingTomahawkF
                                  FlyingTomahawk @mattrixk
                                  last edited by FlyingTomahawk

                                  @mattrixk

                                  Yesterday before I called it a day I reverted everything. I deleted the UXMLS files and downloaded a fresh package from github.
                                  Now I don't know what is going on but I have the following issues now.

                                  1.)
                                  No matter if I choose RetroPie mix profile or regular RetroPie when I scrape now and select create new XML file as option it scrapes 3 images.
                                  Marquee, video.png and screenshot.
                                  Looking at the XML file there I have the right paths but at the end many empty </game> tags.

                                  1. )
                                    If I select "update XML file" in the options he doesn't scrape anything. 0/0 roms.

                                  3.)
                                  Previously I could download only videos or only images without creating an XML file by selecting "create image only" in the options.
                                  When I do that and I select video it creates the videos in png format and also the regular screenshot images.

                                  I have no idea why this all is happening. I mean I deleted everything from my PC, restarted it, downloaded a fresh, new UXMLS package from github. Still it won't go back to normal.
                                  I will take another look at it today but this is getting annoying.

                                  mattrixkM 1 Reply Last reply Reply Quote 0
                                  • FlyingTomahawkF
                                    FlyingTomahawk
                                    last edited by FlyingTomahawk

                                    So back at it again...new day, new luck?

                                    Can someone please give the following a try and tell me what they get?


                                    Load RatroPie MIX [Rom folder]

                                    Select Standard (2img)

                                    Settings in General options

                                    Create New XML file

                                    Picture configuration

                                    default

                                    Options configuration

                                    Item Choice

                                    Marquee

                                    Paramater to change

                                    Wheels

                                    What files do you get and what file extensions do you see in the game list?


                                    Load RatroPie MIX [Rom folder]

                                    Select Standard (2img)

                                    Settings in General options

                                    Create New XML file

                                    Picture configuration

                                    default

                                    Options configuration

                                    Item Choice

                                    Video

                                    Paramater to change

                                    Yes

                                    What files do you get and what file extensions do you see in the game list?

                                    1 Reply Last reply Reply Quote 0
                                    • mattrixkM
                                      mattrixk @FlyingTomahawk
                                      last edited by

                                      @FlyingTomahawk hmm, very strange. Do things scrape correctly if you just use the built-in profiles? The latest version of UXS is v6, so if you're still having problems with that one, you could try v5 or v4. I don't think I'll have a chance to do any testing today (niece's b'day then mate's going away party), but I'll hopefully have a chance tomorrow.

                                      My ES themes: MetaPixel | Spare | Io | Indent

                                      FlyingTomahawkF 1 Reply Last reply Reply Quote 0
                                      • FlyingTomahawkF
                                        FlyingTomahawk @mattrixk
                                        last edited by

                                        @mattrixk

                                        That's the first thing I did this morning. I tested the Standard (2img ) and all was good. Correct images with correct file extension and correct gamelist entries.

                                        I want to scrape that game logo image (wheel? ) only that you can see bottom-right on the Standard 2 and 3img profile.
                                        And only the video.
                                        So 2x scraping total.

                                        Just enjoy your day/weekend, please don't mind me. I will figure it out one way or the other, just a matter of time.

                                        mattrixkM 1 Reply Last reply Reply Quote 0
                                        • mattrixkM
                                          mattrixk @FlyingTomahawk
                                          last edited by

                                          @FlyingTomahawk no worries mate, we'll get this worked out (eventually).

                                          My ES themes: MetaPixel | Spare | Io | Indent

                                          1 Reply Last reply Reply Quote 1
                                          • D
                                            Dochartaigh
                                            last edited by

                                            I swore this was already gone over, but you can't search this entire thread (keeps on loading only sections at a time).... I'm having an issue where scraping Playstation adds a gamelist entry for every single file a certain PS1 game will have (some have just a .BIN and a .CUE - others have like 30 .BIN's and a single .CUE - so that one would have 31 files listed in the gamelist). How to fix?

                                            J 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.