• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

Scheduled Pinned Locked Moved Projects and Themes
scrapescraperuxs
728 Posts 111 Posters 737.3k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mattrixk @FlyingTomahawk
    last edited by 10 Mar 2017, 13:35

    @FlyingTomahawk that's really odd. I didn't have a problem scraping videos, but I don't think I used that exact profile (I have lots of them scattered around my hard drive in various states of completion).

    I'll hopefully get a chance to have a look at it tomorrow.

    My ES themes: MetaPixel | Spare | Io | Indent

    1 Reply Last reply Reply Quote 0
    • M
      mattrixk @FlyingTomahawk
      last edited by 10 Mar 2017, 13:43

      @FlyingTomahawk okay, I just had a quick look and this is what it should scrape:

      <image>gamename-screenshot.jpg</image>
      <marquee>gamename-mix-video.png</marquee>
      <video>gamename-video.mp4</video>
      

      Are you maybe getting confused about the MIX image being called "-mix-video.png"?

      My ES themes: MetaPixel | Spare | Io | Indent

      F 1 Reply Last reply 10 Mar 2017, 14:11 Reply Quote 0
      • F
        FlyingTomahawk @mattrixk
        last edited by 10 Mar 2017, 14:11

        @mattrixk

        hmm...nope not confused....I think.
        I had the video files as png format and when I renamed them to mp4 all was good.
        I am fiddling for 3 hours now and at the same time cleaning out my roms since I try to reduce it to max 50 roms per system. Let's be honest no one really needs 700+ roms per system, there is no way to play all the games anyway. That will reduce my work with this video stuff significantly.
        So I might have a burnout at the moment and lost a bit overview and concentration.

        E M 2 Replies Last reply 10 Mar 2017, 16:27 Reply Quote 0
        • E
          EctoOne @FlyingTomahawk
          last edited by 10 Mar 2017, 16:27

          @FlyingTomahawk
          Tell me about it. I probably spend a whole week just to get perfect Metadata. At the very beginning I've made the decision to use the standard 3 img mix, but I changed it to use 2D boxart. So I was able to simply create/fix images if they had missing elements or none at all (for Romhacks). Then go through the created xml with the Metadata Editor to check if everything is set correctly. It was such a tedious task, I won't even imagine to do that with complete romsets.

          1 Reply Last reply Reply Quote 1
          • M
            mattrixk @FlyingTomahawk
            last edited by 10 Mar 2017, 21:48

            @FlyingTomahawk okay, that's really strange and I have no idea why that's happening. I'll have to do some testing.

            My ES themes: MetaPixel | Spare | Io | Indent

            F 1 Reply Last reply 10 Mar 2017, 23:05 Reply Quote 0
            • F
              FlyingTomahawk @mattrixk
              last edited by FlyingTomahawk 3 Oct 2017, 23:20 10 Mar 2017, 23:05

              @mattrixk

              Yesterday before I called it a day I reverted everything. I deleted the UXMLS files and downloaded a fresh package from github.
              Now I don't know what is going on but I have the following issues now.

              1.)
              No matter if I choose RetroPie mix profile or regular RetroPie when I scrape now and select create new XML file as option it scrapes 3 images.
              Marquee, video.png and screenshot.
              Looking at the XML file there I have the right paths but at the end many empty </game> tags.

              1. )
                If I select "update XML file" in the options he doesn't scrape anything. 0/0 roms.

              3.)
              Previously I could download only videos or only images without creating an XML file by selecting "create image only" in the options.
              When I do that and I select video it creates the videos in png format and also the regular screenshot images.

              I have no idea why this all is happening. I mean I deleted everything from my PC, restarted it, downloaded a fresh, new UXMLS package from github. Still it won't go back to normal.
              I will take another look at it today but this is getting annoying.

              M 1 Reply Last reply 11 Mar 2017, 04:22 Reply Quote 0
              • F
                FlyingTomahawk
                last edited by FlyingTomahawk 3 Nov 2017, 04:23 11 Mar 2017, 04:08

                So back at it again...new day, new luck?

                Can someone please give the following a try and tell me what they get?


                Load RatroPie MIX [Rom folder]

                Select Standard (2img)

                Settings in General options

                Create New XML file

                Picture configuration

                default

                Options configuration

                Item Choice

                Marquee

                Paramater to change

                Wheels

                What files do you get and what file extensions do you see in the game list?


                Load RatroPie MIX [Rom folder]

                Select Standard (2img)

                Settings in General options

                Create New XML file

                Picture configuration

                default

                Options configuration

                Item Choice

                Video

                Paramater to change

                Yes

                What files do you get and what file extensions do you see in the game list?

                1 Reply Last reply Reply Quote 0
                • M
                  mattrixk @FlyingTomahawk
                  last edited by 11 Mar 2017, 04:22

                  @FlyingTomahawk hmm, very strange. Do things scrape correctly if you just use the built-in profiles? The latest version of UXS is v6, so if you're still having problems with that one, you could try v5 or v4. I don't think I'll have a chance to do any testing today (niece's b'day then mate's going away party), but I'll hopefully have a chance tomorrow.

                  My ES themes: MetaPixel | Spare | Io | Indent

                  F 1 Reply Last reply 11 Mar 2017, 04:28 Reply Quote 0
                  • F
                    FlyingTomahawk @mattrixk
                    last edited by 11 Mar 2017, 04:28

                    @mattrixk

                    That's the first thing I did this morning. I tested the Standard (2img ) and all was good. Correct images with correct file extension and correct gamelist entries.

                    I want to scrape that game logo image (wheel? ) only that you can see bottom-right on the Standard 2 and 3img profile.
                    And only the video.
                    So 2x scraping total.

                    Just enjoy your day/weekend, please don't mind me. I will figure it out one way or the other, just a matter of time.

                    M 1 Reply Last reply 11 Mar 2017, 04:40 Reply Quote 0
                    • M
                      mattrixk @FlyingTomahawk
                      last edited by 11 Mar 2017, 04:40

                      @FlyingTomahawk no worries mate, we'll get this worked out (eventually).

                      My ES themes: MetaPixel | Spare | Io | Indent

                      1 Reply Last reply Reply Quote 1
                      • D
                        Dochartaigh
                        last edited by 11 Mar 2017, 15:25

                        I swore this was already gone over, but you can't search this entire thread (keeps on loading only sections at a time).... I'm having an issue where scraping Playstation adds a gamelist entry for every single file a certain PS1 game will have (some have just a .BIN and a .CUE - others have like 30 .BIN's and a single .CUE - so that one would have 31 files listed in the gamelist). How to fix?

                        J 1 Reply Last reply 11 Mar 2017, 17:00 Reply Quote 0
                        • J
                          Jeck11 @Dochartaigh
                          last edited by 11 Mar 2017, 17:00

                          @Dochartaigh add an exclusion for *.bin and you shouldn't get doubles.

                          Also, to search a specific thread on this site just click in the thread the hit ctrl-f. You should see something like thread:678533 show up in the search box.

                          D 1 Reply Last reply 11 Mar 2017, 17:15 Reply Quote 1
                          • D
                            Dochartaigh @Jeck11
                            last edited by 11 Mar 2017, 17:15

                            @Jeck11 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                            @Dochartaigh add an exclusion for *.bin and you shouldn't get doubles.

                            Also, to search a specific thread on this site just click in the thread the hit ctrl-f. You should see something like thread:678533 show up in the search box.

                            Thanks, I'll exclude .bin's!

                            And this forum only loads so many posts at once - so there's 380 posts in this topic, and instead of splitting it into multiple pages like the most common Phpbb and simple machines (and similar) forums do, it'll only load the last 10 or 20 posts so that's all you can search for with CTRL+F unfortunately (unless I'm missing something).

                            1 Reply Last reply Reply Quote 0
                            • S
                              screech
                              last edited by 13 Mar 2017, 09:37

                              Hello Every one.

                              Just pull out a fresh new release (v2.2.0.0)

                              What's new :

                              New Language Set
                              Now English Language are separated in 2 : English UK and English US. So now it handle correctly the media fallback (exemple : In UK fallback if a cover matching the rom is not found it will take the European one and the world one if not found. In US, it will take the US one and the world one... Same for the game naming)

                              A complete new Wizard.
                              I force it at start to make sure every one have a "good" install first.
                              I put a lot's of "verification" to make sure a standard install can scrape easy.
                              You can always tweek what you want after that ;)

                              A New "Options" Menu.
                              Now you will have a lot's of "easy" tweek" like :

                              • Mix or not.
                              • If not Mix : selecting Default picture, alt picture and for Retropie : Marquee and Videos
                              • Naming convention : Default is "Game Name + Options" so you can tweek the options.
                              • Name Tag options : Now you can put the country between [], and some tags between () like Clone, Beta, Demo, Proto, Trad, Hack, Unl, Alt, Best. (To come : Clone type and Hack type) -> So normally no more simple duplicate name (for mame for exemple). If still duplicate, it's because we haven't the info in DB. Don't hesitate to submit correction on that on www.screenscraper.fr
                              • Case for multiple metadata : Default/UPPERCASE/lowercase (depending on Theme, it may matter)

                              New Profil :
                              Profil change "a lot"... Not so much modification of existing things, but only 2 profils now : Recalbox/Retropie. With a lot's of new options (at the end).
                              It create dynamically the option menu.
                              We loose some "options" but I will put them back later (Like the "rom folder" profil or the "scummvm" profil)... I need to work on these.
                              Special dedicace to @mattrixk if you have any difficulty with modification on these profil, let me know.

                              New option in Gen config
                              It was asked, by default, UXS don't scrape "IN" zip file... So it may not found some game, but it's a way faster with romset like mame.
                              You can reactivate that in option.

                              Carbon Wheel on the UI
                              Now you can know what plateform you are scraping, it put a little carbon wheel on the UI.
                              This wheel depend on you language preset (so Genesis for US or Megadrive dor UK ;) )

                              Some debbuging

                              • Full Scrape works now ^^
                              • Full scrape have a good "Result" windows at the end
                              • Normally no more bug when scrape multiple system in the same session (you don't need to close and reopen UXS)
                              • Lot's of small debbuging.

                              To come :

                              • A new server (actually the small server we have suffer a lot. Specially at Week end) thanks to your donation we migrate soon to a new bigger and dedicated physical server.
                              • A new API (With the new server, little time after, we will have a new API with a lot's more new stuff to scrape and more easily ;) )
                              • A new Error handling (Actually error coming from the API aren't so easy to catch. So with the new API, I can have a better return and so a better information on what's going on when it doesn't work properly)
                              • lot's of new stuff in UXS ;)

                              Asking for Help
                              With this new version, some translation aren't up to date. So if you speak German, Spanish, Dutch, Portugaise, Japanaise or Italian, you are welcome to contribute.
                              (Juste look in the LanguageFiles Folder. Open the XML correspondinf. The stuff needing translation are at the end ;) )
                              If you think my English isn't so good (what is true) don't hesitate to correct the EnglishUK or EnglishUS XML ;)

                              @FlyingTomahawk : Just Open the Advanced Menu, Image configuration and select "Default" as extension. It will solve all your extension problem ;)

                              Thank you guys, and don't hesitate to tell me if something don't work well ;)

                              R 1 Reply Last reply 13 Mar 2017, 13:56 Reply Quote 3
                              • R
                                Rion @screech
                                last edited by 13 Mar 2017, 13:56

                                @screech
                                Thats a new hefty release list 😀
                                God to hear about the new server!

                                FBNeo rom filtering
                                Mame2003 Arcade Bezels
                                Fba Arcade Bezels
                                Fba NeoGeo Bezels

                                1 Reply Last reply Reply Quote 0
                                • E
                                  ExarKunIv
                                  last edited by 13 Mar 2017, 16:09

                                  Having a problem getting this to launch

                                  i got the latest version from Github v 2.2.0.0
                                  I run Universal_XML_Scraper64.exe. i get the splash screen and then a error pops up

                                  Line 38253 (file "c:\my stuff/ Universal-XML-Scraper-master\Universal_XML_Scraper64.exe"):

                                  error subscript used on non-accessible variable

                                  i looked in the logs folder and here is what i got

                                  [11:03:09] - C:\My stuff\Universal-XML-Scraper-master\Universal_XML_Scraper64.exe (2.2.0.0)
                                  [11:03:09] - WIN_8(X64) - 0409
                                  [11:03:09] - Scrape in x64
                                  [11:03:09] - > No updated files needed (Version : 2.2.0.0)
                                  [11:03:09] - Starting files installation
                                  [11:03:09] - > Ending files installation
                                  [11:03:09] - /!\ Error Downloading File : C:\My stuff\Universal-XML-Scraper-master\Ressources\ServerList.txt
                                  [11:03:09] - /!\ Bytes read: 0
                                  [11:03:09] - /!\ Size: 0
                                  [11:03:09] - /!\ Complete: True
                                  [11:03:09] - /!\ successful: False
                                  [11:03:09] - /!\ @error: 13
                                  [11:03:09] - /!\ @extended: 12029
                                  [11:03:10] - /!\ Error Downloading File : C:\My stuff\Universal-XML-Scraper-master\Ressources\ServerList.txt
                                  [11:03:10] - /!\ Bytes read: 0
                                  [11:03:10] - /!\ Size: 0
                                  [11:03:10] - /!\ Complete: True
                                  [11:03:10] - /!\ successful: False
                                  [11:03:10] - /!\ @error: 13
                                  [11:03:10] - /!\ @extended: 12029
                                  [11:03:10] - /!\ Error Downloading File : C:\My stuff\Universal-XML-Scraper-master\Ressources\ServerList.txt
                                  [11:03:10] - /!\ Bytes read: 0
                                  [11:03:10] - /!\ Size: 0
                                  [11:03:10] - /!\ Complete: True
                                  [11:03:10] - /!\ successful: False
                                  [11:03:10] - /!\ @error: 13
                                  [11:03:10] - /!\ @extended: 12029
                                  [11:03:10] - > -In 3 try and 0.86s
                                  

                                  so i look in the Ressources folder for the ServerList.txt and i saw it was missing so i unziped Ressources zip and placed it in there. i saw that once i ran the exe it disappeared. and yes i did try in admin mode.

                                  any ideas what im doing wrong here

                                  RPi3B+ / 200GB/ RetroPie v4.5.14, RPi4 Model B 4gb / 256gb / RetroPie 4.8.2
                                  RPi5 4gb / 512gb / RetroPie 4.8.9 -Basic
                                  Maintainer of RetroPie-Extra .

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    screech
                                    last edited by 13 Mar 2017, 16:17

                                    strange, serverList is here : https://raw.githubusercontent.com/Universal-Rom-Tools/Universal-XML-Scraper/master/Ressources/ServerList.txt

                                    It's not a DB issue, it's like UXS can't download it... Have you open your firewall ?
                                    (FYI it's the first file it try to download... So...)

                                    1 Reply Last reply Reply Quote 0
                                    • E
                                      ExarKunIv
                                      last edited by 13 Mar 2017, 16:39

                                      yes i have the firewall turned off. still same thing.
                                      Im currently at work. so i think it is something they have set for the firewall here. :(

                                      ill give it a shot when i get home.
                                      thx for the help

                                      RPi3B+ / 200GB/ RetroPie v4.5.14, RPi4 Model B 4gb / 256gb / RetroPie 4.8.2
                                      RPi5 4gb / 512gb / RetroPie 4.8.9 -Basic
                                      Maintainer of RetroPie-Extra .

                                      1 Reply Last reply Reply Quote 0
                                      • E
                                        ExarKunIv
                                        last edited by 13 Mar 2017, 16:56

                                        my work around is i just install my VPN here, and im useing that. lol

                                        RPi3B+ / 200GB/ RetroPie v4.5.14, RPi4 Model B 4gb / 256gb / RetroPie 4.8.2
                                        RPi5 4gb / 512gb / RetroPie 4.8.9 -Basic
                                        Maintainer of RetroPie-Extra .

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          DynaMight
                                          last edited by 13 Mar 2017, 20:07

                                          The update looks good but it isnt scraping anywhere near as many games? The previous update scraped 896 out of 900 roms (Megadrive) however the latest update only managed to scrape 565? Exactly the same roms. I re-ran it and it did exactly the same amount.

                                          Any ideas?

                                          1 Reply Last reply Reply Quote 0
                                          388 out of 728
                                          • First post
                                            388/728
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received