RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

    Scheduled Pinned Locked Moved Projects and Themes
    scrapescraperuxs
    728 Posts 111 Posters 750.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • FlyingTomahawkF
      FlyingTomahawk
      last edited by FlyingTomahawk

      @screech
      Thanks for your reply.
      Today it was faster, lot faster than yesterday. I have an account at screenscraper.fr but I haven't contributed ($$) anything yet.
      Is it possible to run the scrape again but let it only scrape the roms from the missing list?

      1 Reply Last reply Reply Quote 0
      • screechS
        screech
        last edited by screech

        Very nice ;)
        FYI : $$ donation aren't necessary ;) just contribut to the DB is enough to grant you several Thread ;)
        (We are missing lot's of sinopsys or media you can submit ;) )

        And of course you can just "add" the missing rom when you crape ;) in General menu, look at the scrape mode and select "update" ;)

        1 Reply Last reply Reply Quote 1
        • E
          EctoOne @screech
          last edited by

          @screech yeah i know, but I was scraping custom amiga uae files, so no hashes for me :)

          1 Reply Last reply Reply Quote 0
          • screechS
            screech
            last edited by

            @EctoOne try this :
            https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/Rom-Missing-on-Screenscraper

            Pass your uae file in F-CRC and send them to me ;) if they are well named, they will be autoassociated to there legitimate game ;) so you can scrape directly with CRC ;)

            E 1 Reply Last reply Reply Quote 0
            • hooperreH
              hooperre
              last edited by

              @screech Thanks for this man. It's an amazing tool and you've enhanced my RetroPie experience greatly. So thank you.

              Would creating a file directly from a URL to the screenscraper.fr database be difficult to incorporate for ROMs that don't pull easily? Sorry if this is a... novice question.

              4B ~ RPi PSU 5.1V / 3.0A ~ 32GB SanDisk microSD ~ 128GB USB

              1 Reply Last reply Reply Quote 0
              • screechS
                screech
                last edited by

                @hooperre sorry, my english must be too bad ^^ I don't understand your question :p

                hooperreH 1 Reply Last reply Reply Quote 0
                • hooperreH
                  hooperre @screech
                  last edited by hooperre

                  @screech I was wondering if you could add something that will make a MIX image simply by pasting a link to the ROM in the DB.

                  For example, for Donkey Kong 3, I would just paste "https://www.screenscraper.fr/gameinfos.php?plateforme=4&gameid=2181" into the XML Scraper and it would output the MIX image for that game.

                  4B ~ RPi PSU 5.1V / 3.0A ~ 32GB SanDisk microSD ~ 128GB USB

                  1 Reply Last reply Reply Quote 0
                  • screechS
                    screech
                    last edited by

                    unfortunatly, it's actually impossible...

                    To create a MIX, I need to catch the Media.
                    To Catch the Media, I need to have an API response from the DB.
                    To have an API response from the DB, I need to ask the API with a CRC32, MD5, SHA1, or Filename...

                    I can't with the Game ID...

                    But I can ask the Screenscraper boss to add this to the API ;) but I think it's not for tomorrow ;) (he is pretty busy right now ^^)

                    T 1 Reply Last reply Reply Quote 0
                    • E
                      EctoOne @screech
                      last edited by

                      @screech I don't think that would help in this case. UAE are just text config files and even with the automatic ways to create them i think they would be different from system to system. I mean it uses a template to create them and after that it already makes automatic changes to the controller name when you start them.. It was easier to batch rename them.

                      1 Reply Last reply Reply Quote 0
                      • screechS
                        screech
                        last edited by

                        Arg.. ok ^^ CRC on this is useless ^^ (not very in Amiga ^^)

                        1 Reply Last reply Reply Quote 0
                        • E
                          EctoOne
                          last edited by

                          Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.

                          1 Reply Last reply Reply Quote 1
                          • T
                            tmak @screech
                            last edited by

                            @screech

                            Hi Screech,

                            I am new tot he retropie community and love the way the universal xml 3 image art looks. My only problem is I have a MAC computer, and i can use the universal xml scraper. Do you have any suggestions on another way to quickly scrap this great image art to my retropie?

                            Thanks,

                            1 Reply Last reply Reply Quote 0
                            • D
                              Dochartaigh @screech
                              last edited by Dochartaigh

                              @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                              @Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)

                              So I gave up on NES, trying Genesis now. 1,707 ROMs from the No-Intro romset (totally standard). Turned CRC off and just going by filename to speed things up. AND it's reading directly off my hard drive. AND I joined the screenscraper site and logged in on the program. AND I contributed unique content (still only have 1 thread). AND the scraper site's server load is only 52% right now...

                              ...AND it's still going to take me 14 hours to scrape Sega Genesis/Mega Drive....

                              Sorry, but if this entire program hinged on a slow flaky website (can't register from a Mac running Google Chrome either FYI...have to use Safari) to scape from it's completely flawed I'm sorry to say.

                              P.S. AND the program just crashed like 3 times in a row... lol

                              FlyingTomahawkF P 2 Replies Last reply Reply Quote 0
                              • FlyingTomahawkF
                                FlyingTomahawk @Dochartaigh
                                last edited by

                                @Dochartaigh

                                Many ANDs in there, lol
                                Just start it before going to bed and next day all is done. The time will change. Yesterday when I was scraping my NES roms (750+) the time constantly changed and got shorter and shorter.

                                1 Reply Last reply Reply Quote 1
                                • P
                                  paradadf @Dochartaigh
                                  last edited by

                                  @Dochartaigh usung a filename isn't faster than the crc for small files.
                                  I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
                                  It can also be that your firewall is causing problems, don't know.

                                  1 Reply Last reply Reply Quote 0
                                  • screechS
                                    screech
                                    last edited by

                                    @Dochartaigh An other thing : don't look (too much) at the "estimated" time...
                                    It is really "wrong" at start...

                                    Just to check, launch a scrape, scrape about 10 roms and cancel, look at the Time per Rom .

                                    If it's between 5 and 30sec it's "ok"... you are at a normal speed (depending on your config/network/server load)

                                    Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...

                                    (To explain thread : The Main EXE grab the API return and send it to X engine to manage them in parrallel. X is the thread number. The Engine will grab media, create the MIX and generate a "small" XML. When scrape is finished (or cancel) it will merge all the small XML in a final gamelist.xml )

                                    D 1 Reply Last reply Reply Quote 0
                                    • S
                                      swandog
                                      last edited by

                                      Hi,

                                      Recently came across this program but I have two issues I can't find answers for.

                                      1. I've noticed on the Screenscraper website that it doesn't list multiple disk names.

                                      For example 'Another world' on the Amiga, it will scrape disk 1 but not disk 2.

                                      Anyway around this?

                                      1. On the roms that don't get scraped, do I need to rename these as per the rom names on the website for the scrapper to work?

                                      If not can they be manually added to the gamelist.xml file?

                                      Thanks

                                      D S 2 Replies Last reply Reply Quote 0
                                      • D
                                        Dochartaigh @swandog
                                        last edited by

                                        @paradadf said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                                        @Dochartaigh usung a filename isn't faster than the crc for small files.
                                        I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
                                        It can also be that your firewall is causing problems, don't know.

                                        I've tried it three ways: on Parallels inside OSX, on Windows Bootcamp (which is a native Windows partition on a Mac - NO different whatsoever than running Windows on a Windows PC), and on an old ASUS i3? dual-core laptop I had laying around. I can let it run for 30-60 minutes and it still always tells me something like 6 to 8 to 14+ hours to scrape a NES, SNES, or Sega Genesis romset. No firewall, heck, even turned off my VPN.

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          Dochartaigh @screech
                                          last edited by Dochartaigh

                                          @screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

                                          @Dochartaigh
                                          Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...

                                          Can you explain how to get more threads allowed with my account? I'm able to log into the site, validated my email address, and started to upload "printer spreads" of Sega Genesis manuals I made through a process called imposition (they're the only way to print out a manual in booklet format which will look IDENTICAL to the real Genesis/Mega Drive manual - once folded, cut, and stapled - and I haven't seen these online anywhere...), but can still only scrape 1 at a time...

                                          I also REALLY like the 3-image mix art....just can't get it to work and I'm trying pretty much everything (besides waiting for DAYS to scrape all my art...just not realistic for my needs).

                                          1 Reply Last reply Reply Quote 0
                                          • M
                                            moisessdm
                                            last edited by

                                            Hi, guys.

                                            I'm trying to make a MIX with my arcade roms with flyers. But I'm not finding the correct address of the flyers (Data/jeu/media/media _...) to be put in XML. Can you help me?

                                            P 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.