[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
After 6 hours and 54 minutes scrapping I have 820 Images and.... an empty gamelist.xml file. :-/ lol
I don't think it can handle more then a few roms at the time. I am trying now half of that amount to see if it'll work this time.
I'll know more in 3 hours and 3 minutes... -
I could have totally missed this, but I'm coming from a Mac and installed Parallels so I can use PC programs. To get this scraper to work your Windows PC needs to be able to see the Pi as a network drive, correct? How are you guys doing this?
On my Mac it's not setup as a network drive either - I either SSH in with the built-in Mac Terminal program, or use my FTP program (Cyberduck or FileZilla)...just seeing how you PC users would go about this the best way. All my searches are bringing up how to run ROMs from a Network drive - when I want to go the opposite way (view the Pi as a Network drive in Windows, not have the Pi see a network drive in PC/Windows land).
-
I have Emulationstation setup on Windows with the exact same system setup and roms as on my RPi. I then scrape the roms on PC (RetroPie roms MIX) by selecting the paths that are setup in Emulationstation PC.
And once done I adjust the paths inside the gamelist.xml file to match the folder structure of my RPi and then transfer the images and gamelist.xml files via network access to the RPi. -
@Dochartaigh sadly, UXS doesn't work on a virtual machine. Or at least, nobody has been able to make it work (yet).
-
@paradadf said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh sadly, UXS doesn't work on a virtual machine. Or at least, nobody has been able to make it work (yet).
Parallels isn't like a traditional virtual machine per se, it's using the mac's Bootcamp Windows (10) partition so you can run ANY Windows program as if you were running Windows diretly (including Universal XML Scraper).
...I just need to know how you guys are using Universal XML Scraper. Ignore that I'm on a Mac. How do you setup your Pi as a network drive in Windows? -- that's what needs to be done correct to get it to work, correct? I'm watching YouTube videos now so all will become self evident I'm sure...(maybe ;)
EDIT YouTube seems to say that RetroPie automatically shows up as a network drive in Windows if they're both connected to the same network? If so that would be the first time Windows makes something easier to use than my Mac ;) ...booting up now to see if I've missed that drive in Windows Explorer and will try to run the scraper (going to try the cool 3-image "MIX" option).
-
@Dochartaigh not sure if this is answering your question but to access the retropie shared folder on windows, type this address on the file explorer: "\retropie" without the quotes, if is not working use the IP address instead of "retropie"
-
Got it to work thanks.
Question: one of the biggest draws of this is that it's supposed to be faster than EmulationStation and SSelph's super-slow scrapers, right?
Why would it STILL take 8+ hours to scrape the No-Intro NES/Nintendo romset? Tried on both my Mac, and on a proper Windows 10 system. We're also talking a quad-core i7 processor with 16gb of ram, and an internet connection topping 135 mb/sec...
Even if I pay the 10 Euro fee for the scraper site where I can scrape 5 at once (that's a real thing, right? not some scam I saw on a couple YouTube videos, correct?) We're still looking at probably 24+ hours for my entire rom collection (Atari 2600/7800, master System, TG16, NES, SNES, Genesis, GB, GBC, GBA, Arcade, etc.).
Am I missing something?
-
I don't think this is faster than SSelph's scraper, the only reason I used UXS is because the mixed art, I love it.
UXS is very slow if you are scraping big files like ISO's, there is a way around here https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/ISO-Scrape-Freeze-or-are-very-long I like the option 3 "Experimental function".
-
@Dochartaigh the biggest problem is the server where it gets the data from. Sometimes it's very slow and doesn't find a thing. But I had the same problems with other scrapers/databases.
-
@Tasio said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
I don't think this is faster than SSelph's scraper, the only reason I used UXS is because the mixed art, I love it.
UXS is very slow if you are scraping big files like ISO's, there is a way around here https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/ISO-Scrape-Freeze-or-are-very-long I like the option 3 "Experimental function".
I turned off the CRC file check since that takes the longest and I'm using standard romsets we all use (i..e with standard filenames). It seemed to be going a LOT faster until I get the "problem connecting to the database" error (which has been going on for at least 3 days according to the complaint on GitHub), and/or the lovely "variable used without being declared" error that seems to pop up all the time.
@EctoOne said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh the biggest problem is the server where it gets the data from. Sometimes it's very slow and doesn't find a thing. But I had the same problems with other scrapers/databases.
This is a recurring problem for every single scraping system/database RetroPie uses. I just don't understand why ALL these can't work like Kodi's scraper does. Sure, that'll take like one single hour to scrape my 2,000 or so movies, but it's pretty flawless for well over a decade now (all the way WAY back to XBMC).
-
I also had the can't connect to the database issue yesterday after my first failed attempt to scrape 900+ roms. And I don't think it is a UXS software issue.
Still I restarted my PC and setup UXS again from scratch. Maybe it was just a coincidence but it worked after that.
After letting the PC scrape over night I woke up this morning and he scraped 850 of 947 roms. This time I went for the 3 images MIX instead of the 4 images MIX. All images are there, beautiful btw., and the gamelist has all the records inside.
One system down, a few more to go.Using these images with the simplebigart theme on my RPi Zero, set the scraped images to 900x600 size and they look awesome.
-
Your scraping speeds with UXS are way too low!
I'd recomend you to scrape on your computers and the copy de data. The difference is huge! And I suppose everyone has a backup on his pc, right? XD
Don't forget to set your login on the configuration tab. That speed up things too, and most importantly, it still scrapes despite of an overloaded server. Unregistered users are the first having problems. -
Thanks for the advice.
I did enter my login data and I did everything on my PC still after 800+ scraped roms I get an average of 23s per rom.
Does this has something to do with the image setup? Image size? Image MIX type? Location? (I'm on the other side of the earth ball.)
Eitherway for me this is way better than the scraper inside the RPi that you have to confirm each single scrape. And the MIX images are worth it even more. -
Is XML down at the moment?
-
@KillerQ The website is up but the CPU load is almost at 80% according to the stats on the site.
Edit: When it jumped over 80% it disabled the scrape function for unregistered users
-
Is there any way to get rid of the green, red and blue squares from the FoolScreen (Arcade) images?
The squares within the red circle.
-
@FlyingTomahawk Make a backup of FoolScreen (Arcade).zip in the Mix folder, Unzip it, open the config.xml in a text editor, delete the last 3 elements, zip all files, make sure it still has the FoolScreen (Arcade).zip name, put it back in the Mix folder, select another mix profil in UXS, select FoolFoolScreen (Arcade) again, scrape.
I just took a quick look, but that should work. If not i will look again
Edit: Or try this. its the same file with a renamed profil. Link Just put it in the Mix folder and select it in UXS.
-
It worked like a charm! Thank you!
This is getting better and better. Not only can you get very nice MIX images but you can edit those like you would edit a theme.
-
Several people have mentioned running this on their PC (with the ROMs on their PC) to speed up the process - then editing the gamelist XML files with the proper path for the Pi.
Where would the gamelist XML files be on your PC then? I'm assuming that when the Universal XML Scraper scrapes off the Pi directly it's changing the XML files in
/home/pi/.emulationstation/gamelists(and of course that directory isn't going to exist on my PC). - So what is your process to make this work (hopefully quicker!)? -
I personally installed EmulationStation on my PC and created the exact same folder structure as on my RPi.
All the roms are the same on my PC as they are on my RPi.
This is not necessary but it makes things a bit easier to keep overview later.
I then define the paths for roms, images and gamelists under Configuration -> Path configuration.
I scrape one system after the other so I redefine the paths every time for each system before scrapping.You can create any folder structure and define the paths for roms, images and gamelist to whatever you like and helps you to keep overview.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.