[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
Got it to work thanks.
Question: one of the biggest draws of this is that it's supposed to be faster than EmulationStation and SSelph's super-slow scrapers, right?
Why would it STILL take 8+ hours to scrape the No-Intro NES/Nintendo romset? Tried on both my Mac, and on a proper Windows 10 system. We're also talking a quad-core i7 processor with 16gb of ram, and an internet connection topping 135 mb/sec...
Even if I pay the 10 Euro fee for the scraper site where I can scrape 5 at once (that's a real thing, right? not some scam I saw on a couple YouTube videos, correct?) We're still looking at probably 24+ hours for my entire rom collection (Atari 2600/7800, master System, TG16, NES, SNES, Genesis, GB, GBC, GBA, Arcade, etc.).
Am I missing something?
-
I don't think this is faster than SSelph's scraper, the only reason I used UXS is because the mixed art, I love it.
UXS is very slow if you are scraping big files like ISO's, there is a way around here https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/ISO-Scrape-Freeze-or-are-very-long I like the option 3 "Experimental function".
-
@Dochartaigh the biggest problem is the server where it gets the data from. Sometimes it's very slow and doesn't find a thing. But I had the same problems with other scrapers/databases.
-
@Tasio said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
I don't think this is faster than SSelph's scraper, the only reason I used UXS is because the mixed art, I love it.
UXS is very slow if you are scraping big files like ISO's, there is a way around here https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/ISO-Scrape-Freeze-or-are-very-long I like the option 3 "Experimental function".
I turned off the CRC file check since that takes the longest and I'm using standard romsets we all use (i..e with standard filenames). It seemed to be going a LOT faster until I get the "problem connecting to the database" error (which has been going on for at least 3 days according to the complaint on GitHub), and/or the lovely "variable used without being declared" error that seems to pop up all the time.
@EctoOne said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh the biggest problem is the server where it gets the data from. Sometimes it's very slow and doesn't find a thing. But I had the same problems with other scrapers/databases.
This is a recurring problem for every single scraping system/database RetroPie uses. I just don't understand why ALL these can't work like Kodi's scraper does. Sure, that'll take like one single hour to scrape my 2,000 or so movies, but it's pretty flawless for well over a decade now (all the way WAY back to XBMC).
-
I also had the can't connect to the database issue yesterday after my first failed attempt to scrape 900+ roms. And I don't think it is a UXS software issue.
Still I restarted my PC and setup UXS again from scratch. Maybe it was just a coincidence but it worked after that.
After letting the PC scrape over night I woke up this morning and he scraped 850 of 947 roms. This time I went for the 3 images MIX instead of the 4 images MIX. All images are there, beautiful btw., and the gamelist has all the records inside.
One system down, a few more to go.Using these images with the simplebigart theme on my RPi Zero, set the scraped images to 900x600 size and they look awesome.
-
Your scraping speeds with UXS are way too low!
I'd recomend you to scrape on your computers and the copy de data. The difference is huge! And I suppose everyone has a backup on his pc, right? XD
Don't forget to set your login on the configuration tab. That speed up things too, and most importantly, it still scrapes despite of an overloaded server. Unregistered users are the first having problems. -
Thanks for the advice.
I did enter my login data and I did everything on my PC still after 800+ scraped roms I get an average of 23s per rom.
Does this has something to do with the image setup? Image size? Image MIX type? Location? (I'm on the other side of the earth ball.)
Eitherway for me this is way better than the scraper inside the RPi that you have to confirm each single scrape. And the MIX images are worth it even more. -
Is XML down at the moment?
-
@KillerQ The website is up but the CPU load is almost at 80% according to the stats on the site.
Edit: When it jumped over 80% it disabled the scrape function for unregistered users
-
Is there any way to get rid of the green, red and blue squares from the FoolScreen (Arcade) images?
The squares within the red circle.
-
@FlyingTomahawk Make a backup of FoolScreen (Arcade).zip in the Mix folder, Unzip it, open the config.xml in a text editor, delete the last 3 elements, zip all files, make sure it still has the FoolScreen (Arcade).zip name, put it back in the Mix folder, select another mix profil in UXS, select FoolFoolScreen (Arcade) again, scrape.
I just took a quick look, but that should work. If not i will look again
Edit: Or try this. its the same file with a renamed profil. Link Just put it in the Mix folder and select it in UXS.
-
It worked like a charm! Thank you!
This is getting better and better. Not only can you get very nice MIX images but you can edit those like you would edit a theme.
-
Several people have mentioned running this on their PC (with the ROMs on their PC) to speed up the process - then editing the gamelist XML files with the proper path for the Pi.
Where would the gamelist XML files be on your PC then? I'm assuming that when the Universal XML Scraper scrapes off the Pi directly it's changing the XML files in
/home/pi/.emulationstation/gamelists(and of course that directory isn't going to exist on my PC). - So what is your process to make this work (hopefully quicker!)? -
I personally installed EmulationStation on my PC and created the exact same folder structure as on my RPi.
All the roms are the same on my PC as they are on my RPi.
This is not necessary but it makes things a bit easier to keep overview later.
I then define the paths for roms, images and gamelists under Configuration -> Path configuration.
I scrape one system after the other so I redefine the paths every time for each system before scrapping.You can create any folder structure and define the paths for roms, images and gamelist to whatever you like and helps you to keep overview.
-
@Dochartaigh isn't there a profil called "roms path" or something? IMO that's the best location for the gamelist and images folder, next to your roms. That's also the location of the data when you use sselph scraper.
-
Hi guys, sorry, I were a little bit busy these last time... Not so much Time on UXS...
@Jeck11 : It's unfortunatly false positive. The language I use (Autoit) is mainly detected as malicious. I had contactated many AV editor to say it's a false positive. Lot's of them have answer and it's ok... But some haven't answer (yet ?) like Microsoft :S .
@Dukafleed Yep, it's in a very long todo list ^^ you can try : https://github.com/Universal-Rom-Tools/Universal-XML-Editor but it's "dedicated" to recalbox, maybe it works on retropie (but it's an early Beta ^^)
@BORIStheBLADE1 and others, sorry for the "lost connection" to the DB... When server is very "high" on CPU Use, it temporary "cut" the access to none register user... Try to Register to Screenscraper ;) maybe it can help .
@hooperre You are right... Must add a save function to the original XML.
For the Donkey Kong File. I think there is a probleme with the ' in the name... Need to check that...@FlyingTomahawk near 7h for 820 roms is pretty...long... Normally it can handle much more roms (someone scrape more than 6000 Amiga roms with success ;) ) Try to register and submit some proposition on www.screenscraper.fr to get some extra thread ;) it will be faster ;)
@Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)
(FYI : Thread are simultaneous scrape engine number... The main EXE send to X Engine the API return to manage it and download picture, create the mix, generate a part of the final XML. So normally, except if the server is very loaded, it must go pretty fast...)@Paradadf thank you very much taking the UXS support when I'm not here ^^. And you are right... Prefere the "Roms path" profile when you scrape on you PC... It put the downloaded_image folder and gamelist.xml directly in the roms folder ;)
-
@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@hooperre You are right... Must add a save function to the original XML.
For the Donkey Kong File. I think there is a probleme with the ' in the name... Need to check that...I noticed that last night too. Tried to scrape Chip's Challenge and whatever i tried to name it, it couldn't find it. Also there was some problem with names that have multiple . in them. Like B.A.T. or P. P. Hammer.. I got those working by accidentally renaming them to NAME.ipf.zip
-
Normally, the "good" process is to identify your rom with CRC... Filename is just a "fallback" feature to scrape as much as it can ;) And so it's not very "reliable"...
-
@screech
Thanks for your reply.
Today it was faster, lot faster than yesterday. I have an account at screenscraper.fr but I haven't contributed ($$) anything yet.
Is it possible to run the scrape again but let it only scrape the roms from the missing list? -
Very nice ;)
FYI : $$ donation aren't necessary ;) just contribut to the DB is enough to grant you several Thread ;)
(We are missing lot's of sinopsys or media you can submit ;) )And of course you can just "add" the missing rom when you crape ;) in General menu, look at the scrape mode and select "update" ;)
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.