[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
@Dochartaigh isn't there a profil called "roms path" or something? IMO that's the best location for the gamelist and images folder, next to your roms. That's also the location of the data when you use sselph scraper.
-
Hi guys, sorry, I were a little bit busy these last time... Not so much Time on UXS...
@Jeck11 : It's unfortunatly false positive. The language I use (Autoit) is mainly detected as malicious. I had contactated many AV editor to say it's a false positive. Lot's of them have answer and it's ok... But some haven't answer (yet ?) like Microsoft :S .
@Dukafleed Yep, it's in a very long todo list ^^ you can try : https://github.com/Universal-Rom-Tools/Universal-XML-Editor but it's "dedicated" to recalbox, maybe it works on retropie (but it's an early Beta ^^)
@BORIStheBLADE1 and others, sorry for the "lost connection" to the DB... When server is very "high" on CPU Use, it temporary "cut" the access to none register user... Try to Register to Screenscraper ;) maybe it can help .
@hooperre You are right... Must add a save function to the original XML.
For the Donkey Kong File. I think there is a probleme with the ' in the name... Need to check that...@FlyingTomahawk near 7h for 820 roms is pretty...long... Normally it can handle much more roms (someone scrape more than 6000 Amiga roms with success ;) ) Try to register and submit some proposition on www.screenscraper.fr to get some extra thread ;) it will be faster ;)
@Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)
(FYI : Thread are simultaneous scrape engine number... The main EXE send to X Engine the API return to manage it and download picture, create the mix, generate a part of the final XML. So normally, except if the server is very loaded, it must go pretty fast...)@Paradadf thank you very much taking the UXS support when I'm not here ^^. And you are right... Prefere the "Roms path" profile when you scrape on you PC... It put the downloaded_image folder and gamelist.xml directly in the roms folder ;)
-
@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@hooperre You are right... Must add a save function to the original XML.
For the Donkey Kong File. I think there is a probleme with the ' in the name... Need to check that...I noticed that last night too. Tried to scrape Chip's Challenge and whatever i tried to name it, it couldn't find it. Also there was some problem with names that have multiple . in them. Like B.A.T. or P. P. Hammer.. I got those working by accidentally renaming them to NAME.ipf.zip
-
Normally, the "good" process is to identify your rom with CRC... Filename is just a "fallback" feature to scrape as much as it can ;) And so it's not very "reliable"...
-
@screech
Thanks for your reply.
Today it was faster, lot faster than yesterday. I have an account at screenscraper.fr but I haven't contributed ($$) anything yet.
Is it possible to run the scrape again but let it only scrape the roms from the missing list? -
Very nice ;)
FYI : $$ donation aren't necessary ;) just contribut to the DB is enough to grant you several Thread ;)
(We are missing lot's of sinopsys or media you can submit ;) )And of course you can just "add" the missing rom when you crape ;) in General menu, look at the scrape mode and select "update" ;)
-
@screech yeah i know, but I was scraping custom amiga uae files, so no hashes for me :)
-
@EctoOne try this :
https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/Rom-Missing-on-ScreenscraperPass your uae file in F-CRC and send them to me ;) if they are well named, they will be autoassociated to there legitimate game ;) so you can scrape directly with CRC ;)
-
@screech Thanks for this man. It's an amazing tool and you've enhanced my RetroPie experience greatly. So thank you.
Would creating a file directly from a URL to the screenscraper.fr database be difficult to incorporate for ROMs that don't pull easily? Sorry if this is a... novice question.
-
@hooperre sorry, my english must be too bad ^^ I don't understand your question :p
-
@screech I was wondering if you could add something that will make a MIX image simply by pasting a link to the ROM in the DB.
For example, for Donkey Kong 3, I would just paste "https://www.screenscraper.fr/gameinfos.php?plateforme=4&gameid=2181" into the XML Scraper and it would output the MIX image for that game.
-
unfortunatly, it's actually impossible...
To create a MIX, I need to catch the Media.
To Catch the Media, I need to have an API response from the DB.
To have an API response from the DB, I need to ask the API with a CRC32, MD5, SHA1, or Filename...I can't with the Game ID...
But I can ask the Screenscraper boss to add this to the API ;) but I think it's not for tomorrow ;) (he is pretty busy right now ^^)
-
@screech I don't think that would help in this case. UAE are just text config files and even with the automatic ways to create them i think they would be different from system to system. I mean it uses a template to create them and after that it already makes automatic changes to the controller name when you start them.. It was easier to batch rename them.
-
Arg.. ok ^^ CRC on this is useless ^^ (not very in Amiga ^^)
-
Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.
-
Hi Screech,
I am new tot he retropie community and love the way the universal xml 3 image art looks. My only problem is I have a MAC computer, and i can use the universal xml scraper. Do you have any suggestions on another way to quickly scrap this great image art to my retropie?
Thanks,
-
@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)
So I gave up on NES, trying Genesis now. 1,707 ROMs from the No-Intro romset (totally standard). Turned CRC off and just going by filename to speed things up. AND it's reading directly off my hard drive. AND I joined the screenscraper site and logged in on the program. AND I contributed unique content (still only have 1 thread). AND the scraper site's server load is only 52% right now...
...AND it's still going to take me 14 hours to scrape Sega Genesis/Mega Drive....
Sorry, but if this entire program hinged on a slow flaky website (can't register from a Mac running Google Chrome either FYI...have to use Safari) to scape from it's completely flawed I'm sorry to say.
P.S. AND the program just crashed like 3 times in a row... lol
-
Many ANDs in there, lol
Just start it before going to bed and next day all is done. The time will change. Yesterday when I was scraping my NES roms (750+) the time constantly changed and got shorter and shorter. -
@Dochartaigh usung a filename isn't faster than the crc for small files.
I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
It can also be that your firewall is causing problems, don't know. -
@Dochartaigh An other thing : don't look (too much) at the "estimated" time...
It is really "wrong" at start...Just to check, launch a scrape, scrape about 10 roms and cancel, look at the Time per Rom .
If it's between 5 and 30sec it's "ok"... you are at a normal speed (depending on your config/network/server load)
Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...
(To explain thread : The Main EXE grab the API return and send it to X engine to manage them in parrallel. X is the thread number. The Engine will grab media, create the MIX and generate a "small" XML. When scrape is finished (or cancel) it will merge all the small XML in a final gamelist.xml )
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.