[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
@screech Thanks for this man. It's an amazing tool and you've enhanced my RetroPie experience greatly. So thank you.
Would creating a file directly from a URL to the screenscraper.fr database be difficult to incorporate for ROMs that don't pull easily? Sorry if this is a... novice question.
-
@hooperre sorry, my english must be too bad ^^ I don't understand your question :p
-
@screech I was wondering if you could add something that will make a MIX image simply by pasting a link to the ROM in the DB.
For example, for Donkey Kong 3, I would just paste "https://www.screenscraper.fr/gameinfos.php?plateforme=4&gameid=2181" into the XML Scraper and it would output the MIX image for that game.
-
unfortunatly, it's actually impossible...
To create a MIX, I need to catch the Media.
To Catch the Media, I need to have an API response from the DB.
To have an API response from the DB, I need to ask the API with a CRC32, MD5, SHA1, or Filename...I can't with the Game ID...
But I can ask the Screenscraper boss to add this to the API ;) but I think it's not for tomorrow ;) (he is pretty busy right now ^^)
-
@screech I don't think that would help in this case. UAE are just text config files and even with the automatic ways to create them i think they would be different from system to system. I mean it uses a template to create them and after that it already makes automatic changes to the controller name when you start them.. It was easier to batch rename them.
-
Arg.. ok ^^ CRC on this is useless ^^ (not very in Amiga ^^)
-
Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.
-
Hi Screech,
I am new tot he retropie community and love the way the universal xml 3 image art looks. My only problem is I have a MAC computer, and i can use the universal xml scraper. Do you have any suggestions on another way to quickly scrap this great image art to my retropie?
Thanks,
-
@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)
So I gave up on NES, trying Genesis now. 1,707 ROMs from the No-Intro romset (totally standard). Turned CRC off and just going by filename to speed things up. AND it's reading directly off my hard drive. AND I joined the screenscraper site and logged in on the program. AND I contributed unique content (still only have 1 thread). AND the scraper site's server load is only 52% right now...
...AND it's still going to take me 14 hours to scrape Sega Genesis/Mega Drive....
Sorry, but if this entire program hinged on a slow flaky website (can't register from a Mac running Google Chrome either FYI...have to use Safari) to scape from it's completely flawed I'm sorry to say.
P.S. AND the program just crashed like 3 times in a row... lol
-
Many ANDs in there, lol
Just start it before going to bed and next day all is done. The time will change. Yesterday when I was scraping my NES roms (750+) the time constantly changed and got shorter and shorter. -
@Dochartaigh usung a filename isn't faster than the crc for small files.
I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
It can also be that your firewall is causing problems, don't know. -
@Dochartaigh An other thing : don't look (too much) at the "estimated" time...
It is really "wrong" at start...Just to check, launch a scrape, scrape about 10 roms and cancel, look at the Time per Rom .
If it's between 5 and 30sec it's "ok"... you are at a normal speed (depending on your config/network/server load)
Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...
(To explain thread : The Main EXE grab the API return and send it to X engine to manage them in parrallel. X is the thread number. The Engine will grab media, create the MIX and generate a "small" XML. When scrape is finished (or cancel) it will merge all the small XML in a final gamelist.xml )
-
Hi,
Recently came across this program but I have two issues I can't find answers for.
- I've noticed on the Screenscraper website that it doesn't list multiple disk names.
For example 'Another world' on the Amiga, it will scrape disk 1 but not disk 2.
Anyway around this?
- On the roms that don't get scraped, do I need to rename these as per the rom names on the website for the scrapper to work?
If not can they be manually added to the gamelist.xml file?
Thanks
-
@paradadf said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh usung a filename isn't faster than the crc for small files.
I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
It can also be that your firewall is causing problems, don't know.I've tried it three ways: on Parallels inside OSX, on Windows Bootcamp (which is a native Windows partition on a Mac - NO different whatsoever than running Windows on a Windows PC), and on an old ASUS i3? dual-core laptop I had laying around. I can let it run for 30-60 minutes and it still always tells me something like 6 to 8 to 14+ hours to scrape a NES, SNES, or Sega Genesis romset. No firewall, heck, even turned off my VPN.
-
@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh
Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...Can you explain how to get more threads allowed with my account? I'm able to log into the site, validated my email address, and started to upload "printer spreads" of Sega Genesis manuals I made through a process called imposition (they're the only way to print out a manual in booklet format which will look IDENTICAL to the real Genesis/Mega Drive manual - once folded, cut, and stapled - and I haven't seen these online anywhere...), but can still only scrape 1 at a time...
I also REALLY like the 3-image mix art....just can't get it to work and I'm trying pretty much everything (besides waiting for DAYS to scrape all my art...just not realistic for my needs).
-
Hi, guys.
I'm trying to make a MIX with my arcade roms with flyers. But I'm not finding the correct address of the flyers (Data/jeu/media/media _...) to be put in XML. Can you help me?
-
@moisessdm I don't know what to change, but @screech will tell you that as soon as he reads this.
The only thing I can tell is that flyers are a very new media type so you'll probably first need to add them to the database before scraping. NeoGeo has a few, but general mame games don't. -
Hi @screech you seem to be the goto guy. Are you able to help me?
-
@swandog I missed your question.
You can find instructions to add new roms to the database here:
https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/Rom-Missing-on-ScreenscraperOnce they are added and associated (works automatically most of the time, although with empty systems it doesn't), you'll be able to scrape them.
-
@moisessdm I took a look at the web api and you'll probably find what you need there:
https://www.screenscraper.fr/webapi.php?alpha=0&numpage=0Just look for flyer
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.