[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

Dochartaigh

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Dochartaigh 8+ hours for a NES romset with your config and 5 threads is pretty weird... normally a rom is scraped between 7 and 30s (depending on Server load and you config...)

So I gave up on NES, trying Genesis now. 1,707 ROMs from the No-Intro romset (totally standard). Turned CRC off and just going by filename to speed things up. AND it's reading directly off my hard drive. AND I joined the screenscraper site and logged in on the program. AND I contributed unique content (still only have 1 thread). AND the scraper site's server load is only 52% right now...

...AND it's still going to take me 14 hours to scrape Sega Genesis/Mega Drive....

Sorry, but if this entire program hinged on a slow flaky website (can't register from a Mac running Google Chrome either FYI...have to use Safari) to scape from it's completely flawed I'm sorry to say.

P.S. AND the program just crashed like 3 times in a row... lol

FlyingTomahawk

@Dochartaigh

Many ANDs in there, lol
Just start it before going to bed and next day all is done. The time will change. Yesterday when I was scraping my NES roms (750+) the time constantly changed and got shorter and shorter.

paradadf

@Dochartaigh usung a filename isn't faster than the crc for small files.
I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
It can also be that your firewall is causing problems, don't know.

screech

@Dochartaigh An other thing : don't look (too much) at the "estimated" time...
It is really "wrong" at start...

Just to check, launch a scrape, scrape about 10 roms and cancel, look at the Time per Rom .

If it's between 5 and 30sec it's "ok"... you are at a normal speed (depending on your config/network/server load)

Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...

(To explain thread : The Main EXE grab the API return and send it to X engine to manage them in parrallel. X is the thread number. The Engine will grab media, create the MIX and generate a "small" XML. When scrape is finished (or cancel) it will merge all the small XML in a final gamelist.xml )

swandog

Hi,

Recently came across this program but I have two issues I can't find answers for.

I've noticed on the Screenscraper website that it doesn't list multiple disk names.

For example 'Another world' on the Amiga, it will scrape disk 1 but not disk 2.

Anyway around this?

On the roms that don't get scraped, do I need to rename these as per the rom names on the website for the scrapper to work?

If not can they be manually added to the gamelist.xml file?

Thanks

Dochartaigh

@paradadf said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Dochartaigh usung a filename isn't faster than the crc for small files.
I really believe your instabilities have something to do wth you running UXS on paralell. I would try it on a normal windows installation to compare.
It can also be that your firewall is causing problems, don't know.

I've tried it three ways: on Parallels inside OSX, on Windows Bootcamp (which is a native Windows partition on a Mac - NO different whatsoever than running Windows on a Windows PC), and on an old ASUS i3? dual-core laptop I had laying around. I can let it run for 30-60 minutes and it still always tells me something like 6 to 8 to 14+ hours to scrape a NES, SNES, or Sega Genesis romset. No firewall, heck, even turned off my VPN.

Dochartaigh

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Dochartaigh
Try with more thread (just make some proposal on www.screenscraper.fr, wait for the Moderation validation, and put your ID in UXS) It will be really "fast" with just 2 or 3 threads...

Can you explain how to get more threads allowed with my account? I'm able to log into the site, validated my email address, and started to upload "printer spreads" of Sega Genesis manuals I made through a process called imposition (they're the only way to print out a manual in booklet format which will look IDENTICAL to the real Genesis/Mega Drive manual - once folded, cut, and stapled - and I haven't seen these online anywhere...), but can still only scrape 1 at a time...

I also REALLY like the 3-image mix art....just can't get it to work and I'm trying pretty much everything (besides waiting for DAYS to scrape all my art...just not realistic for my needs).

moisessdm

Hi, guys.

I'm trying to make a MIX with my arcade roms with flyers. But I'm not finding the correct address of the flyers (Data/jeu/media/media _...) to be put in XML. Can you help me?

paradadf

@moisessdm I don't know what to change, but @screech will tell you that as soon as he reads this.
The only thing I can tell is that flyers are a very new media type so you'll probably first need to add them to the database before scraping. NeoGeo has a few, but general mame games don't.

swandog

Hi @screech you seem to be the goto guy. Are you able to help me?

paradadf

@swandog I missed your question.
You can find instructions to add new roms to the database here:
https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/Rom-Missing-on-Screenscraper

Once they are added and associated (works automatically most of the time, although with empty systems it doesn't), you'll be able to scrape them.

paradadf

@moisessdm I took a look at the web api and you'll probably find what you need there:
https://www.screenscraper.fr/webapi.php?alpha=0&numpage=0

Just look for flyer

swandog

@paradadf thanks. Will have a good read tomorrow.

Do you know how to sort the multiple disk issue or is that covered in the article too?

moisessdm

@paradadf Hi. I saw this link. The problem is that I'm not finding the address to put in the MIX XML. I tried "<Source_Value> Date/jeu/media/media_flyer/media_flyer_% COUNTRY% </ Source_Value>", for example, and did not work.

paradadf

@moisessdm what about xx instead lf %country%?

moisessdm

@paradadf According to the API documentation, xx must be replaced by a variable (xx = "nomcourt" variable of the regionsListe.php API). The% COUNTRY% is a variable to say that the flyer will be downloaded according to the ROM region. I believe the error is in the directory where the images of the flyers are. The directory I'm putting in the XML is wrong or does not yet exist.

Furluge

Is there a way to get it to download png for the images when using the standard profile and downloading video? Right now if you change it from default to png or jpg it will save your video files as jpg or png, and default defaults to jpg. I would like to be able to save images and save 2d box art and wheels as pngs. Obviously you could rename the incorrect extension video files from png or jpg back to mp4 but that isn't ideal.

mattrixk

@Furluge There is an option in the Scrape Profile called <Source_Download_Ext> so for videos you can set <Source_Download_Ext>mp4</Source_Download_Ext>, for logos and box-art you can set <Source_Download_Ext>png</Source_Download_Ext> and for screenshots you can set <Source_Download_Ext>jpg</Source_Download_Ext>

This PasteBin dump is a work-in-progress Scrape Profile I made. It does this:

scrapes the video as mp4, appends "-video" to the filename and dumps it into the <video> tag within the gamelist.xml so it comes out as <video>/path-to-media/aladdin-video.mp4</video>.
scrapes the screenshot as jpg, appends "-screenshot" to the filename and dumps it into the <image> tag within the gamelist.xml so it comes out as <image>/path-to-media/aladdin-screenshot.jpg</image>
scrapes the select MIX image as png, appends "-mix" to the filename and dumps it into the <marquee> tag within the gamelist.xml so it comes out as <marquee>/path-to-media/aladdin-mix.png</marquee>

I designed the MIX image to scrape the logo, box-art and cartridge image, and then format them to a shape that sits around the screenshot. I think it looks good. You can see that MIX Profile here. It's also a work in progress.

The Scrape and MIX Profiles listed above worked great in UXS v5, but I haven't tested them in the new v6, so I make no promises.

derFeef

Hello @screech!
Thanks for the awesome tool and your work!
Now I am having a problem hence why I am writing ;)

Everything is working great except... Emulationstation is not showing me the information or images.

I do.

Boot retropie
Quit ES
Run your tool
Scrape (works like a charm).
The images and xml files are in the correct folder and everything.
restart ES
No images or information is shown :(

I noticed that the XML file your tool creates and the XML file the standard ES scraper is creating have a very different layout. Can this be the problem or am I missing something?

Thanks!

FlyingTomahawk

@screech

You said support is not only by money but also by uploading images, logos or anything that might help to improve the database.
Well, I did upload a couple of company logos and system logos.
Now it looks like they accepted the logos but it says Terscreenator as uploader.
~~I think credit should be given if due unless it is technically not possible to do and only those Top 3-4 guys can upload stuff.~~
UPDATE: turned out that it was a minor bug that has been taken care off by the screenscraper crew. Awesome guys over there, very quick and friendly.