[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

hansolo77

I'm having a really hard time with this now.
(refer to this post)
I think UXS has some how corrupted all my gamelist.xml files. Probably because yesterday I was having trouble getting the new version to work with the correct paths. I suspect it has multiple copies of them somewhere, and it's throwing everything out of whack.

But yea, my Amstrad CPC, Atari 800, and Atari St systems all took upwards of 4 hours each to hash, and they're no bigger than 1.2mb at the most (typical 3.5in floppy).

vbs

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

I'm having a really hard time with this now.
(refer to this post)
I think UXS has some how corrupted all my gamelist.xml files. Probably because yesterday I was having trouble getting the new version to work with the correct paths. I suspect it has multiple copies of them somewhere, and it's throwing everything out of whack

Make sure to delete all the partially created gamelist files in all the Rom folders!

paradadf

I'd put the launcher on a new folder and try again. Do you scrape over the network, right? To be honest, I've never done than because the pi is too slow imo. I'm scraping ST right now on my pc with only one thread and with the 3img profil is taking about 30 min for 305 roms.
My results:
alt text

hansolo77

I need to either find an alternative rom set or go through and erase a lot of junk ROMs. My AtariST set has 3260 files. I got this set from my GameBase collection on my PC. It took 4 hours to scrape yesterday.

Rion

@hansolo77 I think it would be best to Scrape your collection on you PC and then copy over the images to your pi.

So make a copy of your roms folder on to the pc.

Or you could like i do run everything of my USB stick. Rom folder only that is.

chuzzwuzzer

I'm having real problems with this scraper for some reason. I setup the autocofiguration on a new retropie 4.1 install. Autoconfiguration path : \retropie\roms. Kill emulation station. Select your system. Choose the system and click the main 'scrape' burron on the bottom right it asks for the sytem again. Then it tells me 'rom path cant be found' so I change the path to the standard roms directory manually. Now its just telling me 'Hashing please wait' which is taking forever. What am I doing wrong here ?

paradadf

I'd say not waiting XD. The hashing of big files takes long, nothing to do.

chuzzwuzzer

If I have the 'downloaded images' folder in each system rom folder backed up from a previous scrape can I just copy and paste 'downloaded images' into the corresponding rom folders of a new install of retropie . Will it automatically recognise the images or do I have to mess about editing paths etc

paradadf

You need to copy both, the downloaded_images folder AND the gamelist.xml file. Don't forget to stop EmulationStation before that. But you must put them inside their original folders, otherwise you'll need to change their paths, which is actually not so difficult to do.

chuzzwuzzer

@paradadf Many thanks for the help. I have an old install on another sd card. I have every system scraped. I can see the images \configs\all\emulationstation\downloaded_images but they dont show in es. I have copied the gamelist.xml for each system and the images into the new install on the other sd card and it just reproduces the same problem with the images not showing ? Is the problem somehow with the xml files ?

paradadf

@chuzzwuzzer i'm sorry but I can't give any advice about that because I don't use retropie. Maybe if make a pastebin an upload some lines of your xml someone else can see what's the problem.

illuminerdi

Is there any way to force it to use certain pictures or override the automatic picture selection?

For example: I want it to use the (super cool) US "Box Art" for Clock Tower (SNES). Clock Tower was never released in the US for the SNES, but I have a translated rom. As a result, UXS grabs the Japanese box art for the rom instead of the US box art.

Do I just need to rename the rom from "Clock Tower (J)" to "Clock Tower (U)" to fake out the scraper, or is there some other method to force it to always grab US boxes first and then roll over to Japanese if US isn't available?

I thought about just manually replacing the box art file, but I actually couldn't figure out how to download the box art from screenscraper.fr - is there some way to download the full size images? The links to box arts aren't clickable links :(

Also lastly is there any way to see the status of your uploads to Screenscraper.fr? I uploaded a box scan from my original copy of Soul Blazer (US), but it hasn't been added yet, and I'm not sure if I did the scan wrong - I couldn't find any instructions on how to do a raw box scan, such as if the scan should include the spine or back of the box, etc, so I just did a front scan and cropped it to fit.

Thanks - I love your database, it's 10x better than most of the other ones out there, I hope to see it continue to improve.

screech

@chuzzwuzzer Look to the path to your image in the gamelist.xml, and check if they are good ;)

@illuminerdi : lot's of things ^^
First : Rename your rom before scrape isn't a good idea. It check the HASH of your file... and renaming doesn't change the Hash ^^
Second : If the rom you have is referenced, maybe it's not referenced as a 'US' one... You can submit the country of a rom ;) So UXS can match the rom you have, the country of the rom, and so the media to take.

For your submission about the boxart, I need to check ;) I know they're some we don't know what to do with. Because it's not the "exact" format ;) I Keep you in touch ;)

For information : Screenscraper V2 will come (very soon ;) with Lot's of new "functions" :
You can see your submission and the stat, when they are refused, we can put a comment to say "why", you can make a comment to a user (for admin to guide you). A lot's of new 'country' possible for media and data (so we can have brazilian or spanish cover ;) ) and lot's more new stuff ;)

illuminerdi

@screech thanks for the reply. I'm still a bit confused though: if I add a translation of a ROM to the list and set its country as USA, wouldn't that be a problem since, for example, an English speaker in Europe might want the PAL version of the box art, but now it's marked as USA and they would get the US version of the box?

The Rom I'm using probably matches the hash of "Clock Tower (J) [T Eng1.01_Aeon Genesis][h1].zip" since that's the translation that I'm using. In screenscraper that ROM currently has no country associated with it. So I could associate it with USA, but then someone in the UK, for example, would get the US box when using the same rom, since the rom is based on language and not country. Technically there is no "PAL" version of the ROM.

Perhaps you should consider adding a "Translation" field to the Country selection so that when a romhash matches a translation it defaults to grabbing the box art for the user's country/region?

Clock tower is a great example of this - the original ROM was probably NTSC, there is no PAL version of the rom since all dumps would be based on the Japanese NTSC cart, and applying the English translation would produce an NTSC rom in English, which is probably the ROM that a UK (or other English-speaking European) would use when playing the game.

In screenscraper there are both EU and NA box arts made by fans. Neither one is the actual box art, since there are no EU and NA native versions of the roms, but they still look very nice and I would rather use these "fan boxes" than the Japanese boxes, for cohesion, and I think I'm not alone here.

Sorry if I've opened up pandora's box with this...

hansolo77

Why not just temporarily replace the artwork locally with what you want.

mattrixk

Hi @screech, have you had a chance to look into the XML problems I had when creating a new MIX Image Profile?

hansolo77

I still think UXS is creating incompatible gamelist.xml files. If I start completly fresh, everything works like always. Soon as I go an use UXS, things start messing up. The ES game lists start looking different (the names), I can't scrape with Selph's to append missing data, booting ES from a cold start takes FOREVER. Long as I don't throw UXS into the mix, things just work better. I really like using the 4-MIX artwork though, since Selph's and the internal scrapers only grab screenshots, and ES has no method of letting you view extra art. So I think I'm just going to forego using UXS for the time being.

paradadf

@hansolo77 it is weird that UXS is working for everyone else but you. I suppose you have a backup of your roms on your pc... why don't you try scrapping them there and just copying the files over. You are not using a VM, right?

hansolo77

By VM you mean "Virtual Machine" right? No I'm not.

Yeah I can't explain why I'm having difficulties. I mean, UXS works.. it scrapes my systems, builds gamelist.xml files, creates artwork that does display in ES. The problem I'm having is when I go to try and use Selph's Scraper after the fact, and it throws out all kinds of errors about not being able to find the original paths. I assumed this is because it's pulling up the existing xml and using that as a basis for figuring out what has already been scraped. The other issue I reported in the other thread comes from when ES is shutting down, where it shows a bunch of errors about not being able to find the gamelist nodes. This is probably due to ES being configured to save the gamelists on exit, and UXS's generated ones aren't compatible or something. For instance.. does the UXS scraper include fields in the xml for data like last played? I just checked my Selph's generated xml for WonderSwan... Here is how it's setup:

<?xml version="1.0"?>
<gameList>
	<game>
		<path>./Dokodemo Hamster (Japan).ws</path>
		<name>Dokodemo Hamster (Japan)</name>
		<playcount>1</playcount>
		<lastplayed>20161217T061718</lastplayed>
	</game>
</gameList>

UXS's generated:

<gamelist>
<game id="46248" source="ScreenScraper">
	<path>/home/pi/RetroPie/roms/wonderswan/Anchorz Field (Japan).ws</path>
	<name>Anchorz Field (Japan)</name>
	<desc>Anchorz Field is a Strategy game, developed and published by Sammy Studios, which was released in Japan in 1999.</desc>
	<image>/home/pi/.emulationstation/downloaded_images/wonderswan/Anchorz Field (Japan)-image.png</image>
	<thumbnail/>
	<rating>1</rating>
	<releasedate>00000000T000000</releasedate>
	<developer>Sammy Studios</developer>
	<publisher>Sammy Studios</publisher>
	<genre>Strategy</genre>
	<players>2</players>
</game>
</gamelist>

By just comparing the two, I can already see a number of differences. First of all UXS isn't writing out the <?xml version="1.0"?> line at the top. Secondly, it's using a different indentation. Thirdly, the path to the ROM is different, where Selphs is using a ./(location) path, but UXS is using a full path. So when Selph is trying to re-scan, it's not using the the format it knows, and can't find the path. So I think these things are what is causing some of my problems. Without the proper format, ES doesn't have the ability to update and save the gamelist on exit, and Selph is unable to update the list because the paths are different.

paradadf

@hansolo77 have you try the profile RetroPie [Sselph Rom] or something? I believe you mist use that profile and store everything next to your roms, as sselph scraper does, if you want to be able to use both scrapers. Otherwise UXS created one gamelist.xml somewhere else and sselph scraper tries to create a different one next to your roms, this the conflict.