[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

mattrixk

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Can you send me your new "zip" so I can check what's going on ?

I've put it in my dropbox here.

there is a problem with XML (I think a wrong tag or something like that)

I made a copy of the existing Standard (3img) and literally the only thing I changed was the <Profil> name to match the folder name.

hansolo77

@vbs I was wondering this as well. There is an option to UPDATE, but according to the program it looks like it just adds new ROMs... it doesn't actually UPDATE the metadata/art. I think a feature that should be added would be some extra data in an xml or something that identifies what data was scraped. Then the next time you run the scrape, it compares what's online with what it already has. That will cut down on all the re-creating identical data, wasting a lot of time. Things you could get would be all the metadata fields (and have the "found.xml" file record with 1's and 0's (like gamename/publisher=1 then gamename/description=0 and it'll skip adding publisher data but get the description if it exists), and the artwork (if using MIX+3 or MIX+4, just have it again record 1's and 0's if it has it/doesn't). After comparing, it would get the new data, update the gamelist.xml as needed, and recreate any new artwork. I could probably write out all that in BASIC (the only programming language I know lol). So I can see this easily being implemented into the UXS program.

paradadf

I believe, without real understanding of how UXS works, that pulling data from the server doesn't that any considerable amount of time, but creating the mix images. I doubt that comparing anything with the db will be faster than just downloading the whole data.

hansolo77

My biggest slowdown when scraping is all the hash-checking it does. There should at least be a file that stores all the hash numbers so it doesn't have to re-hash every time.

paradadf

I don't know how big your roms are (what system), but hashing a regular (not cd based) rom doesn't take any longer than 0,1 s per file.

hansolo77

I'm having a really hard time with this now.
(refer to this post)
I think UXS has some how corrupted all my gamelist.xml files. Probably because yesterday I was having trouble getting the new version to work with the correct paths. I suspect it has multiple copies of them somewhere, and it's throwing everything out of whack.

But yea, my Amstrad CPC, Atari 800, and Atari St systems all took upwards of 4 hours each to hash, and they're no bigger than 1.2mb at the most (typical 3.5in floppy).

vbs

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

I'm having a really hard time with this now.
(refer to this post)
I think UXS has some how corrupted all my gamelist.xml files. Probably because yesterday I was having trouble getting the new version to work with the correct paths. I suspect it has multiple copies of them somewhere, and it's throwing everything out of whack

Make sure to delete all the partially created gamelist files in all the Rom folders!

paradadf

I'd put the launcher on a new folder and try again. Do you scrape over the network, right? To be honest, I've never done than because the pi is too slow imo. I'm scraping ST right now on my pc with only one thread and with the 3img profil is taking about 30 min for 305 roms.
My results:
alt text

hansolo77

I need to either find an alternative rom set or go through and erase a lot of junk ROMs. My AtariST set has 3260 files. I got this set from my GameBase collection on my PC. It took 4 hours to scrape yesterday.

Rion

@hansolo77 I think it would be best to Scrape your collection on you PC and then copy over the images to your pi.

So make a copy of your roms folder on to the pc.

Or you could like i do run everything of my USB stick. Rom folder only that is.

chuzzwuzzer

I'm having real problems with this scraper for some reason. I setup the autocofiguration on a new retropie 4.1 install. Autoconfiguration path : \retropie\roms. Kill emulation station. Select your system. Choose the system and click the main 'scrape' burron on the bottom right it asks for the sytem again. Then it tells me 'rom path cant be found' so I change the path to the standard roms directory manually. Now its just telling me 'Hashing please wait' which is taking forever. What am I doing wrong here ?

paradadf

I'd say not waiting XD. The hashing of big files takes long, nothing to do.

chuzzwuzzer

If I have the 'downloaded images' folder in each system rom folder backed up from a previous scrape can I just copy and paste 'downloaded images' into the corresponding rom folders of a new install of retropie . Will it automatically recognise the images or do I have to mess about editing paths etc

paradadf

You need to copy both, the downloaded_images folder AND the gamelist.xml file. Don't forget to stop EmulationStation before that. But you must put them inside their original folders, otherwise you'll need to change their paths, which is actually not so difficult to do.

chuzzwuzzer

@paradadf Many thanks for the help. I have an old install on another sd card. I have every system scraped. I can see the images \configs\all\emulationstation\downloaded_images but they dont show in es. I have copied the gamelist.xml for each system and the images into the new install on the other sd card and it just reproduces the same problem with the images not showing ? Is the problem somehow with the xml files ?

paradadf

@chuzzwuzzer i'm sorry but I can't give any advice about that because I don't use retropie. Maybe if make a pastebin an upload some lines of your xml someone else can see what's the problem.

illuminerdi

Is there any way to force it to use certain pictures or override the automatic picture selection?

For example: I want it to use the (super cool) US "Box Art" for Clock Tower (SNES). Clock Tower was never released in the US for the SNES, but I have a translated rom. As a result, UXS grabs the Japanese box art for the rom instead of the US box art.

Do I just need to rename the rom from "Clock Tower (J)" to "Clock Tower (U)" to fake out the scraper, or is there some other method to force it to always grab US boxes first and then roll over to Japanese if US isn't available?

I thought about just manually replacing the box art file, but I actually couldn't figure out how to download the box art from screenscraper.fr - is there some way to download the full size images? The links to box arts aren't clickable links :(

Also lastly is there any way to see the status of your uploads to Screenscraper.fr? I uploaded a box scan from my original copy of Soul Blazer (US), but it hasn't been added yet, and I'm not sure if I did the scan wrong - I couldn't find any instructions on how to do a raw box scan, such as if the scan should include the spine or back of the box, etc, so I just did a front scan and cropped it to fit.

Thanks - I love your database, it's 10x better than most of the other ones out there, I hope to see it continue to improve.

screech

@chuzzwuzzer Look to the path to your image in the gamelist.xml, and check if they are good ;)

@illuminerdi : lot's of things ^^
First : Rename your rom before scrape isn't a good idea. It check the HASH of your file... and renaming doesn't change the Hash ^^
Second : If the rom you have is referenced, maybe it's not referenced as a 'US' one... You can submit the country of a rom ;) So UXS can match the rom you have, the country of the rom, and so the media to take.

For your submission about the boxart, I need to check ;) I know they're some we don't know what to do with. Because it's not the "exact" format ;) I Keep you in touch ;)

For information : Screenscraper V2 will come (very soon ;) with Lot's of new "functions" :
You can see your submission and the stat, when they are refused, we can put a comment to say "why", you can make a comment to a user (for admin to guide you). A lot's of new 'country' possible for media and data (so we can have brazilian or spanish cover ;) ) and lot's more new stuff ;)

illuminerdi

@screech thanks for the reply. I'm still a bit confused though: if I add a translation of a ROM to the list and set its country as USA, wouldn't that be a problem since, for example, an English speaker in Europe might want the PAL version of the box art, but now it's marked as USA and they would get the US version of the box?

The Rom I'm using probably matches the hash of "Clock Tower (J) [T Eng1.01_Aeon Genesis][h1].zip" since that's the translation that I'm using. In screenscraper that ROM currently has no country associated with it. So I could associate it with USA, but then someone in the UK, for example, would get the US box when using the same rom, since the rom is based on language and not country. Technically there is no "PAL" version of the ROM.

Perhaps you should consider adding a "Translation" field to the Country selection so that when a romhash matches a translation it defaults to grabbing the box art for the user's country/region?

Clock tower is a great example of this - the original ROM was probably NTSC, there is no PAL version of the rom since all dumps would be based on the Japanese NTSC cart, and applying the English translation would produce an NTSC rom in English, which is probably the ROM that a UK (or other English-speaking European) would use when playing the game.

In screenscraper there are both EU and NA box arts made by fans. Neither one is the actual box art, since there are no EU and NA native versions of the roms, but they still look very nice and I would rather use these "fan boxes" than the Japanese boxes, for cohesion, and I think I'm not alone here.

Sorry if I've opened up pandora's box with this...

hansolo77

Why not just temporarily replace the artwork locally with what you want.