[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

hansolo77

By VM you mean "Virtual Machine" right? No I'm not.

Yeah I can't explain why I'm having difficulties. I mean, UXS works.. it scrapes my systems, builds gamelist.xml files, creates artwork that does display in ES. The problem I'm having is when I go to try and use Selph's Scraper after the fact, and it throws out all kinds of errors about not being able to find the original paths. I assumed this is because it's pulling up the existing xml and using that as a basis for figuring out what has already been scraped. The other issue I reported in the other thread comes from when ES is shutting down, where it shows a bunch of errors about not being able to find the gamelist nodes. This is probably due to ES being configured to save the gamelists on exit, and UXS's generated ones aren't compatible or something. For instance.. does the UXS scraper include fields in the xml for data like last played? I just checked my Selph's generated xml for WonderSwan... Here is how it's setup:

<?xml version="1.0"?>
<gameList>
	<game>
		<path>./Dokodemo Hamster (Japan).ws</path>
		<name>Dokodemo Hamster (Japan)</name>
		<playcount>1</playcount>
		<lastplayed>20161217T061718</lastplayed>
	</game>
</gameList>

UXS's generated:

<gamelist>
<game id="46248" source="ScreenScraper">
	<path>/home/pi/RetroPie/roms/wonderswan/Anchorz Field (Japan).ws</path>
	<name>Anchorz Field (Japan)</name>
	<desc>Anchorz Field is a Strategy game, developed and published by Sammy Studios, which was released in Japan in 1999.</desc>
	<image>/home/pi/.emulationstation/downloaded_images/wonderswan/Anchorz Field (Japan)-image.png</image>
	<thumbnail/>
	<rating>1</rating>
	<releasedate>00000000T000000</releasedate>
	<developer>Sammy Studios</developer>
	<publisher>Sammy Studios</publisher>
	<genre>Strategy</genre>
	<players>2</players>
</game>
</gamelist>

By just comparing the two, I can already see a number of differences. First of all UXS isn't writing out the <?xml version="1.0"?> line at the top. Secondly, it's using a different indentation. Thirdly, the path to the ROM is different, where Selphs is using a ./(location) path, but UXS is using a full path. So when Selph is trying to re-scan, it's not using the the format it knows, and can't find the path. So I think these things are what is causing some of my problems. Without the proper format, ES doesn't have the ability to update and save the gamelist on exit, and Selph is unable to update the list because the paths are different.

paradadf

@hansolo77 have you try the profile RetroPie [Sselph Rom] or something? I believe you mist use that profile and store everything next to your roms, as sselph scraper does, if you want to be able to use both scrapers. Otherwise UXS created one gamelist.xml somewhere else and sselph scraper tries to create a different one next to your roms, this the conflict.

hansolo77

I'm using the plain old RetroPie (MIX) profile. But I think I might have fixed it. I changed the Autoconfiguration line Path to the roms in your XML file to ./. That little change now enables Selph's to work with the existing xml just fine. I wonder if the problem I'm having is limited to my setup, because I have my ROMs stored on an external HD instead of the SD card. In either case, further testing is commencing, but I think that fixed it.

paradadf

Use the new profile I told you, I'm pretty confident it will work.

hansolo77

I was using one of the RetroPie [ROM FOLDER] profiles before, but that's not what I wanted. I didn't want to have the artwork saved next to my ROMs. This method I'm using now seems to be working just fine.

screech

@hansolo77 you are using a pretty "strange" configuration.... Gamelist.xml in rom folder and downloaded_image in an other place... Rom on an external device... Not matching the "standard" Retropie configuration nether sselph's configuration... So it's normal if you scrape with another scraper that's not working :S

For information, XML generated by UXS are legit...

<?xml version="1.0"?> : isn't necessary (But I can add it if you want)
indentation : is just a "visual" thing and change nothing.
<playcount> and <lastplayed> are added by ES... so not generated by UXS (but it's possible with empty value if you want)
Path Rom are just "preset" with standard path... You can change them, but you must change them "correctly" . You can choose relative path or full path.
But if you make "strange" things, don't expect to have a working things...

screech

@mattrixk I found why >< (after a long time searching)...

Your file in the zip must be at the root, not in an folder.

When you open the zip file you must see directly "config.xml"
Not "Standard(test)/config.xml"

^^

chuzzwuzzer

I've tried the latest version of this software and it kept locking up on my pc so had to revert back to the earlier version. So now I have 'downloaded images' folders in the roms folder of some systems and other images in some of the 'downloaded images' folder of configs>all>emulationstation. All I need to know is how do I create or edit the gamelist.xml for each system to direct it to the correct folder so the images show up on ES. Sorry if this is a stupid question but I've spent ages trying to fix this and getting no where fast !

hansolo77

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@hansolo77 you are using a pretty "strange" configuration.... Gamelist.xml in rom folder and downloaded_image in an other place... Rom on an external device...

That's not how I have it setup at all.. :/ All I have is the ROMs on an external device. Instructions for doing so are on the RetroPie wiki. I have it this way because I have SegaCD and PlayStation games that are large. The gamelist.xml files are NOT in the ROM folders... they are located in the /opt/retropie/configs/all/emulationstation/gamelists/"system"/ path, which is the default. The artwork is located in /opt/retropie/configs/all/emulationstation/downloaded_images/"system"/ which is also the default. I don't see how my setup is a "strange" configuration. This is how everything was setup originally. The only difference is I'm storing ROMs on the external drive, but as far as the system knows, everything is still the way it should be. When you're browsing via sftp, samba, or even in the terminal (ssh or local), you don't even know you switched drives. So how is my setup "strange"?

screech

So I don't understand why you are "changing" your path :S

what I read from the wiki (https://github.com/retropie/retropie-setup/wiki/scraper ) is

/home/pi/RetroPie/roms/%system%/
/home/pi/.emulationstation/downloaded_images/%system%/

and

/opt/retropie/configs/all/emulationstation/gamelists/%system%/
/opt/retropie/configs/all/emulationstation/downloaded_images/%system%/

will work the same...

but don't expect it will work with Sselph's scraper.... It don't use at all these path...

Sselph put "all things" in the Rom folder directly...

So if you want to "update" don't use 2 differents soft who don't work the same...
Or, like @paradadf say : Use the same configuration as Sselph. The profil are already created and works : Retropie [Rom folder] (MIX) for exemple.

hansolo77

They are the same paths! lol... They are simply symbolic links. Like "Genesis" actually goes to "MegaDrive".

It doesn't really matter though. It's working correctly.

mattrixk

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Your file in the zip must be at the root, not in an folder.

It was that simple? Argh, thank you so much! It's like wondering why your js/php/css doesn't work and then finding out you've missed a semi-colon somewhere.

herb_fargus

@screech sselphs scraper is modular enough that you can specify the ROM path, image folder etc. Retropie just integrated it as a module with specific file paths. They can really be set to anything you want if you know what you're doing.

chuzzwuzzer

In each rom folder I have my gamelist.xml files plus a downloaded images folder in each rom folder. Only problem is each of my gamelist.xml files are empty which is why I'm not seeing the images. Any idea why they are empty ?

paradadf

@chuzzwuzzer you need to kill emulationstation before scraping. I believe the latest version of UXS has a bug within the ssh commands. So for the time being, do it through other way.

chuzzwuzzer

But I already have the scraped images in the downloaded images folder. Are you saying I need to rescrape the entire system again ?

paradadf

Yes.

daeks

Feature request:

(Optional) Scrape only roms with missing images.
(Optional) Cleanup image folders for missing roms

screech

@chuzzwuzzer If you rescrape and keep your image in your folder, UXS won't re-download them... And just create the XML.

@daeks you already have 2 scrape options :
New rom, who scrape only roms who aren't in your XML
Picture only, who only scrape picture without touching your XML.

And for the cleanup it's already in my todo list ;)
( https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/issues/28 )

chuzzwuzzer

@screech - Just trying this. Should it still be reading : 'Hashing please wait.... Time left 2d, 22h, 41 mins 516/17651 '

Seems a long time just to rewrite the gamelist.xml if its not redownloading the images or is it normal for it to take this long ?