[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

screech

@windale I'm on it ;)

screech

I just release what I think is the first stable version ;)

some improvement for Retropie on it ;)

(autoconfiguration path are normalized with Reptropie Path and Scraped extension file are restricted to the Retropie used extension)

@windale the Add missings roms in XML is corrected ;)
@sonicsheppard the ' problem is normmaly resolved ;)

UXS V2 Stable here

sonicsheppard

@screech

Hi, happy to report the issue about roms with apostrophes being missed is verified fixed in the latest version, as is the fixed pathing for RetroPie.

Couple of questions though, in this new V2 Stable version I've noticed that the rom titles in the gamelist.xml now contain the region, e.g. Aliens vs Predator (USA). This wasn't happening in the previous versions, is there an option to disable that? Also, for roms with a World region, they are being displayed as the French (Monde).

Thanks for sorting this though, it's coming along great! If you need any more info just let me know.

Cheers!

vbs

@sonicsheppard
You can configure in UXS how you want your game names composed. I currently cannot check for the exact menu though. I think its in "options".

sonicsheppard

@vbs

Wicked, found it in Configuration -> Options configuration -> Item Choice/Name Format.

Works a treat. Thanks!

windale

@sonicsheppard said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@screech
Also, for roms with a World region, they are being displayed as the French (Monde).

Yeah, I reported this earlier today (before this version was released). 'Japan' was also misspelt as 'Japon', and 'Platform' was 'Plateform'. I think they said that 'Plateform' was fixed. Can't check at the moment.

screech

Hello all,

I release some new version, with little correction ;) -> 2.0.0.3

For the wrong english translation, it's due to the DB.

Screenscraper (at start) was made for french people :S with more and more people we add (quickly) other language... But it's not so well added ;)

So a Screenscraper V2 is under developpement to manage every language ;) (for data AND media ;) )

Be patient it will come ;)

vbs

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Screenscraper (at start) was made for french people :S with more and more people we add (quickly) other language... But it's not so well added ;)

So a Screenscraper V2 is under developpement to manage every language ;) (for data AND media ;) )

Be patient it will come ;)

Thats really cool to hear! Will the forum be in english too?

screech

I will ask ^^

It's obvious it's not easy for foreign user to speak with us ^^

But if you have problem, don't hesitate to visit us on our IRC Channel, we will be please to answer any question you have ;)

https://kiwiirc.com/client/irc.freenode.net/?#recalbox

Dochartaigh

This looks great, but I'm afraid I'll have to wait until this can be run on the Pi itself, and not a 2nd system which has network drive access to the PI. (plus I'm on a Mac so I feel leftout....and Wine has never worked well for me in the past :(

vbs

Has anyone an idea how to scrape MAME sets with parents and clones so that you get sensible game titles in the end?

Lets say the files are named:

airbustrj.zip
airbustrb.zip
airbustr.zip

Which should map to something like "Air Buster (Japan)", "Air Buster (bootleg)" and "Air Buster (World)". But what whatever filename option I use in UXS I either get the plain filename (cryptic) or for all three the same just "Air Buster".

Rion

@vbs I have been talking about this with @screech on the official irc channel for screenscraper.fr. There have been talks about implementation of Arcade Italia's datbase. What i last heard is that it's planned for v2 of the website.

Mamedb is down as we all know and have/had not been updated as of Mame 0.15X

Right now when you scrape games for Mame/fba you get the same names for the parrent rom and the clones with some few deviation from roms that have (World) or (Japan) in them.

Hopefully this will see the light of day soon.

hansolo77

I hate to drudge up a possible issue since I'm a new user to this. I was not aware a tool like this existed prior to it being mentioned here:

https://retropie.org.uk/forum/topic/5498/scraper-by-sselph-can-t-scrape-chosen-systems/9

I have SAMBA shares enabled on my Pi, connected to the same network. I can browse to the Pi using the \\RetroPie\ address, and can see the various folders (bios, configs, roms, splashscreens). I can also SFTP to the Pi and can browse to the real location of those files (/home/pi/retropie/).

Before I left for work today, I configured the Scraper to use the RetroPie configuration, and went through the process of telling it to use the 4-way MIX view (I like that, the other scrapers just get cover art!). I then started the process off. The first system it started scraping for me was Atari 2600, which is correct. But I waited and watched as it does it's thing. It appeared to be doing something, but I checked the location the files should be saved at via SFTP and didn't find them. I figured it was because it was still scraping, and maybe the program grabs everything FIRST, then uploads them to the Pi rather than each file ASAP. So I went off to work, hoping it would be done scraping everything when I get home. Well, I'm home now and it's still scraping. I SFTP'd again to see if it had sent any images or anything to the Pi, and it hasn't. If I browse to the path /home/pi/.emulationstation (which actually goes to /opt/retropie/configs/all/emulationstation) I see the 2 folders "downloaded_images" and "gamelists", but both folders are empty. This made me think "Do I need to register and PAY to use this program or something?"

Further checking in the logs shows it is indeed sending stuff to the Pi, but I can't find it. According to the logs, there is this line:

[11:05:09] - > $vTarget_ImagePath = /home/pi/.emulationstation/downloaded_images/atari2600/

When I SFTP to that folder, nothing is there. It is currently scraping for NES now, so the Atari sets should be done. Does it not send the images/etc to the Pi until the ENTIRE process is completed?

EDIT (APPEND)

Ok, so I just checked a bit further. I noticed also in the log that it says:

[11:05:09] - > $vSource_ImagePath = \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600

So to me, this means it's checking first to see if images exist somewhere in the \roms path. That can't be right. I don't have any of my other scrapers configured to store images and gamelists next to the roms. Nevertheless, I checked that path, and there still isn't any artwork in /home/pi/retropie/roms/atari2600, just an added text file about what's missing.

Interestingly, there is a NEW folder in the /home/pi/retropie/roms path called "configs" which wasn't there before. When I go in there, there are a few more new nested folders... so now there is a path line going clear to /home/pi/retropie/roms/configs/all/emulationstation. Inside there, I see 2 folders "downloaded_images" and "gamelists". Ah-hah... checking THOSE folders does indeed reveal the uploaded artwork.

Why in the world is this program scraping artwork into that path? That's not the right path at all. Once this is all said and done, I'm pretty sure EmulationStation isn't going to be able to find those folders to display them. What a complete waste of time! Am I correct in seeing all this? As I said, I'm still a new user, just downloaded and installed the latest version from the github today. Specifically, it was THIS version. Should I be using a different version, or is this all correct in it's function?

vbs

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

[11:05:09] - > $vSource_ImagePath = \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600

So to me, this means it's checking first to see if images exist somewhere in the \roms path. That can't be right. I don't have any of my other scrapers configured to store images and gamelists next to the roms. Nevertheless, I checked that path, and there still isn't any artwork in /home/pi/retropie/roms/atari2600, just an added text file about what's missing.

Interestingly, there is a NEW folder in the /home/pi/retropie/roms path called "configs" which wasn't there before. When I go in there, there are a few more new nested folders... so now there is a path line going clear to /home/pi/retropie/roms/configs/all/emulationstation. Inside there, I see 2 folders "downloaded_images" and "gamelists". Ah-hah... checking THOSE folders does indeed reveal the uploaded artwork.

The ..means to go back one directory, so \\retropie\roms\..\configs\all\emulationstation\downloaded_images\atari2600would result in \\retropie\configs\all\emulationstation\downloaded_images\atari2600. But I also stumbled about it and I think it does not work as intended. I made a ticket about my problem here:
https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/issues/51

For me it worked after I replaced in XML file name and Folder Name where pictures will be downloaded (Autoconfig) the %Source_RootPath%\..\configsdirectly with \\mypi\configs.

But this path still did not get used after changing it and it sticked to the old path when scraping. I found the old path also in UXS-config.ini. After fixing it there manually too it worked for me...

screech

sorry to answer so late...

This is how it work :

In Profil XML you have the "autoconfiguration path" standard for retropie :

	<AutoConf>
		<Source_RootPath>\\retropie\roms</Source_RootPath>
		<Target_XMLName>%Source_RootPath%\..\configs\all\emulationstation\gamelists\%System%\gamelist.xml</Target_XMLName>
		<Target_RomPath>/home/pi/RetroPie/roms/%system%/</Target_RomPath>
		<Source_ImagePath>%Source_RootPath%\..\configs\all\emulationstation\downloaded_images\%System%</Source_ImagePath>
		<Target_ImagePath>/home/pi/.emulationstation/downloaded_images/%system%/</Target_ImagePath>
	</AutoConf>

Explaination :
Source_RootPath is where UXS Autoconfig look for system folder
Target_XMLName is where UXS put the final gamelist.xml
Target_RomPath is where emulationstation will found the rom (UXS put this path in gamelist.xml)
Source_ImagePath is where UXS will download picture (mix or not)
Target_ImagePath is where emulationstation will found the picture (UXS put this in gamelist.xml)

I added some "tag" that will be replace automatically :
%Source_RootPath% will be replaced by the path in <Source_RootPath>
%system% will be replaced by the folder of the system you selected in autoconf
%SystemDir% will be a merge of %Source_RootPath%+%system%

Autoconf will replace path and put it in UXS-config.ini, when you select a system in the menu "scrape/select a system"
And the scrape use path in UXS-config.ini

So now if we examin the standard path :
Source_RootPath = \retropie\roms
Target_XMLName = %Source_RootPath%..\configs\all\emulationstation\gamelists\%System%\gamelist.xml
-> If we select nes as system :
\retropie\roms\..\configs\all\emulationstation\gamelists\nes\gamelist.xml (in UXS-config.ini)
So :
\retropie\configs\all\emulationstation\gamelists\nes\gamelist.xml (is the reel path /../ meaning 1 step behind)
Target_RomPath = /home/pi/RetroPie/roms/%system%/
-> If we select nes as system : /home/pi/RetroPie/roms/nes/ (in UXS-config.ini)
Source_ImagePath = %Source_RootPath%\..\configs\all\emulationstation\downloaded_images\%System%
-> If we select nes as system :
\retropie\roms..\configs\all\emulationstation\downloaded_images\nes (in UXS-config.ini)
So :
\retropie\configs\all\emulationstation\downloaded_images\nes (is the reel path /../ meaning 1 step behind)
Target_ImagePath = /home/pi/.emulationstation/downloaded_images/%system%/
-> If we select nes has system : /home/pi/.emulationstation/downloaded_images/nes/ (in UXS-config.ini)

It's a little bit complexe... Sorry, but it's needed to "respect Retropie Path" and keep a simple way to configure path with changing only the Source_RootPath.
So if you want to scrape a USB Key having the "same" type of path, you juste have to change the \retropie\roms By E:\roms for exemple, and all other path will adapte themself...

Last but not least, the easiest way still to use Path like Sselph's scraper (or the path used in Recalbox) where all gamelist and downloaded picture are directly in the rom system folder.

That's why I create a new profil in the last version with [Sselph Path] : much much more easier ;)

hansolo77

@screech Yes, I see what the idea is supposed to be, but it doesn't work that way. It's not saving the information (images/gamelist) in the /home/pi/.emulationstation/downloaded_images/nes/ path. It saves it in the /home/pi/retropie/roms/configs/all/emulationstation/downloaded_images/nes/ path.

screech

That's weird :S

I just retry, and it works for me :S

I haven't retropie, so I make a network directory on my nas named Retropie, and connect a netdrive on it... (just to simulate the retropie).

And I try to scrape

Do I have this : http://hpics.li/841835b

So for me it's ok :S (from my point of view :S)

Can you try delete the UXS folder, and download the very last version ?
(I ask to delete the folder to be sure the "old" profil will be recreated)

vbs

Just tested it again and I have to say for me really ".." does not work. A workaround for now is to go to Autoconfiguration path configuration and edit XML file name and Folder name where pictures will be downloaded. Replace %Source_RootPath%\..\configs with \\<host>\configs. (enter the hostname instead of <host> ofc :)

Important: After changing that you have to re-select the system in the menu "Select your system"!

After that it worked for me and UXS used the correct folders.

hansolo77

I think the problem is that you don't have a RetroPie to test this on. According to @vbs the problem is with the \..\ inside the path. So I'm trying his approach now.

Fresh Install.. Open the program, go to Configuation, go to Path Configuration.... Get this screen:
alt text

Changed XML file path to \\retropie\configs\all\emulationstation\gamelists\amstradcpc\gamelist.xml
Changed Directory where pictures will be downlo to \\retropie\configs\all\emulationstation\downloaded_images\amstradcpc

Will this only save for THAT system?

Fresh install still, go to Configuration, Autoconfiguration path configuration.. Get this screen:
alt text

Changed XML file name to \\retropie\configs\all\emulationstation\gamelists\%System%\gamelist.xml
Changed Folder Name where pictures will be do to \\retropie\configs\all\emulationstation\downloaded_images\%System%

screech

I added a %host% tag.... So now, it will work for standard Retropie Path and if you change these path to a usb stick for exemple, only Root system directory need to be changed ;)

And some other correction : V2.0.0.4