[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

FlyingTomahawk

You said support is not only by money but also by uploading images, logos or anything that might help to improve the database.
Well, I did upload a couple of company logos and system logos.
Now it looks like they accepted the logos but it says Terscreenator as uploader.
~~I think credit should be given if due unless it is technically not possible to do and only those Top 3-4 guys can upload stuff.~~
UPDATE: turned out that it was a minor bug that has been taken care off by the screenscraper crew. Awesome guys over there, very quick and friendly.

paradadf

@FlyingTomahawk sorry but I have never seen that problem. Everyone I know gets its statistics correctly. And believe me, if that isn't your case, it wasn't intentionally (maybe even a bug).
Please join IRC
https://kiwiirc.com/client/irc.freenode.net/#screenzone
and share your experience with the developers. MarbleMad can track it down and correct it.

FlyingTomahawk

@paradadf

Thanks for your reply.
I am sure it was unintentional. I think I know what happened.
When you upload images there it says Source: there I wrote "Self made, Inkscape, exported to PNG file."
Maybe I should have just simply have written FlyingTomahawk

Anyways I am working on a new logo and will upload it when done and then I will just write FlyingTomahawk. Let's see if it works then.

FlyingTomahawk

UPDATE: It seems to be a bug and it has been taken care of. Those guys are very fast! I just joined the IRC and he already replied within seconds.

@paradadf

Thanks again for pointing me in the right direction.

paradadf

@FlyingTomahawk your're welcome.
One suggestion about your self-made logos. I recommend you to, instead of creating two different version depending on the possible backgrounds, create just the "original" one and add a very small white contour to the graphics. That's very easy done on whatever tools you wanna use and so you have one single logo that can be used anywhere. Otherwise it is very difficult to manage it by the scraper.

FlyingTomahawk

@paradadf

That is a good idea.

retrolove

Hi @screech
First of all I absolutely love your work on UXS and the screenscraper database - amazing stuff! I have a question related to something @hansolo77 mentioned in the MAME thread on UXS.

I have all my MAME ROMS listed with not only their names - but the description in the title as well, e.g. 720 degrees rev 1, 720 degrees rev 2 etc etc. These are not in brackets in my setup just part of the name. When I run the UXS setup however it renames these files so now I have multiple 720 degrees all with the same name (720 degrees is just an example, this is true of a lot of my roms such as Street Fighter II etc). So my question is given I have the names how I want them pre-scrape but want to scrape everything else using UXS - artwork, description, year, genre etc etc, is there a way to prevent UXS overwriting the names? I have tried editing one of the xml files in the profile to strip out what I think is the name element, but to no success :-(

Any tips on how to achieve this? This would complete my setup as it only seems to be MAME where I have this issue - all the console scraps work fine!

Thank you...!

Dochartaigh

So I finally got this to work, with 3x threads, and over the course of two days (about 40 hours) scraped all 15,000 or so ROMs...

BIG PROBLEM however - it renamed every single game's name in the RetroPie game menu. Like before I could tell the difference in ROM's between Rev 1 and Rev 2 (or Revision A and B), the difference between the EU version, Asia version, USA, beta version, etc....now everything is just named generic. Like I have three "Castlevania", four "Super Mario Bros", etc. etc. etc. Nothing differentiates the different versions of ROMs now.

If I attempt another 40+ hour scraping convention (like my 6th with this program...), what option should I change so this doesn't happen? Is it:

Options Configuration > Item to Change (Name Format) > Parameter to Change (Filename)

Sidenote: After scraping and rebooting, this is the like 4th SD card where I'm unable to exit to EmulationStation through the RetroPie menu. I just get a black screen no matter how long I wait. Running killall emulationstation via SSH kills it and exits me to the Terminal....but this has happened on literally 4 different images of this setup (where every single one BEFORE running Universal XML Scraper were just fine...).

ruckage

@Dochartaigh said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Sidenote: After scraping and rebooting, this is the like 4th SD card where I'm unable to exit to EmulationStation through the RetroPie menu. I just get a black screen no matter how long I wait. Running killall emulationstation via SSH kills it and exits me to the Terminal....but this has happened on literally 4 different images of this setup (where every single one BEFORE running Universal XML Scraper were just fine...).

I think Retropie is set to save metadata on exit by default so with 15,000 roms that's going to take a long, long while.

You can disable metadata saving within ES. See the FAQ here: Why does shut down and reboot take ages?

Dochartaigh

@ruckage

I have this same exact image on like 10+ SD cards now. EVERY single one exits to terminal in maybe 10 seconds max. My last 4 (ok, maybe 3, haven't been keeping notes ;) I've scraped with this Universal Scraper (unsuccessfully/unfinished for the most part) will NOT exit to terminal when you quit EmulationStation (I only wait a couple minutes though...can't see it taking this much longer) - and that's why I believe it to be something with this specific scraper if every system not-scraped with this is fine (all the others used SSelph's), yet every system this Universal Scraper touches can now no longer exit to terminal without a killall command being remotely sent to it...

Reboot is still perfectly normal and pretty quick (and that's saying a lot as I'm on a Pi Zero lately), FYI.

EctoOne

@Dochartaigh Too me that sounds like your gamelists is really big after scraping the data and that's why Emulation Station needs more time to rewrite it when quiting. At least that's my guess because it was mentioned anywhere in the wiki. Have you tried turning off metatarsal writing like @ruckage suggested?

Dochartaigh

Neither of you are understanding fully what I'm saying: I have multiple identical setups cloned from the same exact image using identical size/brand SD cards. One setup is scraped with Sselph. One with Universal Scraper - that is the ONLY difference.

Sselph's quits EmulationStation in maybe 10 seconds max. Universal Scraper version won't quit even after several minutes (and I just waited, didn't time it, and it does finally exit, but it's literally like 5-7 minutes which is exponentially longer...)

There are multiple setups like this with the only change being the scraper used. Really not a big deal, just friggin weird! And something I don't really know how to fix without turning off a function of the system...

retrolove

@Dochartaigh I bumped into the same issue with my MAME ROMS that the scraper renamed them all to a generic name so I can't differentiate between rev1, rev2 etc. I played around with the option configurations on the name, but that hasn't solved it so far. The only option I haven't tried is Parameter to Change (Filename). My other thought process was to modify the RetroPie MIX XML file directly and take out references to the name element, but so far that hasn't worked for me either... Hoping some folks on here may have figured this out already and can help

daeks

@Dochartaigh

Maybe its a permission issue. Running SSelps's script from Retropie-Setup will be executed as root while UXS connects as pi user. For example, if your ES starts from /etc/rc.local it might get started as root?
As far as I know UXS connects with the default pi user / password. If you have changed the password, it cant connect anymore. If I remember correctly this is hardcoded.

EctoOne

@Dochartaigh I have used sselph used only once way back when I got my Pi, so I don't know what the actual gamelist looks it creates. I do know that UXS adds some values to the game tag (screenscraper name and game id iirc) and also creates an unused thumbnail tag. And depending on your settings, it might be also creating the video and marquee tag as well I think. So with 1500 roms, I assume that's quite some useless data Emulation Station has to deal with.
You can try to pull the gamelists from the Pi and edit them to remove those tags to make the files smaller. If that doesn't work the only other thing I would try is to scrape my roms on my PC and not straight on the Pi. I for myself only scrape on PC, edit the files with RetroPie Metadata Editor to add/change the informations and then use a text editor to remove the values and thumbnail tag with notepad++ doing a search/replace. Never had a problem, although I don't have 1500 roms on my Pi.

EctoOne

@retrolove I don't know how Mame roms are named but if the additional info you want to keep is in brackets, you can go to Settings, Options configuration, Select Name Format, Select Gamename+Bracket, Save.

retrolove

@EctoOne thanks for the reply. Unfortunately that info is not kept in the brackets, it's almost like it is a giant tile. The issue is that a title like Street Fight II World Warrior Rev B USA would get shortened after the scrape to Street Fighter II World Warrior. The problem is I then have 4 or 5 games of the same name and no way to differentiate. If there was a way to tell UXS not to scrape or update the display name that would be fantastic and solve my problem, but unfortunately I can't figure out how to do it....

EctoOne

@retrolove In the same options you can also select to use the filename if that helps. I don't have access to a windows pc right now so i can't tell if it works.

screech

Hi, sorry to be late (again) ^^

@retrolove :
For the naming : You can choose in the Option configuration 4 things :

Game name (it's the default one, and it's the name referenced in the DB... so you can have duplicate in case of multiple rom for 1 game -> Use this if your romset is 1game/1rom)
Game name + Bracket (It add the info under bracket from your rom file to the game name referenced)
Game name + Country (It add the country of your rom referenced in the DB to the game name referenced in the DB)
Filename without extension (It simply take the rom filename, strip out the extension and put it as name... I think it's the option you look for ;) )
An other is under dev : Game name + Info (the DB begin to have more and more info about Hack Type, and Clone Type, so it will take the Game name referenced and add the info about the rom you scrape like REV A, 2 players, Beta, Proto, Translation,.... ) But it's not done yet :S

@Dochartaigh What UXS try to do when you want to "stop" Emulationstation is in the profil.xml you choose in the PLINK part. For Retropie standard are :

<Plink>
	<Ip>RETROPIE</Ip>
	<Root>pi</Root>
	<Pswd>raspberry</Pswd>
	<Command>
		<killall>killall emulationstation</killall>
		<reboot>sudo /sbin/reboot</reboot>
		<poweroff>sudo /sbin/poweroff</poweroff>
	</Command>
</Plink>

You can change these directly in the XML or (only for SSH connection parameter) in the SSH config menu.

When UXS try to kill emulationstation, it use a software name PLINK (it's putty in command line) connect with these info :
RETROPIE is the "IP" (you can put your real IP if there is problem)
Root : pi
Pswd : raspberry
(If you have changed these, you must change it in UXS too... It's the default value for Retropie (normally^^))

And it launch the Command in SSH : "killall emulationstation"

If it doesn't work, and you managed to do the exact same thing with putty (for exemple) there is a problem :S

(And again, 40hour for 15000 Roms with 3 threads seems to be Very very long :S don't know what happen :( )

@EctoOne If you want the "extra" tag in the XML to be rip off, simply "edit" the Profil.xml you choose and delete the element you don't want.

Exemple :

<Element Type="thumbnail">
	<Source_Type>Fixe_Value</Source_Type>
	<Source_Value></Source_Value>
	<Target_Type>XML_Value</Target_Type>
	<Target_Value>game/thumbnail</Target_Value>
</Element>

It's standard in the gamelist from ES (actually there is nothing in Source_Value, but you can put something if you want ;) )... But if you don't want it, simply delete it ;)

EctoOne

@screech Oh I never thought of that. Silly me. Also can you answer my question about scraping ScummVM games?
@EctoOne said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.