[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

FlyingTomahawk

That is a good idea.

retrolove

Hi @screech
First of all I absolutely love your work on UXS and the screenscraper database - amazing stuff! I have a question related to something @hansolo77 mentioned in the MAME thread on UXS.

I have all my MAME ROMS listed with not only their names - but the description in the title as well, e.g. 720 degrees rev 1, 720 degrees rev 2 etc etc. These are not in brackets in my setup just part of the name. When I run the UXS setup however it renames these files so now I have multiple 720 degrees all with the same name (720 degrees is just an example, this is true of a lot of my roms such as Street Fighter II etc). So my question is given I have the names how I want them pre-scrape but want to scrape everything else using UXS - artwork, description, year, genre etc etc, is there a way to prevent UXS overwriting the names? I have tried editing one of the xml files in the profile to strip out what I think is the name element, but to no success :-(

Any tips on how to achieve this? This would complete my setup as it only seems to be MAME where I have this issue - all the console scraps work fine!

Thank you...!

Dochartaigh

So I finally got this to work, with 3x threads, and over the course of two days (about 40 hours) scraped all 15,000 or so ROMs...

BIG PROBLEM however - it renamed every single game's name in the RetroPie game menu. Like before I could tell the difference in ROM's between Rev 1 and Rev 2 (or Revision A and B), the difference between the EU version, Asia version, USA, beta version, etc....now everything is just named generic. Like I have three "Castlevania", four "Super Mario Bros", etc. etc. etc. Nothing differentiates the different versions of ROMs now.

If I attempt another 40+ hour scraping convention (like my 6th with this program...), what option should I change so this doesn't happen? Is it:

Options Configuration > Item to Change (Name Format) > Parameter to Change (Filename)

Sidenote: After scraping and rebooting, this is the like 4th SD card where I'm unable to exit to EmulationStation through the RetroPie menu. I just get a black screen no matter how long I wait. Running killall emulationstation via SSH kills it and exits me to the Terminal....but this has happened on literally 4 different images of this setup (where every single one BEFORE running Universal XML Scraper were just fine...).

ruckage

@Dochartaigh said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Sidenote: After scraping and rebooting, this is the like 4th SD card where I'm unable to exit to EmulationStation through the RetroPie menu. I just get a black screen no matter how long I wait. Running killall emulationstation via SSH kills it and exits me to the Terminal....but this has happened on literally 4 different images of this setup (where every single one BEFORE running Universal XML Scraper were just fine...).

I think Retropie is set to save metadata on exit by default so with 15,000 roms that's going to take a long, long while.

You can disable metadata saving within ES. See the FAQ here: Why does shut down and reboot take ages?

Dochartaigh

@ruckage

I have this same exact image on like 10+ SD cards now. EVERY single one exits to terminal in maybe 10 seconds max. My last 4 (ok, maybe 3, haven't been keeping notes ;) I've scraped with this Universal Scraper (unsuccessfully/unfinished for the most part) will NOT exit to terminal when you quit EmulationStation (I only wait a couple minutes though...can't see it taking this much longer) - and that's why I believe it to be something with this specific scraper if every system not-scraped with this is fine (all the others used SSelph's), yet every system this Universal Scraper touches can now no longer exit to terminal without a killall command being remotely sent to it...

Reboot is still perfectly normal and pretty quick (and that's saying a lot as I'm on a Pi Zero lately), FYI.

EctoOne

@Dochartaigh Too me that sounds like your gamelists is really big after scraping the data and that's why Emulation Station needs more time to rewrite it when quiting. At least that's my guess because it was mentioned anywhere in the wiki. Have you tried turning off metatarsal writing like @ruckage suggested?

Dochartaigh

Neither of you are understanding fully what I'm saying: I have multiple identical setups cloned from the same exact image using identical size/brand SD cards. One setup is scraped with Sselph. One with Universal Scraper - that is the ONLY difference.

Sselph's quits EmulationStation in maybe 10 seconds max. Universal Scraper version won't quit even after several minutes (and I just waited, didn't time it, and it does finally exit, but it's literally like 5-7 minutes which is exponentially longer...)

There are multiple setups like this with the only change being the scraper used. Really not a big deal, just friggin weird! And something I don't really know how to fix without turning off a function of the system...

retrolove

@Dochartaigh I bumped into the same issue with my MAME ROMS that the scraper renamed them all to a generic name so I can't differentiate between rev1, rev2 etc. I played around with the option configurations on the name, but that hasn't solved it so far. The only option I haven't tried is Parameter to Change (Filename). My other thought process was to modify the RetroPie MIX XML file directly and take out references to the name element, but so far that hasn't worked for me either... Hoping some folks on here may have figured this out already and can help

daeks

@Dochartaigh

Maybe its a permission issue. Running SSelps's script from Retropie-Setup will be executed as root while UXS connects as pi user. For example, if your ES starts from /etc/rc.local it might get started as root?
As far as I know UXS connects with the default pi user / password. If you have changed the password, it cant connect anymore. If I remember correctly this is hardcoded.

EctoOne

@Dochartaigh I have used sselph used only once way back when I got my Pi, so I don't know what the actual gamelist looks it creates. I do know that UXS adds some values to the game tag (screenscraper name and game id iirc) and also creates an unused thumbnail tag. And depending on your settings, it might be also creating the video and marquee tag as well I think. So with 1500 roms, I assume that's quite some useless data Emulation Station has to deal with.
You can try to pull the gamelists from the Pi and edit them to remove those tags to make the files smaller. If that doesn't work the only other thing I would try is to scrape my roms on my PC and not straight on the Pi. I for myself only scrape on PC, edit the files with RetroPie Metadata Editor to add/change the informations and then use a text editor to remove the values and thumbnail tag with notepad++ doing a search/replace. Never had a problem, although I don't have 1500 roms on my Pi.

EctoOne

@retrolove I don't know how Mame roms are named but if the additional info you want to keep is in brackets, you can go to Settings, Options configuration, Select Name Format, Select Gamename+Bracket, Save.

retrolove

@EctoOne thanks for the reply. Unfortunately that info is not kept in the brackets, it's almost like it is a giant tile. The issue is that a title like Street Fight II World Warrior Rev B USA would get shortened after the scrape to Street Fighter II World Warrior. The problem is I then have 4 or 5 games of the same name and no way to differentiate. If there was a way to tell UXS not to scrape or update the display name that would be fantastic and solve my problem, but unfortunately I can't figure out how to do it....

EctoOne

@retrolove In the same options you can also select to use the filename if that helps. I don't have access to a windows pc right now so i can't tell if it works.

screech

Hi, sorry to be late (again) ^^

@retrolove :
For the naming : You can choose in the Option configuration 4 things :

Game name (it's the default one, and it's the name referenced in the DB... so you can have duplicate in case of multiple rom for 1 game -> Use this if your romset is 1game/1rom)
Game name + Bracket (It add the info under bracket from your rom file to the game name referenced)
Game name + Country (It add the country of your rom referenced in the DB to the game name referenced in the DB)
Filename without extension (It simply take the rom filename, strip out the extension and put it as name... I think it's the option you look for ;) )
An other is under dev : Game name + Info (the DB begin to have more and more info about Hack Type, and Clone Type, so it will take the Game name referenced and add the info about the rom you scrape like REV A, 2 players, Beta, Proto, Translation,.... ) But it's not done yet :S

@Dochartaigh What UXS try to do when you want to "stop" Emulationstation is in the profil.xml you choose in the PLINK part. For Retropie standard are :

<Plink>
	<Ip>RETROPIE</Ip>
	<Root>pi</Root>
	<Pswd>raspberry</Pswd>
	<Command>
		<killall>killall emulationstation</killall>
		<reboot>sudo /sbin/reboot</reboot>
		<poweroff>sudo /sbin/poweroff</poweroff>
	</Command>
</Plink>

You can change these directly in the XML or (only for SSH connection parameter) in the SSH config menu.

When UXS try to kill emulationstation, it use a software name PLINK (it's putty in command line) connect with these info :
RETROPIE is the "IP" (you can put your real IP if there is problem)
Root : pi
Pswd : raspberry
(If you have changed these, you must change it in UXS too... It's the default value for Retropie (normally^^))

And it launch the Command in SSH : "killall emulationstation"

If it doesn't work, and you managed to do the exact same thing with putty (for exemple) there is a problem :S

(And again, 40hour for 15000 Roms with 3 threads seems to be Very very long :S don't know what happen :( )

@EctoOne If you want the "extra" tag in the XML to be rip off, simply "edit" the Profil.xml you choose and delete the element you don't want.

Exemple :

<Element Type="thumbnail">
	<Source_Type>Fixe_Value</Source_Type>
	<Source_Value></Source_Value>
	<Target_Type>XML_Value</Target_Type>
	<Target_Value>game/thumbnail</Target_Value>
</Element>

It's standard in the gamelist from ES (actually there is nothing in Source_Value, but you can put something if you want ;) )... But if you don't want it, simply delete it ;)

EctoOne

@screech Oh I never thought of that. Silly me. Also can you answer my question about scraping ScummVM games?
@EctoOne said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.

Dochartaigh

@screech

@Dochartaigh What UXS try to do when you want to "stop" Emulationstation is in the profil.xml you choose in the PLINK part. For Retropie standard are :

Hi Screech, I wasn't talking about the kill/exit command Universal Scraper sends to the Pi to shutdown EmulationStation so it can scrape. I'm talking about in the RetroPie menu (on the Pi itself), when you hit Start > Quit > Quit EmulationStation, that no longer works as smoothly as it did before I scraped with your program (probably ~3-5 solid minutes to quit). It works perfect on systems scraped with the regular SSelph scraper (maybe 10 seconds) - others have chimed in your program adds more fields to the XML so it's probably with that it just needs more time to shutdown (albeit like 50 times as long...)

To the 40 hours to scrape everything, that was just how long it took over the course of 2 days at work remoting into my computer at home to start the next leg. Your program will NOT, ever, let me scrape all systems as once (always gives me those errors I posted about before) so I have to manually select the next system and let it run, then select the next system and let it run, etc. etc. etc. (and the hard part is simply remembering to keep checking the progress all day long so I can start it on the next system ;) It is TONS faster with the 3 threads at once - and I honestly think you at least need an account no matter what to ever have it work or it just times out or says like 12 hours to scrape a single romset.

retrolove

@screech Thanks so much for the reply on the game naming, unfortunately none of those options are working with my setup. I do have a starting point of a gameslist.xml file with the game names exactly as I want them - you can see this with the additional info on screenshot 1. If I use UXS using either name or name+bracket then the scraper renames then to the name as it is in the screen scraperdb, see screenshot 2. As you rightly state with multiple roms of the same name it labels them all the same way. If I scrape with the filename option this doesn't work either as my roms are named shortname+zip, see screenshot 3 for outcome and screenshot 4 for my roms. I guess i have two questions:-

Given I already have a gameslist.xml file with names the way I want them can I edit any of the UXS xml files to stop it scraping the name from screenscraper.fr? If this is possible this will solve the issue as I do want to get all of the other info apart from the name from screen scraper.fr (artwork, description etc etc)
If the above is not possible do you or anyone know of a way I can take the name elements from my original gamelist.xml file (which has the names the way I want them) and then merge that in with the new gamelist.xml that UXS created, overwriting the UXS created names with my original one?

Any help would be hugely appreciated! And thanks for all your hard work on UXS - it really is awesome! :-)
Screenshot 1

Screenshot 2

Screenshot 3

Screenshot 4

EctoOne

I'm going to answer my own question about scraping ScummVM games. It doesn't work with RetroPie. At least not in a way that would allow for easy scraping and playing. To make it work, screenscraper.fr would need all the hashes from the different .svm files RetroPie uses to be able to start ScummVM games straight from Emulation Station.

At least i figured out how to use the .scummvm files from this thread over at the Recalbox forum, to get Mix Images.

If you want to get the images/videos:

Get the RAR file from the thread mentioned
Put all .scummvm files into a folder named scummvm in your rom folder.
Open the profil xml you're using (e.g. Retropie (MIX).xml) in a texteditor.
Add *.scummvm;into the <Research> tag.
Save the file and open UXS to scrape.

What i did was to scrape inside Emulation Station (need to manually insert the right game name for every game) after ScummVM created all the .svm files. Then i replaced the scraped images with the Mix images from UXS.

Sidenote:
I also added *.uae; to my <research> tag and it worked right away only using scraping for names without hashes.
If anyone is interested, here is my profil.xml
It has support for uae and scummvm files enabled. It also downloads the 2D boxart, screenshot, marquee and (if available) fanart and video. BUT it doesn't add those to the gamelist! It only adds the selected Mix image and the normal Metadata. I also removed the <thumbnail> tag and the gameid/screenscraper name from the <game> tag.
The only issue with this is, that videos are saved as png files and so far i don't know how to fix it. Although since i don't use video view i don't care.

paradadf

@EctoOne I'm not sure how the .svm files look inside, but if they are empty, you can just do this:
https://retropie.org.uk/forum/topic/4142/test-a-new-data-source-for-sselph-scraper/17
It has a link to the samd recalbox tutorial. Just ñut the official long name inside the .svm file and you are ready to scrape wirhout too much hassle.

EctoOne

@paradadf They are empty but whatever. I just finished scraping all images, so i won't test it. :P
In the end it was easier to scrape the whole scummvm folder from the rar and replace *.scummvm with *.svm inside the gamelist.xml UXS creates with the ones the GUI creates.