[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

moisessdm

@paradadf According to the API documentation, xx must be replaced by a variable (xx = "nomcourt" variable of the regionsListe.php API). The% COUNTRY% is a variable to say that the flyer will be downloaded according to the ROM region. I believe the error is in the directory where the images of the flyers are. The directory I'm putting in the XML is wrong or does not yet exist.

Furluge

Is there a way to get it to download png for the images when using the standard profile and downloading video? Right now if you change it from default to png or jpg it will save your video files as jpg or png, and default defaults to jpg. I would like to be able to save images and save 2d box art and wheels as pngs. Obviously you could rename the incorrect extension video files from png or jpg back to mp4 but that isn't ideal.

mattrixk

@Furluge There is an option in the Scrape Profile called <Source_Download_Ext> so for videos you can set <Source_Download_Ext>mp4</Source_Download_Ext>, for logos and box-art you can set <Source_Download_Ext>png</Source_Download_Ext> and for screenshots you can set <Source_Download_Ext>jpg</Source_Download_Ext>

This PasteBin dump is a work-in-progress Scrape Profile I made. It does this:

scrapes the video as mp4, appends "-video" to the filename and dumps it into the <video> tag within the gamelist.xml so it comes out as <video>/path-to-media/aladdin-video.mp4</video>.
scrapes the screenshot as jpg, appends "-screenshot" to the filename and dumps it into the <image> tag within the gamelist.xml so it comes out as <image>/path-to-media/aladdin-screenshot.jpg</image>
scrapes the select MIX image as png, appends "-mix" to the filename and dumps it into the <marquee> tag within the gamelist.xml so it comes out as <marquee>/path-to-media/aladdin-mix.png</marquee>

I designed the MIX image to scrape the logo, box-art and cartridge image, and then format them to a shape that sits around the screenshot. I think it looks good. You can see that MIX Profile here. It's also a work in progress.

The Scrape and MIX Profiles listed above worked great in UXS v5, but I haven't tested them in the new v6, so I make no promises.

derFeef

Hello @screech!
Thanks for the awesome tool and your work!
Now I am having a problem hence why I am writing ;)

Everything is working great except... Emulationstation is not showing me the information or images.

I do.

Boot retropie
Quit ES
Run your tool
Scrape (works like a charm).
The images and xml files are in the correct folder and everything.
restart ES
No images or information is shown :(

I noticed that the XML file your tool creates and the XML file the standard ES scraper is creating have a very different layout. Can this be the problem or am I missing something?

Thanks!

FlyingTomahawk

@screech

You said support is not only by money but also by uploading images, logos or anything that might help to improve the database.
Well, I did upload a couple of company logos and system logos.
Now it looks like they accepted the logos but it says Terscreenator as uploader.
~~I think credit should be given if due unless it is technically not possible to do and only those Top 3-4 guys can upload stuff.~~
UPDATE: turned out that it was a minor bug that has been taken care off by the screenscraper crew. Awesome guys over there, very quick and friendly.

paradadf

@FlyingTomahawk sorry but I have never seen that problem. Everyone I know gets its statistics correctly. And believe me, if that isn't your case, it wasn't intentionally (maybe even a bug).
Please join IRC
https://kiwiirc.com/client/irc.freenode.net/#screenzone
and share your experience with the developers. MarbleMad can track it down and correct it.

FlyingTomahawk

@paradadf

Thanks for your reply.
I am sure it was unintentional. I think I know what happened.
When you upload images there it says Source: there I wrote "Self made, Inkscape, exported to PNG file."
Maybe I should have just simply have written FlyingTomahawk

Anyways I am working on a new logo and will upload it when done and then I will just write FlyingTomahawk. Let's see if it works then.

FlyingTomahawk

UPDATE: It seems to be a bug and it has been taken care of. Those guys are very fast! I just joined the IRC and he already replied within seconds.

@paradadf

Thanks again for pointing me in the right direction.

paradadf

@FlyingTomahawk your're welcome.
One suggestion about your self-made logos. I recommend you to, instead of creating two different version depending on the possible backgrounds, create just the "original" one and add a very small white contour to the graphics. That's very easy done on whatever tools you wanna use and so you have one single logo that can be used anywhere. Otherwise it is very difficult to manage it by the scraper.

FlyingTomahawk

@paradadf

That is a good idea.

retrolove

Hi @screech
First of all I absolutely love your work on UXS and the screenscraper database - amazing stuff! I have a question related to something @hansolo77 mentioned in the MAME thread on UXS.

I have all my MAME ROMS listed with not only their names - but the description in the title as well, e.g. 720 degrees rev 1, 720 degrees rev 2 etc etc. These are not in brackets in my setup just part of the name. When I run the UXS setup however it renames these files so now I have multiple 720 degrees all with the same name (720 degrees is just an example, this is true of a lot of my roms such as Street Fighter II etc). So my question is given I have the names how I want them pre-scrape but want to scrape everything else using UXS - artwork, description, year, genre etc etc, is there a way to prevent UXS overwriting the names? I have tried editing one of the xml files in the profile to strip out what I think is the name element, but to no success :-(

Any tips on how to achieve this? This would complete my setup as it only seems to be MAME where I have this issue - all the console scraps work fine!

Thank you...!

Dochartaigh

So I finally got this to work, with 3x threads, and over the course of two days (about 40 hours) scraped all 15,000 or so ROMs...

BIG PROBLEM however - it renamed every single game's name in the RetroPie game menu. Like before I could tell the difference in ROM's between Rev 1 and Rev 2 (or Revision A and B), the difference between the EU version, Asia version, USA, beta version, etc....now everything is just named generic. Like I have three "Castlevania", four "Super Mario Bros", etc. etc. etc. Nothing differentiates the different versions of ROMs now.

If I attempt another 40+ hour scraping convention (like my 6th with this program...), what option should I change so this doesn't happen? Is it:

Options Configuration > Item to Change (Name Format) > Parameter to Change (Filename)

Sidenote: After scraping and rebooting, this is the like 4th SD card where I'm unable to exit to EmulationStation through the RetroPie menu. I just get a black screen no matter how long I wait. Running killall emulationstation via SSH kills it and exits me to the Terminal....but this has happened on literally 4 different images of this setup (where every single one BEFORE running Universal XML Scraper were just fine...).

ruckage

@Dochartaigh said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Sidenote: After scraping and rebooting, this is the like 4th SD card where I'm unable to exit to EmulationStation through the RetroPie menu. I just get a black screen no matter how long I wait. Running killall emulationstation via SSH kills it and exits me to the Terminal....but this has happened on literally 4 different images of this setup (where every single one BEFORE running Universal XML Scraper were just fine...).

I think Retropie is set to save metadata on exit by default so with 15,000 roms that's going to take a long, long while.

You can disable metadata saving within ES. See the FAQ here: Why does shut down and reboot take ages?

Dochartaigh

@ruckage

I have this same exact image on like 10+ SD cards now. EVERY single one exits to terminal in maybe 10 seconds max. My last 4 (ok, maybe 3, haven't been keeping notes ;) I've scraped with this Universal Scraper (unsuccessfully/unfinished for the most part) will NOT exit to terminal when you quit EmulationStation (I only wait a couple minutes though...can't see it taking this much longer) - and that's why I believe it to be something with this specific scraper if every system not-scraped with this is fine (all the others used SSelph's), yet every system this Universal Scraper touches can now no longer exit to terminal without a killall command being remotely sent to it...

Reboot is still perfectly normal and pretty quick (and that's saying a lot as I'm on a Pi Zero lately), FYI.

EctoOne

@Dochartaigh Too me that sounds like your gamelists is really big after scraping the data and that's why Emulation Station needs more time to rewrite it when quiting. At least that's my guess because it was mentioned anywhere in the wiki. Have you tried turning off metatarsal writing like @ruckage suggested?

Dochartaigh

Neither of you are understanding fully what I'm saying: I have multiple identical setups cloned from the same exact image using identical size/brand SD cards. One setup is scraped with Sselph. One with Universal Scraper - that is the ONLY difference.

Sselph's quits EmulationStation in maybe 10 seconds max. Universal Scraper version won't quit even after several minutes (and I just waited, didn't time it, and it does finally exit, but it's literally like 5-7 minutes which is exponentially longer...)

There are multiple setups like this with the only change being the scraper used. Really not a big deal, just friggin weird! And something I don't really know how to fix without turning off a function of the system...

retrolove

@Dochartaigh I bumped into the same issue with my MAME ROMS that the scraper renamed them all to a generic name so I can't differentiate between rev1, rev2 etc. I played around with the option configurations on the name, but that hasn't solved it so far. The only option I haven't tried is Parameter to Change (Filename). My other thought process was to modify the RetroPie MIX XML file directly and take out references to the name element, but so far that hasn't worked for me either... Hoping some folks on here may have figured this out already and can help

daeks

@Dochartaigh

Maybe its a permission issue. Running SSelps's script from Retropie-Setup will be executed as root while UXS connects as pi user. For example, if your ES starts from /etc/rc.local it might get started as root?
As far as I know UXS connects with the default pi user / password. If you have changed the password, it cant connect anymore. If I remember correctly this is hardcoded.

EctoOne

@Dochartaigh I have used sselph used only once way back when I got my Pi, so I don't know what the actual gamelist looks it creates. I do know that UXS adds some values to the game tag (screenscraper name and game id iirc) and also creates an unused thumbnail tag. And depending on your settings, it might be also creating the video and marquee tag as well I think. So with 1500 roms, I assume that's quite some useless data Emulation Station has to deal with.
You can try to pull the gamelists from the Pi and edit them to remove those tags to make the files smaller. If that doesn't work the only other thing I would try is to scrape my roms on my PC and not straight on the Pi. I for myself only scrape on PC, edit the files with RetroPie Metadata Editor to add/change the informations and then use a text editor to remove the values and thumbnail tag with notepad++ doing a search/replace. Never had a problem, although I don't have 1500 roms on my Pi.

EctoOne

@retrolove I don't know how Mame roms are named but if the additional info you want to keep is in brackets, you can go to Settings, Options configuration, Select Name Format, Select Gamename+Bracket, Save.