[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

mattrixk

@RedDog: UXS only scrapes about 1/3 of my PSP titles.

@screech: I love your work mate, as always. I had an idea though: the built-in ES scraper gives you the option to verify which game is being scraped, or to pick one from a list. Do you think it would be viable to include an option like this in UXS?

My PSP and PSX systems both use .pbp files, so I imagine the scraper is going off file names. However, lots of the those file names have the colon ":" character, which windows won't let you put in filenames, so the scraper skips over them. It would be great if the scraper would pause and then give you a list to choose from. Even if it was just a list of every game in the system, rather than UXS trying to work out which games would be the most likely.

Obviously this wouldn't be default behaviour, but maybe a checkbox in the General Settings. Maybe UXS would scrape all the ones it can automatically, then at the end it gives you the option to manually select the games it couldn't find on it's own.

screech

@hansolo77
SSH : Normally, it works now (several confirmation from different user) so if it doesn't work on your computer, there is something else (try to use putty to test). And it's normal that when you scrape locally, it don't ask to kill es...
Blank <image/> : I need to make some test.. don't know why it do this (on for now, you are the only one with this issue I know)

@brunocedup
Video quality : It's depending on what is on the DB... I haven't check every video ^^. But normally they are in the "original" resolution (SNES have a bigger resolution than Master system.. it may be a reason why)

@ghostlywindmill
Problem with autoconf : Can you check you can access the different path from your samba Share with your computer ? it's like UXS can't access in these folder. You can try the alternate path (in rom folder) they're is a new option for that in advanced configuration menu.

@mattrixk
Neogeo rom in Mame : That's not a good news :( it mean they're some neogeo rom in the Mame system on Screenscraper DB... I need to do something on that ;)
UXS improvement : Sorry, for now it's impossible. The API can match only on CRC or on filename... No other choice (for now)

@A2ra3L
micro deconnection ? Don't know why it's "very long"... what system are you scraping (a CD one ? so it's about normal since the ISO/BIN are very big to hash).
One thing : At about midnight (french hour) the server do a lot's of stuff and can be very slow for about 10/20 minutes...
PSX without picture : When you scrape PSX, your Emulationstation were killed ? Can you check your psx gamelist.xml and the image picture ?.
And FYI, UXS don't use thegamdb but ScreenScraper DB ;) so if you enjoy UXS result and want to fill the missing you are welcome in Screenscraper Too ^^

@RedDog
Same as above, maybe it's the "midnight" stuff that break your scrape ?
For wheels, we still have a lot's of work to associated these we have to the good country rom (exemple : a wheel is badly referenced as "Japan" and the rom is "US" so the wheel may not be downloaded, just check the fallback in general menu)
PSP : never tryed (I haven't PSP rom ^^) I need to make some test ;) (But for now, the PSP DB isn't so "big" on ScreenScraper, we need some contributor on this system ;)

A2ra3L

@screech

It started working again after a while. Must have been my connection or the website was busy. But the PSX games confused me. The es was killed and the xml file for the psx games showed the same filenames and locations as the png images so they should have worked. All my other systems worked fine. The strange thing is the psx images had doubled names. For example ape escapes png file was called ape escape_ape escape.png only the psx had this issue.

RedDog

@RedDog
Same as above, maybe it's the "midnight" stuff that break your scrape ?
For wheels, we still have a lot's of work to associated these we have to the good country rom (exemple : a wheel is badly referenced as "Japan" and the rom is "US" so the wheel may not be downloaded, just check the fallback in general menu)
PSP : never tryed (I haven't PSP rom ^^) I need to make some test ;) (But for now, the PSP DB isn't so "big" on ScreenScraper, we need some contributor on this system ;)

@screech I assume the 'midnight stuff' has to do with server maintenance or some such thing around midnight CEST. I am in Florida (USA) and we are 6 hours behind CEST. The thing is that for the most part, I have done the majority of my scrape tests in the later evening which puts the times after midnight CEST. I had been fighting with it for several days. However, last night around 2:00am to 3:00am CEST they worked fantasticly. I do not think I did anything to make a difference, but I had two 700 item sets finish with a 4-Mix at about 10 minutes each.

As for PSP, I have to disagree with you. While the PSP database may need some lovin, it does not explain why some of my titles that do exist on the web site will not pull. Let me give an example:
SSX on Tour.iso returns an empty image. When I look up the game on Screenscraper.fr:

The Name of the Game is "SSX on Tour".
The Game Name (by Region) is "SSX on Tour" for USA.
A Publisher, Developer, Rating, Synopsys (English), Screenshot, and Logo exists.
I don't think the matching criteria could get much better than this, yet I still get no Metadata or images.

I also wanted to say that you've got a really great little program here and it has a lot of potential for this hobby. I hope this all comes over as constructive feedback and not negative. :) Keep up the good work. I know a lot of people appreciate what you have put together.

hansolo77

How do I manually download the images hosted at screenscraper.fr? I just completed a massive scan of my Playstation (PSX) games, and all it downloaded was screenshots. I have UXS configured to download 2D art (box art) with screenshots as a 2nd choice. But it completely ignored the boxes. I only have like 40 games, so I can get the art manually if need be.

EctoOne

@hansolo77 I've never found a way to get the full size images from screenscraper.fr
Rightclick and save only gave me small thumbnails, so i got my missing covers from other sites like gamesdb or mobygames

hansolo77

That's what I had to do too.. oh well.

hansolo77

Just going through all my gamelist.xml files for each system.. looks like every single system has a random assortment of <image/> tags left over, right after the </desc>. When I say "random" I mean, it's not there for every single game.. just randomly scattered through the list, but always in the same location. What I don't understand is why the / is located AFTER the word "image" when it should be before. What's more.. why is it creating that line by itself in the first place? I only notice a problem with rendering the gamelist inside EmulationStation if there are no associated videos with the game. If there is a video, there isn't a problem. But when no video exists, the game list displayed doesn't have any art.

For a bit of history, if it helps.. I had a previous version of UXS create the original gamelist.xml. Then after a few subsequent updates, I've had it set to just UPDATE the existing xml to add back in the missing ROMs. So maybe the "bug" exists in the function to update? I have yet to try, but maybe it's not even a problem with the most recent version of UXS. It could be something with the versions prior. And creating a NEW list instead of updating might not have this problem either.

RedDog

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Just going through all my gamelist.xml files for each system.. looks like every single system has a random assortment of <image/> tags left over, right after the </desc>. When I say "random" I mean, it's not there for every single game.. just randomly scattered through the list, but always in the same location. What I don't understand is why the / is located AFTER the word "image" when it should be before. What's more.. why is it creating that line by itself in the first place? I only notice a problem with rendering the gamelist inside EmulationStation if there are no associated videos with the game. If there is a video, there isn't a problem. But when no video exists, the game list displayed doesn't have any art.

@hansolo77 I noticed the same thing when I was doing some manual editing. I got the feeling that the slash after the data object just represents an empty object. For instance, instead of using <marquee></marquee> it uses <marquee/>. I admit that it would make more sense to me for nothing to be present for an empty data object; smaller data files, less to parse, etc...but maybe ES isn't written with that in mind.

EDIT: Tested it with empty data sets removed. ES doesn't care. worked fine without. Maybe it is something that will be changed later.

tien_huu_1408

Dear all,
I tried Universal XML Scraper to scrap Neogeo Samurai Shodown series (fba folder) for Mix3 (mixed 3 images) on Retropie. I found media in screenscraper.fr. However, when finish scraping, just Samurai Shodown V and Samurai Shodown V special zero have full scrap (all information and mix 3D boxed art with screenshot and wheel). Could you help me scrap the others? Even I tried sselph scraper on command line with mix3 option did not work, it just come up with simple 2D box art. That's strange. Thanks for your help!
For example, Samurai Shodown 2 neogeo on the screenscraper.fr link
https://www.screenscraper.fr/gameinfos.php?plateforme=142&gameid=37630

hansolo77

The scraper works (by default) by checking the CRC and then the filename. If your ROM doesn't match the information in their databases, that's why it doesn't pull it even if it exists. Double-check the name. I managed to force a match by renaming several of my ROMs to match what the database had for it's name. A few of my games were listed multiple times in their database although with many different CRC results but none that matched what I had. So I ended up renaming it and then it worked. If doing that doesn't work either, you can always ask @screech to add your CRC to their list. I've not done that, but he's said in the past it's no problem.

tien_huu_1408

Thanks for your quick reply. I just used another romset and bumped it worked with full scrap. Viola!

mattrixk

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

I managed to force a match by renaming several of my ROMs to match what the database had for it's name

I need to do that for a bunch of my PSX and PSP games, but they all have the ":" character, which Windows won't let me put in the filename, so they will never scrape.

BORIStheBLADE1

I'm noticing it's not actually shutting down emulation station to scrap like it used to or letting me and none of the SSH commands are working . Anyone having the same issues?

Mike

hansolo77

I have an issue I'd like to recommend getting resolved...

Multi-Disk games don't include their (disc #) in the created gamelist.xml files. This is EXTREMELY annoying. For instance..

/home/pi/RetroPie/roms/America/Fear Effect (USA) (Disc 1).cue
/home/pi/RetroPie/roms/America/Fear Effect (USA) (Disc 2).cue
/home/pi/RetroPie/roms/America/Fear Effect (USA) (Disc 3).cue
/home/pi/RetroPie/roms/America/Fear Effect (USA) (Disc 4).cue

Generates a gamelist.xml file, and inside EmulationStation, the results look like this:

Fear Effect [USA]
Fear Effect [USA]
Fear Effect [USA]
Fear Effect [USA]

Is there a different setting I should be using under the Options->Name->Name Format area of the program? Right now, I'm using "Game name + Options". Should I be using one of the others, like perhaps "Game name + filename tag"? To me, that sounds like I'd generate a list with files called Fear Effect [USA].cue multiple times. Then there is the "Filename (without ext)" which sounds like it would create exactly what the file name is WITHOUT the .cue parts.. and that can get pointless too when you have excessively large file names.

So what's the best way to do this to just get a clean list like (even better without the [USA] bit:

Fear Effect (disc 1)
Fear Effect (disc 2)
Fear Effect (disc 3)
Fear Effect (disc 4)

screech

You can do it in 2 pass :

Move your multiDisk in an other folder, Scrape your Solo Disc like you want, Then readd your MultiDisk Game and Update the gamelist.xml.

Inbetween, you change the Name Format.
Game name + Options : is the Std one (you can choose option)
Game name + filename tag : Take the scraped Name and add the "tag" between () or [] from the filename at the end. (without extension)
Filename (without ext) : Obviously Filename without extension ;)

hansolo77

So what your saying is, if I pick the Game name + filename tag, it will take

/home/pi/RetroPie/roms/America/Fear Effect (USA) (Disc 2).cue

and create a

Fear Effect (USA) (Disc 2)

Or just up to the Disk 2 part, using only the first set of ()?

Panther7

This crashes no matter what I do, immediately after starting a scrape.

It says "hashing" on the bottom line, then stops responding until I kill the process. The log reads as such:

[03:18:32] - C:\Users\Redacted\Desktop\UXS\Scraper64.exe (1.2.0.10)
[03:18:32] - WIN_7(X64) - 0409
[03:18:32] - > systemlist.xml Opened
[03:18:32] - > Genrelist.xml Opened

screech

@Panther7 are you scraping iso ?
Big file May Take long time to hash. ...
Look at the uxs wiki There is several solutions.

Panther7

@screech I actually think I figured that one out. I was using the wrong password for my SSH. I didn't realize I had to independently sign in on that application. I still can't get the correct marquee download though. Didn't mess with it much more. I was only looking at it because the actual built-in scraper wasn't working.

I've spent all night trying to get the actual built in scraper to work. That thing is all over the place for me. It usually finishes without even scanning the roms (Dreamcast/Playstation). I'm at a loss, as I had it working fine just a few days ago.