[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

Darksavior · 23 May 2017, 00:37

@AlexMurphy said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Darksavior Update it from github and tell it where to save stuff. Don't just click on the default settings.

It was already the latest version. I changed the defaults, no change. I'm not wasting my time any further. I'll try it again in a future version.

mattrixk · 23 May 2017, 00:38

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

My advice ... it will work ;)

That's awesome @screech, thank you. I'll give that a go when I get home from work tonight.

AlexMurphy · 23 May 2017, 00:41

@Darksavior It can be fussy and a pain sometimes. Other times it works great and the images are fantastic. Sure, what can you do?

kbronctjr · 23 May 2017, 06:41

@screech thx I will try this later today, although I have already tried by changing the name to the internal game name in screenscraper, but maybe the scraper doesn't check that name

screech · 23 May 2017, 07:08

@Darksavior If UXS can't connect the DB (and the DB is up) it because.... UXS can't connect the DB ;)
I can't have more information on why... (Check the log, I think you will find a Timeout error)

Check your AV/Firewall.

With about 2.000.000 requests a day, I think the soft work ;) If it's not the case on your computer, they're a lot's of chance your computer is the problem ;)

@kbronctjr the matching process is : Try to match the CRC or MD5 or SHA1 to one on the DB. If not, try to match the filename without extension to a filename in the DB (without extension). That's why You musn't rename rom ;) or at least keep a "standard" name (By standard I mean a well known name from a well known romset for exemple).

Darksavior · 23 May 2017, 07:38

@screech Yea I do have a log..
[00:28:35] - /!\ Error Downloading File : C:\xml\Ressources\systemlist.xml
[00:28:35] - /!\ Bytes read: 0
[00:28:35] - /!\ Size: 0
[00:28:35] - /!\ Complete: True
[00:28:35] - /!\ successful: False
[00:28:35] - /!\ @error: 13
I did disable my firewall. I do have an ad blocker running on the router, but since i can access the screenscraper website, I'll rule that out in being the problem. No worries, I have working alternatives to scrape with.

Rion · 23 May 2017, 10:09

@Darksavior Have you tried to whitelist screenscraper.fr in your ad block script/db?

Darksavior · 23 May 2017, 10:25

@Rion said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Darksavior Have you tried to whitelist screenscraper.fr in your ad block script/db?

No, since I can access the website on my browser and with sselph's scraper.

DynaMight · 23 May 2017, 16:04

Is there any chance you can make it so when you upgrade to a new version it keeps the settings from the previous version? I had everything setup but now upgraded to the latest version (Copied the new .exe into the directory where the prev .exe etc was) but it started up fresh again and started asking to be setup again, which normally brings up an error about not being able to find the roms folder then I have to end the task and restart it.

I dont personally use it to connect directly to my Pi so all my scraping is completed with roms on my PC and this program is a little tough to use for this unless you get the correct settings done.

It is an awesome program though.

kbronctjr · 23 May 2017, 17:50

The scraper can't find the following psx rom name:

Disney Lilo & Stitch en Problemas en el Paraiso (Spain)

This name is registered in the screenscraper database.

Another ROM name that doesn't detect is:

"Tom Clancy's Splinter Cell (Europe) (En,Fr,De,Es,It,Nl)"

Also registered in screenscraper

@screech

hansolo77 · 24 May 2017, 14:02

What is the best way to scrape systems like PSX that have multiple files for a single game? Meaning (for instance)...

Alien Resurrection (USA) (Track 1).bin
Alien Resurrection (USA) (Track 2).bin
Alien Resurrection (USA) (Track 3).bin
Alien Resurrection (USA) (Track 4).bin
Alien Resurrection (USA) .cue

This returns 5 copies of the same file, and 5 entries in my gamelist.xml. I just need one. Also, for SegaCD, I have things like this....

Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 1 of 6)[!].iso
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 2 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 3 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 4 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 5 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 6 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)[!].cue

Which, inevitably generates 7 copies of all the artwork, and 7 entries for the same game in the gamelist.xml. It's really quite annoying. So what is the best way to scrape this?

kbronctjr · 24 May 2017, 14:05

@hansolo77 go to the scraper settings and remove the .bin and .wav extensions from the list of file extensions to look for

hansolo77 · 24 May 2017, 14:05

Yeah that's what I'm trying now.. I removed *.wav and *.bin and put them in the excluded line... seems to be working.

kbronctjr · 24 May 2017, 14:07

@hansolo77 Also I think that retropie by default doesn't show rom files with .bin extension for psx, so only the entry of the .cue should appear in the front end

screech · 24 May 2017, 15:05

@hansolo77
I added in the last version this :

New Element Type : RomExcluded

Exemple :

<Element Type="RomExcluded">
	<Source_Type>Variable_Value</Source_Type>
	<Source_Value>%AutoExclude%</Source_Value>
	<AutoExcludeEXT>bin|img|iso|ccd|sub</AutoExcludeEXT>
	<AutoExcludeValue>(Track |[Bios]|(Bios)</AutoExcludeValue>
</Element>

Mean :
When a duplicate name file (without extension) is found:

it check the extension. If in list (bin|img|iso|ccd|sub) it won't scrape it
it check if the file contain a value ("(Track ","[Bios]","(Bios)"). If yes, it won't scrape it

normaly it works (on my test)...
You can add in <AutoExcludeEXT> the WAV format. have you the last one ?

kbronctjr · 24 May 2017, 16:06

@screech sorry to bother you, but do you know why the scraper doesn't detect some roms even when the names are registered in the list of rom names for that game in screenscraper?

EctoOne · 24 May 2017, 17:57

@kbronctjr The games you listed before contain special characters (i had the same problems with Chip's Challenge because of the ') and there always was a problem with them. The only thing you can try is to use the hash method instead of just names.

kbronctjr · 24 May 2017, 19:42

@EctoOne The problem is that the psx games are compressed in PBP format and the crc is not the same I guess.

The GBA splinter cell is a regular version but still is undetected with hash method

screech · 25 May 2017, 10:10

I still have problem with Filename Match with the '.
It's not only on my side... I ask the API with an URL request, so I can put ' directly in the url or urlencode it.. but in both case, the API don't find it... I need to work with the API dev but he (and me) are really busy these last time... can't manage to find some time to check that.. Sorry

Still the Hash method who is the best to be sure match is OK, but with PBP it's a problem (every PBP will be different so we can't add all PBP Hash in the DB, it's a nonsense. They're some, but it's because they are coming from "kown" romset)...

kbronctjr · 25 May 2017, 10:23

@screech don't worry, you already have done a lot of work by providing this application.

For one of the games I tried using and original .cue and hashing it and it worked, then I rename the rom name to the pbp one in the gamelist. Hashing bin takes a lot of time so if I find .cue for the remaining roms I will use that method