[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

AlexMurphy

@kbronctjr It's just ALL about you, isn't it??? Have you tried rubbing a potato on it at midnight and burying it under a mulberry bush?

kbronctjr

@AlexMurphy I tried burying it under a RASPBERRY bush but no luck

AlexMurphy

@kbronctjr You still have the warts? See a qualified physician or a TV doctor... whichever your healthcare plan allows for.

screech

As I says, it's a false positiv... and no trace of wannacry... (I'm an IT in a compagny and pass my last week preventing he don't come to my compagny ^^)

By security, I make some test :
https://www.virustotal.com/en/file/9a15495065c2b979b04fc92d5a2051c8cbc13e942dc1e1a8b25f2004b0452183/analysis/1495453332/
The 4 first result are "generic" Malware code... (something like : "the file want to execute an other exe, or the file acces internet, or the file open XML".... And it's true ;) but it's not a malware ^^ it's the main function of UXS :p

https://virusscan.jotti.org/fr-FR/filescanjob/g4czisptuw
ClamAV detect one, but it's a well known false positive...

Tested on Kapersky too :
Résultat de l’analyse : le fichier est sain
Taille du fichier : 6,45 Мо
Type du fichier : PE64/EXE
Date d’analyse : 22 mai 2017 13:54:51
Date d’édition des bases : 22 mai 2017 11:34:53 UTC
MD5 : 2409df4503fdcc49cffcc7360870090b
SHA1 : e3a52966542fd0f8e6123c5baeda2a38440ce246
SHA256 : 9a15495065c2b979b04fc92d5a2051c8cbc13e942dc1e1a8b25f2004b0452183

And on Metadefender :
https://www.metadefender.com/#!/results/file/ZTE3MDUyMkJ5Wnh1dUdJeC1XU2tHZXVfR0xnV2I/regular/analysis

No threat found...

I send about 30 mails to majors AV antivirus to put UXS in their trusted list... I already have some response, but not all of them.
Windows Defender is "the worst" one... It's the "better than nothing" solution from microsoft but it put lot's of false positiv.

For the other problem :
PBP file aren't all referenced. The best way to scrape them : rename your file with a referenced FILEname. Just search on www.screenscraper.fr your game. look at the end of the web page. You will found all the referenced file. Just take one matching (country/language).
The matching is done on the Filename without extension. So it will work ;)

Creating Mix Outside UXS : I'm working on a solution with Imagemagick (UXS convert Mix profil to Imagemagick command line). But it's not done yet... sorry... I'll keep you up to date ;)

AlexMurphy

@screech Thorough as always @screech . Yeah, "Windows Defender" is pure merde. Why do people tolerate it on their systems?

Darksavior

Anyone having issues using this? It runs the first time, finds my files, doesn't find any roms (0/0), takes a few minutes to check the hashes, but scrapes nothing. Restarting it gives me the error popup that it failed to connect to the database. The screenscraper site seems to be up and working from what I can see. It was down all weekend.

AlexMurphy

@Darksavior It may well be working. Did you specify the path / file to save the images to?

Darksavior

@AlexMurphy said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Darksavior It may well be working. Did you specify the path / file to save the images to?

I never had to. It just asks for the rom folder and that's it. It generates the saved images folder by itself. They're just empty. I've used this months ago but I wanted to scrape some other roms I had. It still doesn't explain why it won't connect to the database again if I restart it. I'm just going back to sselph's scraper for now.

AlexMurphy

@Darksavior Update it from github and tell it where to save stuff. Don't just click on the default settings.

Darksavior

@AlexMurphy said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Darksavior Update it from github and tell it where to save stuff. Don't just click on the default settings.

It was already the latest version. I changed the defaults, no change. I'm not wasting my time any further. I'll try it again in a future version.

mattrixk

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

My advice ... it will work ;)

That's awesome @screech, thank you. I'll give that a go when I get home from work tonight.

AlexMurphy

@Darksavior It can be fussy and a pain sometimes. Other times it works great and the images are fantastic. Sure, what can you do?

kbronctjr

@screech thx I will try this later today, although I have already tried by changing the name to the internal game name in screenscraper, but maybe the scraper doesn't check that name

screech

@Darksavior If UXS can't connect the DB (and the DB is up) it because.... UXS can't connect the DB ;)
I can't have more information on why... (Check the log, I think you will find a Timeout error)

Check your AV/Firewall.

With about 2.000.000 requests a day, I think the soft work ;) If it's not the case on your computer, they're a lot's of chance your computer is the problem ;)

@kbronctjr the matching process is : Try to match the CRC or MD5 or SHA1 to one on the DB. If not, try to match the filename without extension to a filename in the DB (without extension). That's why You musn't rename rom ;) or at least keep a "standard" name (By standard I mean a well known name from a well known romset for exemple).

Darksavior

@screech Yea I do have a log..
[00:28:35] - /!\ Error Downloading File : C:\xml\Ressources\systemlist.xml
[00:28:35] - /!\ Bytes read: 0
[00:28:35] - /!\ Size: 0
[00:28:35] - /!\ Complete: True
[00:28:35] - /!\ successful: False
[00:28:35] - /!\ @error: 13
I did disable my firewall. I do have an ad blocker running on the router, but since i can access the screenscraper website, I'll rule that out in being the problem. No worries, I have working alternatives to scrape with.

Rion

@Darksavior Have you tried to whitelist screenscraper.fr in your ad block script/db?

Darksavior

@Rion said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@Darksavior Have you tried to whitelist screenscraper.fr in your ad block script/db?

No, since I can access the website on my browser and with sselph's scraper.

DynaMight

Is there any chance you can make it so when you upgrade to a new version it keeps the settings from the previous version? I had everything setup but now upgraded to the latest version (Copied the new .exe into the directory where the prev .exe etc was) but it started up fresh again and started asking to be setup again, which normally brings up an error about not being able to find the roms folder then I have to end the task and restart it.

I dont personally use it to connect directly to my Pi so all my scraping is completed with roms on my PC and this program is a little tough to use for this unless you get the correct settings done.

It is an awesome program though.

kbronctjr

The scraper can't find the following psx rom name:

Disney Lilo & Stitch en Problemas en el Paraiso (Spain)

This name is registered in the screenscraper database.

Another ROM name that doesn't detect is:

"Tom Clancy's Splinter Cell (Europe) (En,Fr,De,Es,It,Nl)"

Also registered in screenscraper

@screech

hansolo77

What is the best way to scrape systems like PSX that have multiple files for a single game? Meaning (for instance)...

Alien Resurrection (USA) (Track 1).bin
Alien Resurrection (USA) (Track 2).bin
Alien Resurrection (USA) (Track 3).bin
Alien Resurrection (USA) (Track 4).bin
Alien Resurrection (USA) .cue

This returns 5 copies of the same file, and 5 entries in my gamelist.xml. I just need one. Also, for SegaCD, I have things like this....

Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 1 of 6)[!].iso
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 2 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 3 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 4 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 5 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)(Track 6 of 6)[!].wav
Wing Commander (1994)(Electronic Arts)(NTSC)(US)[!].cue

Which, inevitably generates 7 copies of all the artwork, and 7 entries for the same game in the gamelist.xml. It's really quite annoying. So what is the best way to scrape this?