[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

rapidfire

i dont know if this topic is still active or not, but ive got several problems with the xml scraper

it keeps downloading corrupt or empty .PNG files.
these corrupt files slow down emulation station to a crawl and when you quit emulation station you can see a screen full of error messages about wrong dimensions in png files.
it keeps naming some of the roms in gamelist.xml as "(CLONE, PROTO, HACK ... etc.

when i first downloaded the scraper it worked perfectly on all games and gave nice results.
now its doing this

im using V2.2.0.1

retropie v4.2
a pi3 b

mattrixk

@rapidfire What Scrape/Mix Profiles are you using? Have you started using different ones recently? Are they the ones built into the system, have you made your own, or are you using the ones from the ES Toolkit?

mattrixk

@screech: I'm using v2.1.0.6 and I've been scraping stuff without a problem for the last couple of days, but I just got this error while scraping PC Engine:
Imgur

I haven't made any changed to the scraper or the Profiles used. The scraper made it 153 entries into a total of 234 items.

This is the end of the log file: Pastebin.

It seems to have borked at "Power Eleven". When I clicked "OK" on the error box, the program continued scraping the rest of the files, but it skipped the next 8 files after "Power Eleven".

It seemed to scrape the rest of the files without issue.

rapidfire

@mattrixk said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@rapidfire What Scrape/Mix Profiles are you using? Have you started using different ones recently? Are they the ones built into the system, have you made your own, or are you using the ones from the ES Toolkit?

i use the 2img profil
i was using 4img and arcade-zoomed-moon-retropie previously, but changed to 2img because i just wanted a screenshot and not much else

i see from another post you are using 2.2.0.6. where do i download that version from? i went to github and it only has 2.2.0.1.

ive never used estoolkit (didnt even know it was a thing until you mentioned it)

mattrixk

@rapidfire I'm using the 2106 version, not the 2206 version. I haven't really tried any of the 2.2 versions yet because they're a pretty big change from v2.1, and 2.1.0.6 does what I need it to.

I put together the Toolkit (with the help of Herb Fargus) so people could easily get started with making themes.

rapidfire

follow up to my problem.
just tried it again using only the visual option (not mix)

fba, and nes all give gamelist.xml similar to this (this is a small section from the fba gamelist)
zx spectrum downloads no images whatsoever. i know they exist on screenscrapers because sselphs scraper grabs them.

<gameList>
<game id="" source="ScreenScraper">
<path>/home/pi/RetroPie/roms/fba/1941.zip</path>
<name> (Clone, Beta, Demo, Proto, Trad, Hack, Unl)</name>
<desc/>
<image/>
<image/>
<marquee/>
<video/>
<thumbnail/>
<rating>0</rating>
<releasedate>00000000T000000</releasedate>
<developer/>
<publisher/>
<genre/>
<players/>
</game>
<game id="" source="ScreenScraper">
<path>/home/pi/RetroPie/roms/fba/1942.zip</path>
<name> (Clone, Beta, Demo, Proto, Trad, Hack, Unl)</name>
<desc/>
<image/>
<image/>
<marquee/>
<video/>
<thumbnail/>
<rating>0</rating>
<releasedate>00000000T000000</releasedate>
<developer/>
<publisher/>
<genre/>
<players/>
</game>

mattrixk

@screech: I've been using the same copy of 2106 on my work computer for a while now, but today I just started getting this error when I try to open the program:

Imgur

I get this error with the following versions: 2105, 2106, 2201.

When I first run the program, this is what the log says:

[12:34:53] - C:\MattK\UXS\Universal_XML_Scraper64.exe (2.1.0.6)
[12:34:53] - WIN_7(X64) - 0409
[12:34:53] - Scrape in x64
[12:34:53] - Starting files installation
[12:34:56] - Error Downloading URL : https://raw.githubusercontent.com/Universal-Rom-Tools/Universal-XML-Scraper/master/Ressources/ServerList.txt

Then when I try to run it again it says this:

[12:35:21] - C:\MattK\UXS\Universal_XML_Scraper64.exe (2.1.0.6)
[12:35:21] - WIN_7(X64) - 0409
[12:35:21] - Scrape in x64
[12:35:21] - > No updated files needed (Version : 2.1.0.6)
[12:35:21] - Starting files installation
[12:35:21] - > Ending files installation
[12:35:22] - Error Downloading URL : https://raw.githubusercontent.com/Universal-Rom-Tools/Universal-XML-Scraper/master/Ressources/ServerList.txt
[12:35:22] - /!\ Error Downloading File : C:\MattK\UXS\Ressources\ServerList.txt
[12:35:22] - /!\ Bytes read: 0
[12:35:22] - /!\ Size: 0
[12:35:22] - /!\ Complete: True
[12:35:22] - /!\ successful: False
[12:35:22] - /!\ @error: 13
[12:35:22] - /!\ @extended: 12029

When I go to https://raw.githubusercontent.com/Universal-Rom-Tools/Universal-XML-Scraper/master/Ressources/ServerList.txt it shows:

Fallback
https://www.screenscraper.fr/
https://mirror.screenscraper.fr/

I've tried both the 64bit and normal versions and get the same error message.

Note: This is happening on my work computer, as far as I'm aware, my home computer still works fine, but I haven't used UXS at home for a day or 2.

BORIStheBLADE1

Ok so has anyone scraped their ROM collection on their computer with this program? I've been trying to do it but it's not working right. It will scrap one system but after that it doesnt save any pictures. What folder structure are you also using?

Thanks

screech

@rapidfire Bug referenced (and corrected on the next version I must release) : It's happen when you check "Missing Roms" in the general config, and when the rom is not found (but Added to the XML)

@mattrixk Pretty weird... It's like UXS were blocked by your Firewall/AV ? or Github is inaccessible...

@BORIStheBLADE1 If you use well the wizard, it use the "default" path to store gamelist and picture on your PI (like the Retropie integrated scraper)
Be aware there is a probleme with the "auto kill es function". (When you scrape, UXS offer you to Kill ES, if it doesn't work you need to do it manually. It's already corrected in the next version... Really need to release :S )

BORIStheBLADE1

@screech
When scraping for the RPI it works fine. Im talking about scraping locally.

Ive used the wizard many times for scraping to the RPI. Im talking about "locally " to my PC.
alt text

hansolo77

So it's been a while since I've updated my system. I'm trying to get UXS to scrape but I've altered the paths and it's not working. In my setup, I've got all my roms on a hard drive, which is attached to the Pi. My plan was to use the larger capacity of the hard drive to store large games (.iso's, etc) and video screen shots. The way UXS sets up for default on the auto-configure, it tries to store all the artwork on SD card inside a nested folder for EmulationStation. I've changed it so it will put it in a subfolder with the ROMS. However, it's not working. It will go through and scrape everything fine. I can load up the created gamelist.xml and can see where it wrote out the location for the various artwork. But the path I told it use doesn't exist. "/home/pi/RetroPie/roms/%system&/art". Everything up to the "art" bit works, but it doesn't create that new path and save images there.

Is this related to the problem with the SSH commands not shutting down EmulationStation? I've got ES turned off before I scrape. I'm scraping again now, storing the scraped artwork locally on my computer and I will try to move them to the Pi's drive when finished. Is this going to be the only way I can do this?

screech

@BORIStheBLADE1 and @hansolo77

In ProfilsFiles Folder, open Retropie.xml

Look for the section <AutoConf>

and replace it by :

<AutoConf>  
<Source_RootPath>\\retropie\roms</Source_RootPath>  
<Target_XMLName>%Source_RootPath%\%system%\gamelist.xml</Target_XMLName>  
<Target_RomPath>./</Target_RomPath>
<Source_ImagePath>%Source_RootPath%\%system%\downloaded_images</Source_ImagePath>
<Target_ImagePath>./downloaded_images/</Target_ImagePath>
</AutoConf>

For both of you : Change the <Source_RootPath> with the "reel path to your rom's systems folder"
For @hansolo77 :
Change <Source_ImagePath> by : %Source_RootPath%\%system%\art
and
Change <Target_ImagePath> by : ./art/

It must work ;)
(You can do it through the interface ^^ but it's easier to copy/past ;) )

hansolo77

I've got it working by downloading locally and copying the files over. I think the problem I was having was due to SSH and permissions. It wouldn't create the path I needed, and didn't download the images to it.

BORIStheBLADE1

@screech Will give it a try, thanks

mattrixk

Hi @screech, you probably know this already, but the screenscraper.fr site is currently down. Did you ever get around to making the change that lets a user run the program, even if it can't connect to the screenscraper server?

I'd still like to be able to play with MIX profiles and other things that don't necessarily require access to the server.

Note: I still use v2.1.0.6 because it works well for me. I understand that v2.2.0.0 was a pretty major upgrade, but I just haven't had to the time to sit down and play with it yet to work out all the differences.

ortsac

Help is appreciated for a newbie here. Is UXS a separate from Retropie? Where can I get it and how to install it? I need to get download game videos and someone suggested it is a great avenue for such tasks...

AlexMurphy

@ortsac Yes, it's totally separate and needs to be run using a Windows PC. Link: https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/releases

ortsac

@AlexMurphy Awesome!!! thanks for the reply. will give it a try.

ortsac

@ortsac Does anyone know if screescraper.fr is down? I subscribed to the site and it replied afterwards that my credentials were saved but will require confirmation via link that will be provided on an email. I repeated the subscription about 30 minutes later, making sure the email address I provided was good. It's been 1 hour since and I have not gotten the confirmation link email. Maybe the site is not overseen during off hours or week end, so I'm wondering if the site is currently down?

AlexMurphy

@ortsac It's working fine for me. https://screenscraper.fr/ Have you checked your spam folder?

EDIT: It will still work without an account but you only get a single thread, so will take longer.