[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

[mattrixk]

@EctoOne
This is a WIP Scrape Profile for UXS.

http://pastebin.com/bZNYQ29h

It downloads screenshots, 3d box art (with 2d box art backup), fan art, game logo (wheel art), game cartridge (media) and game video. It saves them all to the 'downloaded_images' folder with suffixes of "-screenshot", "-boxart", "-fanart", "-logo", "-media" and "-video" respectively.

It also puts each of them into a gamelist.xml in <screenshot>, <boxart>, <fanart>, <logo>, <media> and <video> tags respectively.

It downloads each element as large as it can, up to 1024x1024 (videos will pretty much always be 640x480, as will screenshots, as that is the size of them within screenscraper.fr).

To use the Scrape Profile, save the xml as downloadeverything.xml (or whatever you want to call it), then put it in the "ProfilsFiles" folder within UXS. You can then choose it from the Scrape Profile menu.

Keep in mind that Scrape Profiles and MIX Profiles are 2 different things (something it took me a while to understand. I was confused as balls until I did, but once I did, it made things much easier).

Scrape Profiles are for scraping individual elements. MIX Profiles scrape the elements and combine them into a single image.

Once again, this is a work in progress, so I take no responsibility for your computer blowing up, or aliens abducting your loved ones.

[screech]

Sorry, it's impossible to launch UXS without a net connexion to the website... Too much problem after that...

But I have in my pretty big todolist : Make a MIX Generator soft ^^

Where you can scrape 1 game (with all the media/data it is possible) and create a MIX with them...
But it's a project nearly as huge as UXS >< so it's not for now ;)

If I can, I will try to make a "light" version of that ;)

[mattrixk]

No worries @screech. Love your work.

[screech]

Hi, the new 2.1.0.5 just come out.

A lot's of change :

• Corrected :

° Better stability on downloaded images (Some never download :S and miss in the MIX)

• Added :

° Dynamique Handle to API server, now I can force the server to ask (for the futur when we have a fallback server ;) or country based server )
° Lot's of new Picto media (Nb of player, Notes, Classification, Editor, Developer, Genre) -> Beta it can change soon.
° Added 3 news MIX Template for Bigshot Theme with these new picto (1 is directly on local, the 2 other are on the repository, you can download them directly from UXS)

• WARNING Added things that can "broke" you existing MIX Template

° You need to put the full Xpath to the Source_Value node (exemple : Game need to start by 'Data/jeu/' and System by 'Data/systeme [id="%IDSYSTEM%"]')
° The CENTER/LEFT/RIGHT/UP/DOWN function in Target_TopLeftX, Target_TopLeftY (and other) now take the "origin" point of the picture (default are TopLeft corner of the Picture)
° You can now use Target_OriginPicX and Target_OriginPicY to fixe the origin point of the picture. So now to center a picture you need to put :

• <Target_TopLeftX>CENTER</Target_TopLeftX>
• <Target_OriginPicX>CENTER</Target_OriginPicX>
• <Target_TopLeftY>CENTER</Target_TopLeftY>
• <Target_OriginPicY>CENTER</Target_OriginPicY>

(The Origin point of the picture will be the center, and we center this origin point to the final picture)

This is exemple of the New MIX for BigShot Template :

[mattrixk]

This is huge. Great work @screech!

[a7mag3ddon]

Is it possible to use this scraper on roms folder on the PC hard drive instead of on the Pi itself? I have a backup of my pi roms folder on the PC so want to run it on that instead.

[mattrixk]

@a7mag3ddon you can download the 32 or 64 bit version of the scraper here and run it on your PC. It's what I use to build scrape and mix profiles. (I don't even know how to use it on the Pi itself shhh, don't tell anybody).

[a7mag3ddon]

@mattrixk

Sorry i didnt explain, i have it for PC but i want to scrape FOLDERS on my pc instead of on the PI :-)

[mattrixk]

@a7mag3ddon Oh, sorry. It's late and I've had a big day. I can't really brain at the moment, I have the dumb.

[screech]

Just to inform you I just release a new version with a little change on Retropie Profil.

No more (Video) profil. Now you can add Marquee and/or video (+ the standard "image") juste with a litle configuration in the "Option configuration menu" ;)

@a7mag3ddon Of course you can scrape a folder on your PC, USB Key, external hardrive, network drive, or directly to the PI ;)
My advice : choose a Retropie Profil with [ROM FOLDER] in the name ;)

So Gamelist and downloaded_image folder will be put in the "rom folder" ;)

when selected (use the wizzard), just select the "main" folder with all the subfolder to the plateform rom. Then use the autoconfig ;)

[daeks]

@a7mag3ddon share your local folders as "roms" and "configs" and add "\\yourpcname\roms" to the scraper as path. Works fine.

[a7mag3ddon]

Ooohhh thanks! I think it will be quicker, and i dont have to have the Pi on for days when my pc is already on anyway.

Can i also ask. will it put the files and gamelist in the folder with them like it does on the pi?

[screech]

On retropie, you have to way to handle the Scrape :

What I call the "easy" one ;) All in the rom folder :
Exemple :
In your Rom/Snes/ you have

• your roms
• the gamelist.xml
• the downloaded_images folder (with all the picture in it).

If I don't mistake, it's the first place where emulationstation check for these info.
But it's not compatible if you scrape with the Retropie Scraper.
And the good thing, it's easily transportable.

So, for UXS, the "[Rom folder]" profile are these one.

The second way is the "standard" way Retropie default folder.

• Your roms are in : /home/pi/RetroPie/roms/%system%/
• Your gamelist.xml are in : /home/pi/.emulationstation/gamelists/%System%/
• Your picture are in : /home/pi/.emulationstation/downloaded_images/%system%/

So, for UXS, the "standard" profile are these one.

[a7mag3ddon]

See i want the gamelist.xml to have the correct paths to games/ artwork etc. but i want to do all the scraper FROM and ON the pc hard drive.

[screech]

so you need to "simulate" the Retropie standard path on your PC... like I did for the test ^^

Make a folder name Retropie.
and subfolder /configs/all/emulationstation/downloaded_images
and subfolder /configs/all/emulationstation/gamelists
and subfolder /roms

So you just need to put your different plateform folder (named exactly the same as retropie) in the Roms folder and just tell UXS you want to autoconfigure on this one ;) (with the wizzard and a standard profil ;) )

[C hicaned]

Hello!

Anyone experiencing this error when scraping roms :-

Seeing this error several times when scraping 800 Megadrive roms with 3 pictures per rom.

Should probably add I'm on the very latest version as well (2.1.0.6). I wasn't going to mention the 'unusual' error ;-)

Thanks,

Chicaned

[Nismo]

@C-hicaned error.. png fucked... lol XD

I'm sure @screech can help you with that.

[UDb23]

@Nismo Unusual error message lol!

[screech]

Oups !! sorry for this unusual error message ^^
It's a debug one who normally don't happen anymore (it's when I try to resize a picture and the original one have no dimension... so the picture is a fu£$*%^ one ;) )

Need to recheck why it can appear again :S

[Jeck11]

Will this scrape videos as well? Looking for a way to build up a collection for the new themes.