[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

[screech]

@firewater you can help the DB with your ZIP ;)

We can add directly your zip to the "Mass Import" system... Just need to associate them (with an auto association engine, if your image file are well named it's a question of "minutes" to update these 2000~ snapshot directly on the DB ;) )

For the MIX, sorry, I haven't one like that, but it's pretty easy to make the change ;)

just have a look here : https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/wiki/[TUTO]-Making-a-MIX-Template

If you have problem, you can contact me (or if I have some time I can do it ^^ )

If you managed to do it, don't hesitate to share it directly on the Github MIX Repository (https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/tree/master/MIX Repository ) So every one can dl it and use it directly from UXS ;)

(Or if you can't push it to github, send it to me ;) I'll add it ;) )

[firewater]

@screech I found the zip and it turns out it was a mirror of these. I think these are the standard snaps most use, so it's weird that there are other ones (the ones already uploaded, with bilinear filtering and so on). The whole pack is uploaded on Mega and is updated to the penultimate romset version so I think it's fair to assume it would be a good update from the ones that are included already... but I'll leave my involvement of this part at that, I'd rather get the the mix template (and the rest of my setup) working first.

So I'll get back to you soon... or in some hours depending on how difficult it is, with a MIX template result!

[dankcushions]

@firewater said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

@screech I found the zip and it turns out it was a mirror of these. I think these are the standard snaps most use, so it's weird that there are other ones (the ones already uploaded, with bilinear filtering and so on).

right! this is a huge annoyance of mine with a lot of scraping databases. for any pre N64 era game, all screenshots should be lossless .PNGS at the exact resolution of the original game, and they should be about 30KB max. and other format will be loads worse looking and/or a larger filesize.

with stuff like mame, i've found some good snaps out there. see https://github.com/libretro/RetroArch/issues/4333#issuecomment-270173083

[firewater]

@dankcushions Yes! I agree completely. I also didn't know all (or most) databases did this, I've only ever dealt with boxarts most of my short frontend life so I assumed it was a specific problem on this one.

edit: well this is what I've got so far, it puts the box at the top and the screenshot at the bottom... more or less. Problems I've found is that 1. on NES, the boxart is not found 99% of the time (so it's empty) and 3. for arcade there's never a boxart at top (sure they don't have a "boxart" but it could put on a flyer, though I don't know how those are called).

<?xml version="1.0" encoding="UTF-8"?><Profil Name="Dual Vertical (2img)">
<Infos>
<Author>Anonymous</Author>
<Description>2 images assembled : 2D Box, and Screenshot</Description>
</Infos>
<General>
<Empty_Exemple>Empty_example.jpg</Empty_Exemple>
<Full_Exemple>Full_example.jpg</Full_Exemple>
<Target_Width>650</Target_Width>
<Target_Height>825</Target_Height>
</General>
<Compression>
<use>no</use>
<soft>pngquant.exe</soft>
<parameter>--force --verbose --ordered --speed=1 --quality=30-90 --ext .png</parameter>
</Compression>
<Root>
<Source_Value>Data</Source_Value>
<Target_Value>gameList</Target_Value>
</Root>
<Game>
<Source_Value>Data/jeu</Source_Value>
<Target_Value>gameList/game</Target_Value>
</Game>
<Country>
<Source_Value>Data/jeu/region</Source_Value>
</Country>
<Element>
<Name>box2dTEMP</Name>
<Source_Type>XML_Value</Source_Type>
<source_Origin>Game</source_Origin>
<Source_Value>Data/jeu/medias/media_boxs/media_boxs2d/media_box2d_%COUNTRY%</Source_Value>
<Target_Width>%0.5</Target_Width>
<Target_Height>%0.5</Target_Height>
<Target_TopLeftX>CENTER</Target_TopLeftX>
<Target_TopLeftY>UP</Target_TopLeftY>
<Target_Maximize>Yes</Target_Maximize>
</Element>
<Element>
<Name>screenTEMP</Name>
<Source_Type>XML_Value</Source_Type>
<source_Origin>Game</source_Origin>
<Source_Value>Data/jeu/medias/media_screenshot</Source_Value>
<Target_Width>%0.5</Target_Width>
<Target_Height>%0.5</Target_Height>
<Target_TopLeftX>CENTER</Target_TopLeftX>
<Target_TopLeftY>DOWN</Target_TopLeftY>
<Target_Maximize>Yes</Target_Maximize>
</Element>
</Profil>

[mattrixk]

@screech: I noticed you've updated the MIX tutorial, and that is awesome, thank you. However, I can't seem to get the Target_OriginX and Target_OriginY to work. I'm tyring to position the screenshot. If I don't use Target_Origin, the image shows up fine, with the Origin Point in the top-left of the image (basic default position).

If I use:

<Target_OriginX>%0.500</Target_OriginX>
<Target_OriginY>%0.500</Target_OriginY>

Then the screenshot image disappears completely.

If I use:

<Target_OriginX>CENTER</Target_OriginX>
<Target_OriginY>CENTER</Target_OriginY>

The screenshot shows up, but it ignores the Target_Origin, as though it isn't there.

Do I need to put it in a specific place in the element? Does it have to go before or after the Target_Width/Target_height? What about before/after the Target_TopLeftX/Y? Or does the position in the element not matter?

[screech]

@firewater nice job ;) I'll give it a try (not sur %0.5 on both size is the best ;) I think %1 on width and %0.5 on height with the Maximize will be better ;) (exemple on SNES with horizontal Box)

For the flyer, they are comming ;) (it's a new thing on Screenscraper and we are pretty poor on that.)

What about the Marquee ? Put Marquee if no 2D Box ? on top of Screenshot it may be nice.

@mattrixk I found a bug on the 'target origin' can you retry with the very last version ? Normally it will work better ;)
(No need of a specifique position IN the Elements node)

[firewater]

@screech thanks! Oh yes I could try other values for that, maybe that's what's causing NES not to pick up box art (because I remember when I had a smaller size they would always be picked up)?

Ahh that's okay. Flyer is the word I was looking for, but I'll wait for now.

I scrapped my arcade roms, and they all have their cryptic "fgtlayer.zip" kind of names, did I do something wrong or is this how it works? I wish they were changed to the name of the games.

[mattrixk]

@screech I updated to the latest UXS. I think Origin just works differently to how I was expecting it to. I thought it would work the same way <pos> and <origin> work in EmulationStation themes, where <origin> sets the Origin Point of the element and <pos> sets where the Origin Point sits on the screen.

I posted about how <pos> and <origin> work in ES here (complete with visual aides).

It seems like Origin in UXS sets the Origin Point of the screen, not the element, so every element is positioned relative to it. This unfortunately makes it virtually impossible to do what I wanted (and I can't figure out how it would be useful considering you can't do negative positioning).

What I want is to set the size of the screenshot image (eg: %25 wide, %35 high) with <Target_Width>, then position the center of that image in a specific part of the screen (eg: X = %40, Y = %55). I'm using <Target_Maximise>Yes</Target_Maximise> to keep the ratio.

I would do this in an ES theme like this:

<maxSize>0.25 0.35</maxSize>
<origin>0.5 0.5</origin>
<pos>0.40 0.55</pos>

So in UXS I used the code:

<Target_Width>%0.25</Target_Width>
<Target_Height>%0.35</Target_Height>
<Target_OriginX>%0.5</Target_OriginX>
<Target_OriginY>%0.5</Target_OriginY>
<Target_TopLeftX>%0.40</Target_TopLeftX>
<Target_TopLeftY>%0.55</Target_TopLeftY>
<Target_Maximize>Yes</Target_Maximize>

Expecting it to do the same thing. It doesn't. Instead it pushes the screenshot image off the bottom-right of the screen because the position is now taken from the middle of the screen instead of the top-left.

I tried setting both TopLeft and TopRight, but that just stretched the image, ignoring both Target_Width and Target_Maximize (but kept the correct height).

So, my question is: Is there a way to set the Origin Point of the Element, rather than the screen?

[screech]

@mattrixk OH !! Yes... You Are right ^^ (My Origin refere to the Screen origin to allow CENTER/LEFT/RIGHT on this Origin point ^^)

I think I never understand well how Origin works in theme ;)

I'll do some modification on the Mix Template function so ;)

Wait for the next release ^^

[mattrixk]

That's great to hear @screech.

While I have you, I have 2 questions about what the scraper can do:

1. Can you use the scraper to download the game videos? Maybe save them out as .mp4 or .gif?
2. Can you use the scraper to save individual images separately? Let me explain. Your default "Standard (3img)" MIX profile downloads 3 different images (screenshot, 3d box art and wheel logo) and combines them into a single image. Would it be possible to make the scraper download those same 3 images, but save them out separately? Either into separate folders within "downloaded_images", or with "-screenshot", "-boxart", "-wheel" appended to the end of the filenames?

[Concat]

So... when the scraper can't find a match, the recommendation is to change the file name?

When I do that, it seems as though I have to change the extension as well... and then change it back, and manually edit the gamelist so the extension is corrected there as well. Mind numbing progress on my Dreamcast games and PS games...

Then save states won't work after, so have to change name again and manually edit gamelist. Hmm...

Anyone figure out a faster way to resolve these orphaned roms?

[screech]

@mattrixk
1- It's already possible (I have add a new profil with video downloading in MP4 ;) )
2- It's possible, look at the scrapeprofile. They are in XML, you can add everything you want ;) (like the MIX Profile ;) ) And you can choose to put an extension like -screenshot, -boxart,....
Don't hesitate to ask if you can't manage to do it ;) I will send you a special profil ;)

@Concat If your Rom come from a well known source.
You can use F-CRC Calculator to calculate CRC and MD5 (don't forget to check the MD5 Checkbox).

Then send me a pastebin with the new Hash. I will add them to the DB ;)

It's the easiest way ;)

http://ffulgore.free.fr/site/f-crc/us/download.htm

[mattrixk]

That's awesome, thanks @screech. I'll hopefully have a chance to look at that tomorrow or the day after.

[bunman]

First off, thanks for UXS. My Retropie Picade and my Recalbox mini NES cased Pi now have an amazing looking frontend.

Quick question, is the server down? I almost finished scraping my systems, but now it completes hashing the roms, but finds no details.

Thanks,
Ali

[screech]

normally not... The server is up and responsive right now :S

wich system ?

[bunman]

Neo Geo Pocket and Color. There is a very good possibility it's just me! I'll check over my settings again. Thanks for getting back to me though.

[bunman]

I started UXS from a scratch installation and it works perfectly. I did say it was probably me! I'm just wondering if I didn't reset the profile when I went from scraping for RetroPie to Recalbox?! Should have just stuck to one OS for both Pis ;O) Anyway, thanks again for UXS.

[screech]

Normally you can swap from retropie to Recalbox easily, just change your profil ;)

[Concat]

@screech
Hey, for roms without a match, can I just send you the missing .txt file for each system after the scrape?

I created an account on screenscraper.fr, but I can't find a way of submitting these. All I found was info on how to update Roms and Games that already exist on the site. If it makes your life easier, I can follow the website's process. Looks like I get extra threads this way too...

Oh and if I haven't said it already, great tool. Thank you!

[bunman]

I'm having problems when I try to scrape just a 2D box image. I select 'Box 2D' for the default picture setting and 'Screenshot' for the alternative picture setting, but I'm mostly only get screenshots. I know there are box images as they appeared when I used the Mix profile. I'm using the Recalbox V4 profile if that makes any difference. Some systems come up with no boxarts at all, like the Lynx, then some, like 32X get most as boxarts, but 6 or 7 as screenshots. But if I used the mix profile, all the 32X pictures have box and screenshot, so the boxarts must be available?!