[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
Awesome! Thx.
I test it as soon I get back from work. -
@FlyingTomahawk I had to update that xml file as it still had some unneeded parts in it, but it's fixed now.
-
I had a chance to test the code of yours and it does give me the correct entries inside the gamelist xml file but the videos are downloaded as "gamename"-video.png
I can see that the entry for video in your file is correct and says mp4 as file format but I think the Standard (2img) code is saying to save it as png. I tried to change that to match a video file and mp4 format but then I get all kinds of errors.
Right now I am scraping the roms twice, once for marquee and once for video. At the same time I set it to update the XML file to get the right entries.
That works good for me right now, I just thought I could reduce 1 scraping step and have both at once. -
@FlyingTomahawk Can you not just batch rename the *-video.png files? I'm using a similar scrape profil as @mattrixk has and it also downloads the video files as png. My script downloads all files (2D Box, marquee, screenshot, fanart, video), creates a Mix Image but only put the mix image in the xml (because i don't care about video, but i want to have the other images). So that's why i deleted all my *-video.png files after scraping without testing if those where just falsly named mp4 files.
-
Yes, I did that. I batch renamed them and I can do that for all the others too.
Still it does not reduce my work, I still need to do 2 steps either scraping, scraping or scraping, renaming.
But then again I don't think I will be doing a lot scraping once everything is setup, so a 2 step solution is not bad after all. -
@FlyingTomahawk
OH right, I forgot that you have to edit the xml files as well if you rename the files. I probably purged the memory from my brain about all the work I had with my setup. :D -
@FlyingTomahawk that's really odd. I didn't have a problem scraping videos, but I don't think I used that exact profile (I have lots of them scattered around my hard drive in various states of completion).
I'll hopefully get a chance to have a look at it tomorrow.
-
@FlyingTomahawk okay, I just had a quick look and this is what it should scrape:
<image>gamename-screenshot.jpg</image> <marquee>gamename-mix-video.png</marquee> <video>gamename-video.mp4</video>Are you maybe getting confused about the MIX image being called "-mix-video.png"?
-
hmm...nope not confused....I think.
I had the video files as png format and when I renamed them to mp4 all was good.
I am fiddling for 3 hours now and at the same time cleaning out my roms since I try to reduce it to max 50 roms per system. Let's be honest no one really needs 700+ roms per system, there is no way to play all the games anyway. That will reduce my work with this video stuff significantly.
So I might have a burnout at the moment and lost a bit overview and concentration. -
@FlyingTomahawk
Tell me about it. I probably spend a whole week just to get perfect Metadata. At the very beginning I've made the decision to use the standard 3 img mix, but I changed it to use 2D boxart. So I was able to simply create/fix images if they had missing elements or none at all (for Romhacks). Then go through the created xml with the Metadata Editor to check if everything is set correctly. It was such a tedious task, I won't even imagine to do that with complete romsets. -
@FlyingTomahawk okay, that's really strange and I have no idea why that's happening. I'll have to do some testing.
-
Yesterday before I called it a day I reverted everything. I deleted the UXMLS files and downloaded a fresh package from github.
Now I don't know what is going on but I have the following issues now.1.)
No matter if I choose RetroPie mix profile or regular RetroPie when I scrape now and select create new XML file as option it scrapes 3 images.
Marquee, video.png and screenshot.
Looking at the XML file there I have the right paths but at the end many empty </game> tags.- )
If I select "update XML file" in the options he doesn't scrape anything. 0/0 roms.
3.)
Previously I could download only videos or only images without creating an XML file by selecting "create image only" in the options.
When I do that and I select video it creates the videos in png format and also the regular screenshot images.I have no idea why this all is happening. I mean I deleted everything from my PC, restarted it, downloaded a fresh, new UXMLS package from github. Still it won't go back to normal.
I will take another look at it today but this is getting annoying. - )
-
So back at it again...new day, new luck?
Can someone please give the following a try and tell me what they get?
Load RatroPie MIX [Rom folder]
Select Standard (2img)
Settings in General options
Create New XML file
Picture configuration
default
Options configuration
Item Choice
Marquee
Paramater to change
Wheels
What files do you get and what file extensions do you see in the game list?
Load RatroPie MIX [Rom folder]
Select Standard (2img)
Settings in General options
Create New XML file
Picture configuration
default
Options configuration
Item Choice
Video
Paramater to change
Yes
What files do you get and what file extensions do you see in the game list?
-
@FlyingTomahawk hmm, very strange. Do things scrape correctly if you just use the built-in profiles? The latest version of UXS is v6, so if you're still having problems with that one, you could try v5 or v4. I don't think I'll have a chance to do any testing today (niece's b'day then mate's going away party), but I'll hopefully have a chance tomorrow.
-
That's the first thing I did this morning. I tested the Standard (2img ) and all was good. Correct images with correct file extension and correct gamelist entries.
I want to scrape that game logo image (wheel? ) only that you can see bottom-right on the Standard 2 and 3img profile.
And only the video.
So 2x scraping total.Just enjoy your day/weekend, please don't mind me. I will figure it out one way or the other, just a matter of time.
-
@FlyingTomahawk no worries mate, we'll get this worked out (eventually).
-
I swore this was already gone over, but you can't search this entire thread (keeps on loading only sections at a time).... I'm having an issue where scraping Playstation adds a gamelist entry for every single file a certain PS1 game will have (some have just a .BIN and a .CUE - others have like 30 .BIN's and a single .CUE - so that one would have 31 files listed in the gamelist). How to fix?
-
@Dochartaigh add an exclusion for *.bin and you shouldn't get doubles.
Also, to search a specific thread on this site just click in the thread the hit ctrl-f. You should see something like thread:678533 show up in the search box.
-
@Jeck11 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
@Dochartaigh add an exclusion for *.bin and you shouldn't get doubles.
Also, to search a specific thread on this site just click in the thread the hit ctrl-f. You should see something like thread:678533 show up in the search box.
Thanks, I'll exclude .bin's!
And this forum only loads so many posts at once - so there's 380 posts in this topic, and instead of splitting it into multiple pages like the most common Phpbb and simple machines (and similar) forums do, it'll only load the last 10 or 20 posts so that's all you can search for with CTRL+F unfortunately (unless I'm missing something).
-
Hello Every one.
Just pull out a fresh new release (v2.2.0.0)
What's new :
New Language Set
Now English Language are separated in 2 : English UK and English US. So now it handle correctly the media fallback (exemple : In UK fallback if a cover matching the rom is not found it will take the European one and the world one if not found. In US, it will take the US one and the world one... Same for the game naming)A complete new Wizard.
I force it at start to make sure every one have a "good" install first.
I put a lot's of "verification" to make sure a standard install can scrape easy.
You can always tweek what you want after that ;)A New "Options" Menu.
Now you will have a lot's of "easy" tweek" like :- Mix or not.
- If not Mix : selecting Default picture, alt picture and for Retropie : Marquee and Videos
- Naming convention : Default is "Game Name + Options" so you can tweek the options.
- Name Tag options : Now you can put the country between [], and some tags between () like Clone, Beta, Demo, Proto, Trad, Hack, Unl, Alt, Best. (To come : Clone type and Hack type) -> So normally no more simple duplicate name (for mame for exemple). If still duplicate, it's because we haven't the info in DB. Don't hesitate to submit correction on that on www.screenscraper.fr
- Case for multiple metadata : Default/UPPERCASE/lowercase (depending on Theme, it may matter)
New Profil :
Profil change "a lot"... Not so much modification of existing things, but only 2 profils now : Recalbox/Retropie. With a lot's of new options (at the end).
It create dynamically the option menu.
We loose some "options" but I will put them back later (Like the "rom folder" profil or the "scummvm" profil)... I need to work on these.
Special dedicace to @mattrixk if you have any difficulty with modification on these profil, let me know.New option in Gen config
It was asked, by default, UXS don't scrape "IN" zip file... So it may not found some game, but it's a way faster with romset like mame.
You can reactivate that in option.Carbon Wheel on the UI
Now you can know what plateform you are scraping, it put a little carbon wheel on the UI.
This wheel depend on you language preset (so Genesis for US or Megadrive dor UK ;) )Some debbuging
- Full Scrape works now ^^
- Full scrape have a good "Result" windows at the end
- Normally no more bug when scrape multiple system in the same session (you don't need to close and reopen UXS)
- Lot's of small debbuging.
To come :
- A new server (actually the small server we have suffer a lot. Specially at Week end) thanks to your donation we migrate soon to a new bigger and dedicated physical server.
- A new API (With the new server, little time after, we will have a new API with a lot's more new stuff to scrape and more easily ;) )
- A new Error handling (Actually error coming from the API aren't so easy to catch. So with the new API, I can have a better return and so a better information on what's going on when it doesn't work properly)
- lot's of new stuff in UXS ;)
Asking for Help
With this new version, some translation aren't up to date. So if you speak German, Spanish, Dutch, Portugaise, Japanaise or Italian, you are welcome to contribute.
(Juste look in the LanguageFiles Folder. Open the XML correspondinf. The stuff needing translation are at the end ;) )
If you think my English isn't so good (what is true) don't hesitate to correct the EnglishUK or EnglishUS XML ;)@FlyingTomahawk : Just Open the Advanced Menu, Image configuration and select "Default" as extension. It will solve all your extension problem ;)
Thank you guys, and don't hesitate to tell me if something don't work well ;)
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.