[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
-
Maybe its a permission issue. Running SSelps's script from Retropie-Setup will be executed as root while UXS connects as pi user. For example, if your ES starts from /etc/rc.local it might get started as root?
-
As far as I know UXS connects with the default pi user / password. If you have changed the password, it cant connect anymore. If I remember correctly this is hardcoded.
-
-
@Dochartaigh I have used sselph used only once way back when I got my Pi, so I don't know what the actual gamelist looks it creates. I do know that UXS adds some values to the game tag (screenscraper name and game id iirc) and also creates an unused thumbnail tag. And depending on your settings, it might be also creating the video and marquee tag as well I think. So with 1500 roms, I assume that's quite some useless data Emulation Station has to deal with.
You can try to pull the gamelists from the Pi and edit them to remove those tags to make the files smaller. If that doesn't work the only other thing I would try is to scrape my roms on my PC and not straight on the Pi. I for myself only scrape on PC, edit the files with RetroPie Metadata Editor to add/change the informations and then use a text editor to remove the values and thumbnail tag with notepad++ doing a search/replace. Never had a problem, although I don't have 1500 roms on my Pi. -
@retrolove I don't know how Mame roms are named but if the additional info you want to keep is in brackets, you can go to Settings, Options configuration, Select Name Format, Select Gamename+Bracket, Save.
-
@EctoOne thanks for the reply. Unfortunately that info is not kept in the brackets, it's almost like it is a giant tile. The issue is that a title like Street Fight II World Warrior Rev B USA would get shortened after the scrape to Street Fighter II World Warrior. The problem is I then have 4 or 5 games of the same name and no way to differentiate. If there was a way to tell UXS not to scrape or update the display name that would be fantastic and solve my problem, but unfortunately I can't figure out how to do it....
-
@retrolove In the same options you can also select to use the filename if that helps. I don't have access to a windows pc right now so i can't tell if it works.
-
Hi, sorry to be late (again) ^^
@retrolove :
For the naming : You can choose in the Option configuration 4 things :- Game name (it's the default one, and it's the name referenced in the DB... so you can have duplicate in case of multiple rom for 1 game -> Use this if your romset is 1game/1rom)
- Game name + Bracket (It add the info under bracket from your rom file to the game name referenced)
- Game name + Country (It add the country of your rom referenced in the DB to the game name referenced in the DB)
- Filename without extension (It simply take the rom filename, strip out the extension and put it as name... I think it's the option you look for ;) )
An other is under dev : Game name + Info (the DB begin to have more and more info about Hack Type, and Clone Type, so it will take the Game name referenced and add the info about the rom you scrape like REV A, 2 players, Beta, Proto, Translation,.... ) But it's not done yet :S
@Dochartaigh What UXS try to do when you want to "stop" Emulationstation is in the profil.xml you choose in the PLINK part. For Retropie standard are :
<Plink> <Ip>RETROPIE</Ip> <Root>pi</Root> <Pswd>raspberry</Pswd> <Command> <killall>killall emulationstation</killall> <reboot>sudo /sbin/reboot</reboot> <poweroff>sudo /sbin/poweroff</poweroff> </Command> </Plink>You can change these directly in the XML or (only for SSH connection parameter) in the SSH config menu.
When UXS try to kill emulationstation, it use a software name PLINK (it's putty in command line) connect with these info :
RETROPIE is the "IP" (you can put your real IP if there is problem)
Root : pi
Pswd : raspberry
(If you have changed these, you must change it in UXS too... It's the default value for Retropie (normally^^))And it launch the Command in SSH : "killall emulationstation"
If it doesn't work, and you managed to do the exact same thing with putty (for exemple) there is a problem :S
(And again, 40hour for 15000 Roms with 3 threads seems to be Very very long :S don't know what happen :( )
@EctoOne If you want the "extra" tag in the XML to be rip off, simply "edit" the Profil.xml you choose and delete the element you don't want.
Exemple :
<Element Type="thumbnail"> <Source_Type>Fixe_Value</Source_Type> <Source_Value></Source_Value> <Target_Type>XML_Value</Target_Type> <Target_Value>game/thumbnail</Target_Value> </Element>It's standard in the gamelist from ES (actually there is nothing in Source_Value, but you can put something if you want ;) )... But if you don't want it, simply delete it ;)
-
@screech Oh I never thought of that. Silly me. Also can you answer my question about scraping ScummVM games?
@EctoOne said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:Speaking of complicated scraping. How exactly do i scrape ScummVM? I've found a rar containing all shortname.scumm files you uploaded once. When i put them in my rom folder (or their own subfolders), it says Roms Path can't be reached.
-
@Dochartaigh What UXS try to do when you want to "stop" Emulationstation is in the profil.xml you choose in the PLINK part. For Retropie standard are :
Hi Screech, I wasn't talking about the kill/exit command Universal Scraper sends to the Pi to shutdown EmulationStation so it can scrape. I'm talking about in the RetroPie menu (on the Pi itself), when you hit Start > Quit > Quit EmulationStation, that no longer works as smoothly as it did before I scraped with your program (probably ~3-5 solid minutes to quit). It works perfect on systems scraped with the regular SSelph scraper (maybe 10 seconds) - others have chimed in your program adds more fields to the XML so it's probably with that it just needs more time to shutdown (albeit like 50 times as long...)
To the 40 hours to scrape everything, that was just how long it took over the course of 2 days at work remoting into my computer at home to start the next leg. Your program will NOT, ever, let me scrape all systems as once (always gives me those errors I posted about before) so I have to manually select the next system and let it run, then select the next system and let it run, etc. etc. etc. (and the hard part is simply remembering to keep checking the progress all day long so I can start it on the next system ;) It is TONS faster with the 3 threads at once - and I honestly think you at least need an account no matter what to ever have it work or it just times out or says like 12 hours to scrape a single romset.
-
@screech Thanks so much for the reply on the game naming, unfortunately none of those options are working with my setup. I do have a starting point of a gameslist.xml file with the game names exactly as I want them - you can see this with the additional info on screenshot 1. If I use UXS using either name or name+bracket then the scraper renames then to the name as it is in the screen scraperdb, see screenshot 2. As you rightly state with multiple roms of the same name it labels them all the same way. If I scrape with the filename option this doesn't work either as my roms are named shortname+zip, see screenshot 3 for outcome and screenshot 4 for my roms. I guess i have two questions:-
- Given I already have a gameslist.xml file with names the way I want them can I edit any of the UXS xml files to stop it scraping the name from screenscraper.fr? If this is possible this will solve the issue as I do want to get all of the other info apart from the name from screen scraper.fr (artwork, description etc etc)
- If the above is not possible do you or anyone know of a way I can take the name elements from my original gamelist.xml file (which has the names the way I want them) and then merge that in with the new gamelist.xml that UXS created, overwriting the UXS created names with my original one?
Any help would be hugely appreciated! And thanks for all your hard work on UXS - it really is awesome! :-)
Screenshot 1
Screenshot 2
Screenshot 3
Screenshot 4
-
I'm going to answer my own question about scraping ScummVM games. It doesn't work with RetroPie. At least not in a way that would allow for easy scraping and playing. To make it work, screenscraper.fr would need all the hashes from the different .svm files RetroPie uses to be able to start ScummVM games straight from Emulation Station.
At least i figured out how to use the .scummvm files from this thread over at the Recalbox forum, to get Mix Images.
If you want to get the images/videos:
- Get the RAR file from the thread mentioned
- Put all .scummvm files into a folder named scummvm in your rom folder.
- Open the profil xml you're using (e.g. Retropie (MIX).xml) in a texteditor.
- Add
*.scummvm;into the <Research> tag. - Save the file and open UXS to scrape.
What i did was to scrape inside Emulation Station (need to manually insert the right game name for every game) after ScummVM created all the .svm files. Then i replaced the scraped images with the Mix images from UXS.
Sidenote:
I also added*.uae;to my <research> tag and it worked right away only using scraping for names without hashes.
If anyone is interested, here is my profil.xml
It has support for uae and scummvm files enabled. It also downloads the 2D boxart, screenshot, marquee and (if available) fanart and video. BUT it doesn't add those to the gamelist! It only adds the selected Mix image and the normal Metadata. I also removed the <thumbnail> tag and the gameid/screenscraper name from the <game> tag.
The only issue with this is, that videos are saved as png files and so far i don't know how to fix it. Although since i don't use video view i don't care. -
@EctoOne I'm not sure how the .svm files look inside, but if they are empty, you can just do this:
https://retropie.org.uk/forum/topic/4142/test-a-new-data-source-for-sselph-scraper/17
It has a link to the samd recalbox tutorial. Just ñut the official long name inside the .svm file and you are ready to scrape wirhout too much hassle. -
@paradadf They are empty but whatever. I just finished scraping all images, so i won't test it. :P
In the end it was easier to scrape the whole scummvm folder from the rar and replace *.scummvm with *.svm inside the gamelist.xml UXS creates with the ones the GUI creates. -
New Feature - It would be cool to be able to use your combination engine to create an image offline. It would be able to take 3 images that we supply and combine them into the single image then write to the Pi.
-
I have been getting the following error when trying to use "Scrap all your system":
Line 22140 (File "C:\winapps\Universal Scraper\Universal_XML_Scraper64.exe"}:
Error: Variable used without being declared.
Because of this error I have had to scrape each system separately instead of just letting it run overnight. Is anyone else getting a similar error?
-
Yep :( sorry.... Scrape all is broken right now :( I need to correct it...
-
How can I scrape PORTS? I can use the ES scraper to find DOOM, DUKENUKEM3d, Quake... But not Outrun etc. Can I use U XML for this? Or any way to manually do this? I already tried editing the gamelist.xml under /home/pi/RetroPie/roms/ports but that doesn't work.
I tried another app (Emulationstation XML maker) but that results in the same and editing /opt/retropie/configs/all/emulationstation/gamelists/ports/gamelist.xml also doesn't work. :(
-
@stet said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
I have been getting the following error when trying to use "Scrap all your system":
Line 22140 (File "C:\winapps\Universal Scraper\Universal_XML_Scraper64.exe"}:
Error: Variable used without being declared.
Because of this error I have had to scrape each system separately instead of just letting it run overnight. Is anyone else getting a similar error?
I'm having the same issue with NES and N64. SNES was able to scrap just fine. I tried SCRAPE ALL and got that problem, hence why I was scraping them individually all day today. Just got down to N64 and NES when the "LINE 22140" error appeared.
-
Is it possible to create a custom Mix image profile with video and marquee similar to the Standard (2img) but instead of a screenshot have a video?
-
@FlyingTomahawk: There are 2 different Profile types:
- Scrape Profile
- MIX Profile
The Scrape Profile is the one that grabs the metadata, and downloads the single images (and videos) and creates the gamelist using them.
The MIX Profile gets the selected images and combines them together in the positions you want them.
You tell the Scrape Profile to create a MIX image, and then you tell the UXS program which MIX Profile you want to use to create that image.
eg:
You have 3 tags to fill: <video>, <image> and <marquee>.
- "md_video" uses "md_image" as a backup if the video is missing, so a Screenshot is best for that.
- "md_marquee" sits over the video/screenshot (in the z-index), so you might want the game Logo and 3D Box-art, so you want to make a MIX image for that.
You create a MIX Profile that combines the Logo and the 3D Box-art into a single image, and call the profile "Logo Box-art MIX".
You then create a Scrape Profile called "Scrape [video][image][marquee]" that:
- Downloads the Video and puts it into the <video> tag of the gamelist.xml.
- Downloads the Screenshot and puts it into the <image> tag of the gamelist.xml.
- Downloads the selected MIX Profile image and puts it into the <marquee> tag of the gamelist.xml.
In UXS, select the "Scrape [video][image][marquee]" Scrape profile. (You'll have to set up the autoconfig to point to your Roms, and also make sure the gamelist.xml has all the paths set correctly. You might need to play with that a little to get it right.)
Now set the MIX Profile to "Logo Box-art MIX" and scrape your Roms.
I suggest testing it first on a System that only has 2 or 3 Roms in it. It's annoying to scrape the data/images for a few hundred Roms only to discover the MIX Image doesn't look how you want it to, or you've missed something in the code so the Screenshot didn't download, forcing you to do it all again.
If you need examples of the code let me know, but you can usually suss it out from the Profiles provided with UXS.
-
I knew that it is possible to create mix profiles I just didn't know how to add the video part.
I wanted to take the Standard 2img profile and just replace the screenshot code with the video code.
You provided your code which will help me do that. Thanks.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.