Universal XML Scraper images not showing up on ES

MrSkyle

Greetings fellow pie-eaters, I bring you yet another puzzle to solve: this time it's scraper-related! But before that, let me give you the hardware and software stats first:

Pi Model: 2
Power supply: HN-528i (input: 100-240V AC, 50/60 HZ; output: 5V/2000mA)
Retropie version currently used: 4.4
Image: fresh install on a Kingston 16GB MicroSD card
USB devices connected: 1 Edup Wi-Fi adapter; 1 Logitech Rumblepad; 1 Maxtor External Hard Drive (1Tb, USB 3.0 compatible with 2.0) where all the roms are located

The deal:
After trying first the SSelph scraper and then the built-in scraper and feeling somewhat disappointed with the results of both, I decided to try the Universal XML Scraper, using the "Retropie Mix" option (box art + screenshot + marquee).
I have all my roms stored in an external HDD, so I simply removed it from the Pi and connected it to my PC and then ran UXS, which downloaded and placed all the images in "\configs\all\emulationstation\downloaded_images" and all the generated gamelists in "\configs\all\emulationstation\gamelists".
However, upon connecting the HDD back to the Pi and launching Retropie, nothing changes. All the images that are shown are still the ones obtained via the previous scrapers - both the SSelph and built-in one. The new UXS images, even though they are still in their respective folders (I can still access them and see them when connecting the HDD to the PC), simply don't appear in EmulationStation whenever I launch Retropie from the Pi.

Any ideas on what went wrong?

mitu

@mrskyle My guess is that the gamelist files have not been saved in the correct location. The paths you mentioned seem to be specific for accessing the RetroPie file shares over network, so I'm not sure they'd work when using the HDD with the ROMs folders - for instance \\retropie\configs\ is not part of the ROM folder structure. Try to save the artwork and gamelists to the ROM folders directly, if there's such an option in UXS. Better yet, check your C:\ drive on the PC and see if there's an \config folder where UXS has saved your gamelist and artwork files.

MrSkyle

So the artwork and gamelists should be saved in the same folders as the roms? I could do that manually perhaps - simply copying and pasting them in the correct folder?
UXS has been saving everything in the external HDD via that path I gave, not the PC.

mitu

@mrskyle said in Universal XML Scraper images not showing up on ES:

So the artwork and gamelists should be saved in the same folders as the roms

Emulationstation expects a gamelist.xml in the following locations

The ROM folders
The /home/pi/.emulationstation/gamelists/<SYSTEM_NAME> folder.

The gamelist.xml file can reference the artwork anywhere on the filesystem, as long as it's accessible by ES, the downloaded_images folder is the default location used by the Emulationstation built-in scraper.
While you can copy the generated gamelist.xml to the ROMs folder, you'll have to make sure the paths to the artwork files (video/marquees/boxart) are correctly set up so that Emulationstation can find them on the disk.I haven't used UXS, but I know that Steven Selph's scraper has this option - using the ROM folder for gamelist and artwork:
Scraper configuration

Daistaar

@mitu is definitely right (as always lol). It sounds like EmulationStation is reading the "Gameslists" from the previous scrapes and not the newly created one by XML scraper. I would delete the previously generated gameslists (check the Retropie\roms<console> for them) using the built in File Manager (MC) or hit F4 to get to the terminal and type "mc" and hit enter. I'm sure you know this but the numbered commands are done with the Function keys, not the regular numbers (F4 to edit, even though it shows a 4).

Delete your previous gameslists and restart ES. Hopefully upon reinitialization of ES, it finds your XML generated scrapes and applies them in ES.

MrSkyle

Thanks for the replies @mitu and @Daistaar. Trying to erase the previous gamelists was my first goal, until I gave up after failing to find their location (/home/pi/.emulationstation/gamelists/<SYSTEM_NAME>). I'm afraid my noobism got the upper hand, as I couldn't find that path using Windows and I didn't even try it via Linux because that's pure hell on Earth for me... :
I did however check the ROM folders. The gamelists are not there. But I'm guessing they are still there somewhere. I just thought the new ones would replace them.

mitu

@mrskyle said in Universal XML Scraper images not showing up on ES:

/home/pi/.emulationstation/gamelists/<SYSTEM_NAME>

If you're using file shares (Samba) from the PC, the /home/pi/.emulationstation folder is available at \\retropie\configs\all\emuationstation.

MrSkyle

@mitu Nope, not using Samba. I simply connected the external HDD where all the roms are stored and ran UXS targetting the HDD as both the source and output destination.

In the meantime, I have managed to delete all previous gamelists and downloaded images via the File Manager that @Daistaar mentioned (I hate Linux so much...) and everything reverted back to its original format - just text. Which is fine for now.

Regarding the images obtained through the UXS, I dragged and dropped them inside the respective rom folder to no avail. Am I supposed to keep them within the "downloaded images" folder and drop it in the rom folder instead?

Edit: I forgot to put the "gamelist" xml file there as well. I did so and now I can see the metadata, which wasn't there before. As for the images, they remain absent.

steptoe

I came across a new scraper, by I believe the same guy who does UniversalScraper

https://www.skraper.net/

Takes a bit of setting up to get it right as it can scrape pretty much any emulation system, but it scanned all my RetroPie ROMs and correctly set-up the video and extras such as marquees without any issues and more importantly worked first time I booted RetroPie back up. All gamelist.xml were set-up and in the places RetroPie knew where to look plus extra game info

The only thing you need to do is register to get access to his scrape data, or pay a couple of euro extra to get more scraping threads, unless you leave it running overnight

It also has direct experimental support for PI over SAMBA but I couldn't get it to work

MrSkyle

@steptoe From the description it seems pretty much the same as the UXS. It even downloads the images from the same database.
After watching some tutorials and other threads in this forum I'm starting to realize that I will be needing some more tools to get this done, namely something that will connect my PC to the Pi via some sort of network, do the scraping on the PC and then somehow migrate the downloaded images to the Pi while it is still running...

steptoe

Are you running RetroPie off a USB hard drive or just the SD card/USB stick. I use a 1TB portable USB drive. Its tiny, and runs happily directly off the PI3, but I use an official PI power supply that is guaranteed at 2.5A so never seen the power warning

Should still work either way .....

I copied the RetroPIe ROMs onto the PC using FileZilla as any other way is just far to slow due to encryption, then ran SkraperUI and set it up as a GENERIC emulator config and pointed it to where I copied the ROMs to as the base folder, then messed about how I wanted the marquee, video, etc, etc to look and let it run. It auto detected all my systems straight away and didn't take that long. Maybe 2-3 hours considering how many ROMs I have

Make sure it stores the images and videos in the ROMs folder along with the gamelist.xml in the ROMs folder in the SkraperUI settings

I then simply copied all the downloaded data back across to the RetroPie ROMs folder, using FileZilla again as encryption is even slower copying back to the PI, I'm using an ethernet cable and its still painful using anything other than FileZilla than does a simply copy across the network without any encryption at all

FileZilla is FREE, and works. I set-up a quick connect and just log in using the quickconnect history tab. Fast and clean :

https://filezilla-project.org/

Once you copy everything back across, run RetroPie and everything should be there

MrSkyle

@steptoe Retropie is installed in a 16GB SD Card, while all the roms are stored in a 1TB external HDD connected to the Pi2.

Maybe what I did wrong was selecting the HDD to store all the downloaded images and gamelists instead of the Pi, but I did so because I simply connected the HDD to my PC when running UXS while the Pi was off. I thought that the program would automatically select the right paths and folders within the external HDD (and it did) and that somehow Retropie would be able to detect them. However, some people say they need to be in the same path as the roms (I tried that and didn't work either), while others mention a path that cannot be manually accessed via Windows, meaning that I would have to transfer everything via network to the Pi while it's running.

I'm assuming FileZilla (I'm unfamiliar with it as of yet) is one such program used to transfer all the images I have stored on the PC (or the external HDD in my case) to the right path in Retropie (I'm assuming it's some hidden directory in the SD card only accessible via the Retropie setup menu?), but for that purpose I will have to make Retropie visible in my network. How can I do that?

Sorry if I appear to be babbling but I'm trying to make some sense into something that is completely strange to me.

steptoe

If you look at the gamelist.xml, it explains it a bit more clearer as the path in the gamelist.xml points to the images and videos, so if its told to save all images and videos in the ROM folder there shouldn't be any issues as the path will be correctly set in the gamelist.xml and you can then copy the ROMs to your PC, scan them and then simply copy back the downloaded images and videos and not forgetting the gamelist.xml. It should then find everything as the path is now set to the ROM folder and not anywhere else. It could your path that is set wrong. I did consider search and replace the path in each gamelist.xml but once I had fugured out how to use SkraperUI just rescanned everything and it worked perfectly first time .... once I copied everything back over to the RetroPIE ROM folders

This is a copy and paste of just one game from my arcade ROM folder, the path is the important part. This is set to load images and videos from within the ROM folder. If you set-up UXs to save the gamelist.xml in the ROM folder, and tell it to look for images in the ROM folder you shouldn't have any issues.

SkraperUI has much more control over what is saved and how, but does take a bit more setting up

<game>
	<path>./p47.zip</path>
	<name>P-47 - The Phantom Fighter</name>
	<desc>P-47 - The Phantom Fighter (c) 1988 Jaleco.  P-47 is a two-dimensional, scrolling shooter type of game. You play as the pilot of a P-47. Your mission is to basically destroy as many of the enemies as you possibly can. This means eliminating as many air an</desc>
	<image>./downloaded_images/p47.png</image>
	<marquee>./wheel/p47.png</marquee>
	<video>./snap/p47.mp4</video>
	<rating>0</rating>
	<releasedate>19880000T000000</releasedate>
	<developer>Jaleco</developer>
	<publisher>Jaleco</publisher>
	<genre>Shoot'em up / Horizontal</genre>
	<players>1-2</players>
</game>

MrSkyle

@steptoe Thanks, I will check back the configurations of UXS and see what's wrong. So far the scraper has been downloading images and creating gamelists in a separate folder within the external HDD ("\configs\all\emulationstation") rather than the ROM folder, which is inside the "retropie-mount" folder. Moving the gamelist file into the ROM folder does make the metadata appear in EmulationStation, but the same doesn't happen with images...

I'll try to scrape a different system after trying to change the path and see what happens... please stand by!

mitu

@mrskyle said in Universal XML Scraper images not showing up on ES:

, but the same doesn't happen with images...

As @steptoe said, check the XML file and see where the path for the image/video/marquee is pointing. You can find/replace the wrong path with the correct one so the gamelist will be correct.

MrSkyle

Rejoice and praise the Pie in the Sky! It worked!

Yes, I changed the path configuration on the UXS scraper (there is an option called "alternate path" or something) and by simply selecting it, it sets the correct path for the gamelists and images - i.e., the respective rom folders!

Upon connecting the HDD back on the Pi, everything is there. My little dream machine has just taken a huge leap forward into Coolsville!

Thank you all for your patience and assistance!

steptoe

If one system works, they all should scrape correctly

This is basically what happened to me, the default points to emulation station and I just assumed that was that as I wanted just the ROMs in the ROM folder to keep thing neat and less confusing and all images and videos in one places. I didn't realise the gamelist.xml also has the filepath in it until I started looking at it to try and understand what was happening, so it was looking in the wrong places (wrong for me anyway)

Glad its all sorted now, RetroPie ain't the plug it and go system everybody thinks it i s, but goes a loooong way to almost being there. It needs some user effort to confugure it to how they want

Try configuring and using HyperSpin, thats fun ..... its the only way I'm going to get pinball in my arcade desktop I've been working on for about 18 months when I get a burst of enthusiasm. That was why I bought a PI to start with then started adding more systems and soon realised pinball just isn't a realistic option or the more powerful console systems like the WII

MrSkyle

@steptoe Precisely!

I have been setting up my Pi2 for almost a year now and it's been quite a bumpy but rewarding ride. Bumpy because I'm not remotely tech-savy and know diddly-squat about Linux, but rewarding for the experience in itself: building the romsets, configuring emulators, choosing ES themes and now scraping roms... seeing it all come together as something that I could only dream of as a teenager is truly heartwarming.

Very soon it will be finally ready to simply plug and play! :)