Transfer ROM's with USB - Issue with files being stored as ".XML"

[Rohan82]

I don't have the Pi in front of me, but I can open an SFC file with a text editor and confirm that what I see on my laptop is the same as what I saw last night on the Pi. This is from "Super Mario World.sfc"

[pjft]

@Rohan82 Hi.

Not at all, thanks for looking into it. I'm just intrigued.

You have pretty much confirmed that things are fine on the PC - no need to spend more time with that particular aspect. :)

I'm keen now on understanding what exactly is getting to the pi.

Let me know if you're not comfortable with the Pi terminal and I'll give you some more commands, but ls -l is equivalent to dir, and cat <filename> will show the content of the file.

Was that the content you saw on the Pi when you opened one of the xml-extension files there?

[Rohan82]

@pjft Yes, that's correct. This is the same content that I saw on the Pi. I found it by selecting F5 (i think) to edit the file in File Manager. I was trying to edit the filename to remove the extension. After opening the file in edit mode, I realized that F6 was move/rename, so I tabbed over so that bith sides of File Manager were in the same SNES directory and used F6 to rename the file without the XML extension.

So just by chance, I was able to see the contents of the SFC file on the Pi.

[pjft]

@Rohan82 I am utterly puzzled then.

What does the runcommand log show when you try to run one such rom? It should be in /dev/shm/runcommand.log, if memory serves me right.

[Rohan82]

I meant check that last night but I got caught up investigating elsewhere and ended up going to be a little after midnight. I have a conference call in 30 minutes, but I will boot it up and check the log after that.

It's pretty strange to rename files during copy. I'm curious what is triggering it to do that. Like, what condition(s) need to be met in order to add the XML file extension.

[pjft]

@Rohan82 Yeah, apologies - I missed one of your questions.

No, the ROMs are exactly the same regardless of the OS, so my suspicion is that something must be off.

I'd recommend, worst case scenario, getting a MD5 hash of the file on the PC and on the Pi to compare them.

[Rohan82]

I'll have to google that. I write powershell, but I'm not a developer. And i'm definitely pretty light on unix-based systems since I worked for a MSFT consulting firm for the last 10 years.

It makes sense to compare the files on each system to see why one works and the other doesn't.

[Sano]

Ok, I read this and see no reason as well to rename files with xml suffix. Nothing in the scripts would have this behaviour.

First of all, in a vast majority of cases, you don't have to unzip roms to get nes/sfc/whatever files. The emulators are able to deal with a compressed file.

Second, could we have a screenshot of the detailed content (just a few lines) :

• of your rom directory on your computer
• of the USB thumstick after you copy the roms from Windows, but befor you plug it in the RPi
• of the raspberry pi rom directory after you plug the USB drive

Please confirm as well you copy the roms in the good directory of the thumbdrive : sfc in roms/snes, nes in roms/nes, ...

[Rohan82]

@Sano thanks for the response. I wasn't sure if I needed to unzip, and many tutorials included unzipping non-mame roms. As a test, I also transferred the zipped versions of the nes roms and got the same result. Appended xml file extension.

The screen shots in my previous posts might show you what you're looking for. The one from Windows Explorer that shows the file properties was taken from my rom source, which is a UNC path to my download server. The screen shot of the command window shows the contents and folder path of the USB drive used to transfer files to the Pi.

I'm not sure how to grab a screen shot on the Pi so I used my camera to take a pic of the tv screen after opening File Manager and browsing to the nes directory. You can see where I renamed one file to remove the xml extension, just to get it to show in emulation station. I tried to run that rom to generate a log file, but there was nothing much to see in the log.

Edit: I tried to add the photo of the rom directory on the Pi but I'll have to resize it first. I'll add it in a sec.

[Rohan82]

[Rohan82]

It's hard to read from the photo resizing, but the log file only has one line:
Parameters:
Execution: /opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-nestopia_libretro.so --config /opt/retropie/configs/config /dev/shm/retroarch.cfg

It's worth noting that I tried all three installed emulators with the same results.

[Sano]

Ok, can't explain this at all. Strange as f**k.
A nice test would be to compare original roms and renamed ones (byte size, md5, diff, ...) to see if the files are modified or just renamed.
Another approach would be to manually copy roms by other means, such as samba, winscp, or just a cp from the USB thumbdrive.

[Rohan82]

Well, I'm glad, at least, that it seems strange to others. It definitely doesn't make sense to me. I may not be an expert with the Pi or the distro, but I can't think of a reasonable explanation for a file copy action to add the file extension.

My next plan is to find my USB hub to give me a couple extra ports so I can manually copy from the USB stick and go from there. In the mean time, I'll some file comparisons to see if there's any differences and post the results.

[Rohan82]

Here is a file size comparison for Battletoads.nes on the USB stick, vs Battletoads.nes.xml on the Pi (just above the highlighted file.)

It's blurry, but you can see the USB file size is 262,160 bytes while the Pi file size is 263,696. That would seem to indicate something is happening during the automated file copy. I really need to try a different copy method for comparison to see if I get different results.

I will put a fresh RetroPie image on the card (using the same one I downloaded, for consistancy) before I try a new file copy method.

[Rohan82]

I'll have my USB hub sometime tomorrow, so I'll post the results of manually copying files via wifi or USB later in the day.

[Rohan82]

Well, I feel a little bit dumb here. I was asked several times if the files on the thumb drive were, in fact, xml files. Everything I checked from the Windows OS showed that the files were of the correct type (SMC, NES, GB, SG, etc). The files showed the correct file extensions and file associations.

For some reason though, the Pi was reading them as XML. I can only guess that something strange happened with the formatting of the USB stick that cause the Pi to view them as XML. When I used file manager to view the files on the USB drive, before transferring them, found they each had the XML extension following the true file extension.

I was able to transfer the files using the \RETROPIE\ unc path and everything worked fine. I'm still stumped as to why the XML showed up in the first place. I'm going to reformat the USB stick and copy some fresh roms to it and see if I can reproduce the issue. I will do one transfer manually, without putting anything in the roms folder on the USB stick. Then I will try one using the roms folder and the auto-transfer feature.

So far, All the consoles I've tested have worked except GBA (but I'm pretty sure that is because the bios file was saved using all caps.) I only found Super Mario All-Stars controller not working, which I think I read this is a known issue.

[Sano]

@Rohan82 Haha !
Now we are sure the file change is between your computer and the USB drive, we can narrow down the reasons of this behaviour.

After a quick search, it seems that Symantec Endpoint Encryption (and maybe other) automatically encrypt files as XML when copied on external storage.
Could you check this ?

[Rohan82]

I didn't even think of that. I recently started a new job and I was using my work computer to transfer the file from my media server (over UNC path) to my thumb drive. My previous work computer didn't really have any extra software on it like that, but this one does. I can almost guarantee that is exactly what happened.

I definitely feel like this is a Homer Simpson moment. Doh!! And i'm a little irritated I spent so much time on it knowing now what was happening. I could have transferred the files directly from my server and saved a couple days.

[Sano]

I definitely feel like this is a Homer Simpson moment. Doh!! And i'm a little irritated I spent so much time on it knowing now what was happening. I could have transferred the files directly from my server and saved a couple days.

Absolutely, but we all would have missed this little bit of knowledge :)
Moreover op @duviuvi has the same problem. Hope this thread can help him too.

Anyway, my work here is done
flying away

[pjft]

@Sano If I am to be nitpicky, flying away is a bit out of character for Sano. :) I'd expect you to go back to being a regular at Akabeko instead!

Well done on the investigation piece here, though!