Can I Scrape to my USB stick with Selphs scraper?
-
@mitu said in Can I Scrape to my USB stick with Selphs scraper?:
@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:
Yes, the 1st will copy the ROMs from the usb to the sdcard (this is a ROM transfer method), the 2nd will auto-mount the USB and will run the ROMs from it.… after copying the RetroPie folder and all of its contents to the usb drive.
Just to be precise to avoid further confusion.
@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:
When I do a "df", I thought I would see the USB drive, but I don't.
You should if it's mounted in the directory tree. By the way,
df -hgives you a "human readable" (-h) output that lists the size in K/M/Gbytes instead of filesystem blocks. It's a quick way to check the filling level of all mounted file systems. -
@clyde @mitu
Even if it's not official supported ... Symbolic links will help out in this situation.
Old thread: can-my-usb-device-have-data-on-it-already -
@clyde @mito, would you recommend running everything off the USB? Seems like a win-win. Only con is something else sticking out of the Pi. Should be faster, more reliable, easy to move, backup, etc.
Also, will the second method (make a retropie-mount folder on the USB drive and have retropie automatically copy the retropie folder and leave the roms already on the stick) and then run everything off the USB keep what roms are on the USB? I will back up the USB first, but if I just add a retropie-mount with it in my PC, then move it to the Pie, will it do what it needs to do? Or would it erase the roms on the USB or even see that it is supposed to copy anything because it already saw the "retropie" folder and copied the roms and this is a one time thing?
Also, is the rom folder then still visible so that I can copy roms to and from the USB drive from my Windows PC instead of having to use a Samba link, SSH, etc.?
And last, the link @cyperghost gave mentioned issues if I move everything over because fat32 does not use linux permissons and I might have a lot of trouble configuring things to run scripts or whatever else I need for MAME and Dos games. I mostly care about the arcade games and less so with the consoles.
-
@flightrisk The usual reasons for running ROMs off the USB drive is reliability and portability. The USB is less likely to be corrupted than a SD card. Of course, if you're removing it, then you'll only see what you had previously copied on the SD card in the ROMs folder.
There is no '2nd method', the 1st method is for transferring ROMs via the USB drive onto the SD card, the only method to run off the USB card is this to use the retropie-mount folder on the USB driver and copy all the ROMs there. The 1st time after you insert the USB on the drive, the ROM folders will be created under the retropie-mount and you'll be able to copy ROMs to any of the sub-folders to make them appear on the RetroPie system.
As for the issue mentioned by @cyperghost, the problem is that symlinks are not supported by FAT32 partitions so you would not be able to symlink the artwork from the SD card folder to the ROM folder.TL:DR - use the automated method mentioned in the docs (https://retropie.org.uk/docs/Running-ROMs-from-a-USB-drive/) and you'll be fine.
-
@mitu said in Can I Scrape to my USB stick with Selphs scraper?:
As for the issue mentioned by @cyperghost, the problem is that symlinks are not supported by FAT32
There are other issues with this method too: ie it doesn't always keep the same name so your symlinks may break, it's just not a very elegant solution.
-
@herb_fargus @mitu
Indeed the method is not very elegant but it's possible that this work without rescrape if the image folder will be copied to USB and a symlink that points from sdcard to USB is created. The other way a symlink file on a FAT32 does not work. So this is just a fast workaround.But we want to avoid diversity in setup for maintaining reasons!
-
The more I think about it, the more I don't understand why the gamelists and images aren't stored parallel to the roms by default. Same goes for all configs, now that I think about it.
-
@stoo Because you can use relatives pathes.
But I see no advantage in this./home/pi/RetroPie/roms/gb/Tetris.gbfor a GameBoy game
can result to a imagepath./Tetris.jpgI would also like to see a database like
SQLite
That would make things much easier ;)
So maybe every system got's it's own DB in a folder/home/pi/RetroPie/scraperdata/ -
@stoo Regarding images and gamelists, I suppose it's the default for ease of use and cleanliness. Same types of files are in the same directory, so you could do batch processing more quickly and safely, you can easily only backup roms and not scrapped data, ... It also allows to have default similar configurations for all setups, which is better for dev and support. It's quite logical IMHO.
Let's illustrate : You don't have a "Friends" folder containing pictures/videos of them, their VCF contact vcards and all your mail history with them, do you ? :)For configuration files, it's an all-time unix best practice to store them aside from data. Again for backup and security reasons, data management (different partitions for data), and because an unix/linux systems are multi-users. A config file is nearly always system wide, but data is by nature specific to a user.
-
@FlightRisk As most of you questions were answered by the others, I'll just add some information.
@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:
I will back up the USB first, but if I just add a retropie-mount with it in my PC, then move it to the Pie, will it do what it needs to do? Or would it erase the roms on the USB or even see that it is supposed to copy anything because it already saw the "retropie" folder and copied the roms and this is a one time thing?
As far as I know, it doesn't erase anything, but just copies the files. It may overwrite any existing files in the same subdirectory and with the same name, but that should seldom be a problem and that's what backups are for, after all.
Also, is the rom folder then still visible so that I can copy roms to and from the USB drive from my Windows PC instead of having to use a Samba link, SSH, etc.?
The directory structure in the directory
RetroPiewill be recreated on the USB drive, and the drive stays formatted with VFAT, so you should be able to access your roms on it from Windows. However, the roms on the SD card won't be accessible at all as long as the USB drive is connected to the Pi, because the USB drive's directory structure will be mounted over the respective directories on the SD card. To access the latter over the network or from within the Retropie system, you'll have to start the Pi without the USB drive connected to it (there are other ways, but this is the easiest one).@cyperghost @mitu @herb_fargus Does the automatic method set the
execflag? Just curious, because the manual method only mentions it as an option in the docs. -
@sano Yeah. It would just be nice if you could backup your home dir and know that all your scraped data and configs are covered.
-
@stoo Actually if you backup your whole home dir (/home/pi) it should be the case, provided your backup method doesn't ignore hidden files and follows symlinks.
You can also just add the/opt/retropie/configs/all/emulationstationpath to your backup task. -
It is all clear to me and working now, so thanks to everyone. I think the Wiki should have a warning box that creating a USB drive with a "retropie" folder is very, very different that creating one with a "retropie-mount" folder. I made a retropie folder on the USB days ago and thought I was running RetroPie on the SD card and the roms from the USB. Instead, as was pointed out here, it was just copying the roms on the stick to the SDcard. Pull the stick after it loads them up and things still work. Creating a "retropie-mount" folder copies all of retropie, files, images, roms, etc. from the sdcard to the USB drive and runs it all from the USB. The USB overrides the sdcard version of retropie as long as it's plugged in. Take the USB out and the last version of what was on the SDcard works just fine. Remove retropie and all the roms from the sdcard, and then the USB drive must be inserted to play the games.
If something goes wrong after plugging in the USB drive during the conversion (like shutting down before the migration is finished) , you can just delete the retropie-mount folder and stick it back in the Pi. It will automatically start copying again. I hate there is no progress indicator anywhere. This service runs in the background. And while I always buy USB drives with led indicators, this time I didn't realize I was buying one without. What a pain. I watch the green light on the Pi and run a "df" on it every half hour or so (had over 1400 roms with scrapes) and watch to see the bytes used keep going up. When it stops changing for a while, I'll know it is finished (or locked up ;) ) So another big red box that should be in the Wiki is "This process runs in the background and you will have no indication of whether it is finished unless ... <fill out the rest>"
-
@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:
And while I always buy USB drives with led indicators, this time I didn't realize I was buying one without. What a pain
I usually monitor it over ssh as I don't have any fancy flashy usb sticks either
-
@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:
I think the Wiki should have a warning box that creating a USB drive with a "retropie" folder is very, very different that creating one with a "retropie-mount" folder.
i mean, the guides for the two things are on entirely different pages. i don't know what the warning would achieve. it's like us saying "don't do something that we haven't told you to do here" :)
-
Quick question. When using the RetroPie-Mount on an external 1TB drive it takes a little over 3 minutes to boot from the emulationstation logo screen.
I had previously used another solution and it didn’t take much longer than using just the SD card.
I have the full PSX and SegaCD collections so there is a ton of data (more than 800gb in all). Has anyone seen this on their setups?
Pi3, RetroPie 4.3
-
@chuckyp Yeah... gonna play all those games are you?
Love a bit of Mary-Kate & Ashley: Magical Mystery Mall?
Can't wait to get back to Barbie: Explorer?
Going to unleash your creativity in Make My Video: Marky Mark and the Funky Bunch?
I'm trying my best to be polite here: maybe a more curated collection would be a good idea?
In all seriousness: https://github.com/RetroPie/RetroPie-Setup/wiki/FAQ#why-does-shut-down-and-reboot-take-ages
Try turning on Parse Gamelists Only so it doesn't scan for new ROMs at startup.
-
@stoo I’ll try your suggestion. And I’ve played the hell out of Babie explorer by the way. :-)
-
@dankcushions Because people google something like "copy roms to USB" or "run roms from USB" and you read a link that seems to give you the answer, but it trips you up because it is not clear, you miss something, or there isn't a link to other options. Just the distinction between the words "transfer", "copy", "run from USB" can mean different things to different people. "Transfer" to me meant transferring my games to the USB to run there. Instructions in 2 different places that say to create a folder called "retropie" or "retropie-mount" and it lets you use roms on a USB are very similar. One could even think one is a revision in a later version. And you are at the luck of the draw in your google search as to which one you get.
I am an engineer and owned an arcade in the 80's and 90's, not a completely unintelligent guy (though maybe slipping in my middle age ;) ) and I got it wrong. Not all of us PC folks are as fluent in Linux and emulation is a very complex and convoluted hobby just by nature of a bunch of volunteers working with many different open source pieces. I deal with going to resources like the MSDN every day because I write code directly to the low-level Windows API, and they all have "see section xxx for related info". The API for the low level audio drivers and multiple conflicting speech APIs for one, remind me of this confusion over two different features of "automagically" doing things with roms using a USB drive. Something like, "Don't use this function to run roms from the USB drive, this is just a method to transfer roms from a PC to the Pi. If you are trying to actually run your emulators and roms from the USB drive instead of the SDcard, use the instructions in section xxx" and a cross link from there back would be helpful, IMHO.
And you guys can do as you will, it is just a suggestion. I'm just trying to answer the question you asked and hopefully see how things to someone not as knowledgeable as you can become confused. I researched this for a few hours googling into several forums and what also tripped me up is that there are a lot of other people getting confused and passing along the wrong information. You can see where people are using the wrong folder name for what they are trying to do.
-
Old thread, but this is what i did.
Once the usb is setup and all files have copied move the scraper files to the retropie-mount folder on the USB.mv ~/.emulationstation/downloaded_images /media/usb/retropie-mount/ mv ~/.emulationstation/gamelists /media/usb/retropie-mount/Since retropie automounts the items in the retropie-mount dir into the ~/RetroPie/ dir we can create symlinks to point the that ~/RetroPie/ and no need to change the sources in the files.
ln -s ~/RetroPie/downloaded_images/ ~/.emulationstation/downloaded_images ln -s ~/RetroPie/gamelists/ ~/.emulationstation/gamelists
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.