Can I Scrape to my USB stick with Selphs scraper?

mitu

@flightrisk Even if you move the scraped art to the ROM folder, you'd still have to correct the gamelist file and replace the paths, so it would be easier to remove the files and re-scrape again.

The scrapers (selph's scraper and the build in one) both store the images in /home/pi/.emulationstation/, you should check that folder's size and clear up some room.

stoo

https://github.com/RetroPie/RetroPie-setup/wiki/scraper

Enable "Use rom folder for gamelist & images"

Scrape again.

If you don't want to scrape again, move the images to an images subdir of your roms dirs, move the gamelist.xml to their respective rom dirs and do a search and replace on the gamelist.xml files to point to the new location.

Clyde

@stoo said in Can I Scrape to my USB stick with Selphs scraper?:

If you don't want to scrape again, move the images to an images subdir of your roms dirs, move the gamelist.xml to their respective rom dirs and do a search and replace on the gamelist.xml files to point to the new location.

Those who don't shy away from a little Linux command line magic can use the tool "sed" for this:

sed -i 's/.\/images/\/media\/usb0\/images/g' gamelist.xml

This changes all occurances of ./images to /media/usb0/images in the file gamelist.xml. The pattern of the command is this:

sed -i 's/old/new/g' filename

That means that any occurance of the string "old" in the file "filename" shall be replaced by the string "new". Mind that any special characters like slashes / in both strings have to be prefixed by a backslash \ to mark them as a parts of the strings and not part of sed's syntax. Thus, ./images becomes .\/images and /media/usb0/images becomes \/media\/usb0\/images in the above example.

As always, make a backup of your gamelist.xml before making any changes.

FlightRisk

Thanks for all the ideas. Gotta love anyone who has mastered sed, grep, etc. ;) The odd thing is that I DID set the scraper to "use rom folder". I assumed that would go to where I actually store my roms, which is an external USB stick plugged into the Pi 3. I did a "df" and don't see it listed, at least nothing that says USB. I use windows to copy the roms to the stick and then move it over to the Pi. Not sure if that has anything to do with anything or if it just isn't mounted. Retripie/ES finds all the roms.

herb_fargus

@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:

I assumed that would go to where I actually store my roms

It will if you have the USB mounted properly. If you just have it set to transfer it won't work that way.

https://retropie.org.uk/docs/Running-ROMs-from-a-USB-drive/

FlightRisk

I guess I need a refresher on Linux and mounting volumes, etc. When I first set this up, I ceated a folder on a windows pc called "retropie" onto the USB drive and stuck it in the Pi. It automatically copied my roms and a rom folder structure to the drive. When I do a "df", I thought I would see the USB drive, but I don't. I'm confused. It must be playing the games from it. Now I am looking at the "running roms from a USB drive" that you gave me and it says to enable a rom service and then create a folder called retropie-mount (not simply "retropie"). Do these steps do completely different things? I am afraid to add that folder without knowing what it will do, though I'll make a backup, of course, whatever I try.

mitu

@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:

Do these steps do completely different things?

Yes, the 1st will copy the ROMs from the usb to the sdcard (this is a ROM transfer method), the 2nd will auto-mount the USB and will run the ROMs from it.

Clyde

@mitu said in Can I Scrape to my USB stick with Selphs scraper?:

@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:
Yes, the 1st will copy the ROMs from the usb to the sdcard (this is a ROM transfer method), the 2nd will auto-mount the USB and will run the ROMs from it.

… after copying the RetroPie folder and all of its contents to the usb drive.

Just to be precise to avoid further confusion.

@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:

When I do a "df", I thought I would see the USB drive, but I don't.

You should if it's mounted in the directory tree. By the way, df -h gives you a "human readable" (-h) output that lists the size in K/M/Gbytes instead of filesystem blocks. It's a quick way to check the filling level of all mounted file systems.

cyperghost

@clyde @mitu
Even if it's not official supported ... Symbolic links will help out in this situation.
Old thread: can-my-usb-device-have-data-on-it-already

FlightRisk

@clyde @mito, would you recommend running everything off the USB? Seems like a win-win. Only con is something else sticking out of the Pi. Should be faster, more reliable, easy to move, backup, etc.

Also, will the second method (make a retropie-mount folder on the USB drive and have retropie automatically copy the retropie folder and leave the roms already on the stick) and then run everything off the USB keep what roms are on the USB? I will back up the USB first, but if I just add a retropie-mount with it in my PC, then move it to the Pie, will it do what it needs to do? Or would it erase the roms on the USB or even see that it is supposed to copy anything because it already saw the "retropie" folder and copied the roms and this is a one time thing?

Also, is the rom folder then still visible so that I can copy roms to and from the USB drive from my Windows PC instead of having to use a Samba link, SSH, etc.?

And last, the link @cyperghost gave mentioned issues if I move everything over because fat32 does not use linux permissons and I might have a lot of trouble configuring things to run scripts or whatever else I need for MAME and Dos games. I mostly care about the arcade games and less so with the consoles.

mitu

@flightrisk The usual reasons for running ROMs off the USB drive is reliability and portability. The USB is less likely to be corrupted than a SD card. Of course, if you're removing it, then you'll only see what you had previously copied on the SD card in the ROMs folder.
There is no '2nd method', the 1st method is for transferring ROMs via the USB drive onto the SD card, the only method to run off the USB card is this to use the retropie-mount folder on the USB driver and copy all the ROMs there. The 1st time after you insert the USB on the drive, the ROM folders will be created under the retropie-mount and you'll be able to copy ROMs to any of the sub-folders to make them appear on the RetroPie system.
As for the issue mentioned by @cyperghost, the problem is that symlinks are not supported by FAT32 partitions so you would not be able to symlink the artwork from the SD card folder to the ROM folder.

TL:DR - use the automated method mentioned in the docs (https://retropie.org.uk/docs/Running-ROMs-from-a-USB-drive/) and you'll be fine.

herb_fargus

@mitu said in Can I Scrape to my USB stick with Selphs scraper?:

As for the issue mentioned by @cyperghost, the problem is that symlinks are not supported by FAT32

There are other issues with this method too: ie it doesn't always keep the same name so your symlinks may break, it's just not a very elegant solution.

cyperghost

@herb_fargus @mitu
Indeed the method is not very elegant but it's possible that this work without rescrape if the image folder will be copied to USB and a symlink that points from sdcard to USB is created. The other way a symlink file on a FAT32 does not work. So this is just a fast workaround.

But we want to avoid diversity in setup for maintaining reasons!

stoo

The more I think about it, the more I don't understand why the gamelists and images aren't stored parallel to the roms by default. Same goes for all configs, now that I think about it.

cyperghost

@stoo Because you can use relatives pathes.
But I see no advantage in this.

/home/pi/RetroPie/roms/gb/Tetris.gb for a GameBoy game
can result to a imagepath ./Tetris.jpg

I would also like to see a database like SQLite
That would make things much easier ;)
So maybe every system got's it's own DB in a folder /home/pi/RetroPie/scraperdata/

Sano

@stoo Regarding images and gamelists, I suppose it's the default for ease of use and cleanliness. Same types of files are in the same directory, so you could do batch processing more quickly and safely, you can easily only backup roms and not scrapped data, ... It also allows to have default similar configurations for all setups, which is better for dev and support. It's quite logical IMHO.
Let's illustrate : You don't have a "Friends" folder containing pictures/videos of them, their VCF contact vcards and all your mail history with them, do you ? :)

For configuration files, it's an all-time unix best practice to store them aside from data. Again for backup and security reasons, data management (different partitions for data), and because an unix/linux systems are multi-users. A config file is nearly always system wide, but data is by nature specific to a user.

Clyde

@FlightRisk As most of you questions were answered by the others, I'll just add some information.

@flightrisk said in Can I Scrape to my USB stick with Selphs scraper?:

I will back up the USB first, but if I just add a retropie-mount with it in my PC, then move it to the Pie, will it do what it needs to do? Or would it erase the roms on the USB or even see that it is supposed to copy anything because it already saw the "retropie" folder and copied the roms and this is a one time thing?

As far as I know, it doesn't erase anything, but just copies the files. It may overwrite any existing files in the same subdirectory and with the same name, but that should seldom be a problem and that's what backups are for, after all.

Also, is the rom folder then still visible so that I can copy roms to and from the USB drive from my Windows PC instead of having to use a Samba link, SSH, etc.?

The directory structure in the directory RetroPie will be recreated on the USB drive, and the drive stays formatted with VFAT, so you should be able to access your roms on it from Windows. However, the roms on the SD card won't be accessible at all as long as the USB drive is connected to the Pi, because the USB drive's directory structure will be mounted over the respective directories on the SD card. To access the latter over the network or from within the Retropie system, you'll have to start the Pi without the USB drive connected to it (there are other ways, but this is the easiest one).

@cyperghost @mitu @herb_fargus Does the automatic method set the exec flag? Just curious, because the manual method only mentions it as an option in the docs.

stoo

@sano Yeah. It would just be nice if you could backup your home dir and know that all your scraped data and configs are covered.

Sano

@stoo Actually if you backup your whole home dir (/home/pi) it should be the case, provided your backup method doesn't ignore hidden files and follows symlinks.
You can also just add the /opt/retropie/configs/all/emulationstation path to your backup task.

FlightRisk

It is all clear to me and working now, so thanks to everyone. I think the Wiki should have a warning box that creating a USB drive with a "retropie" folder is very, very different that creating one with a "retropie-mount" folder. I made a retropie folder on the USB days ago and thought I was running RetroPie on the SD card and the roms from the USB. Instead, as was pointed out here, it was just copying the roms on the stick to the SDcard. Pull the stick after it loads them up and things still work. Creating a "retropie-mount" folder copies all of retropie, files, images, roms, etc. from the sdcard to the USB drive and runs it all from the USB. The USB overrides the sdcard version of retropie as long as it's plugged in. Take the USB out and the last version of what was on the SDcard works just fine. Remove retropie and all the roms from the sdcard, and then the USB drive must be inserted to play the games.

If something goes wrong after plugging in the USB drive during the conversion (like shutting down before the migration is finished) , you can just delete the retropie-mount folder and stick it back in the Pi. It will automatically start copying again. I hate there is no progress indicator anywhere. This service runs in the background. And while I always buy USB drives with led indicators, this time I didn't realize I was buying one without. What a pain. I watch the green light on the Pi and run a "df" on it every half hour or so (had over 1400 roms with scrapes) and watch to see the bytes used keep going up. When it stops changing for a while, I'll know it is finished (or locked up ;) ) So another big red box that should be in the Wiki is "This process runs in the background and you will have no indication of whether it is finished unless ... <fill out the rest>"