how do you get faster write speeds to SSD only getting 24mb/s

mitu

@comet424 said in how do you get faster write speeds to SSD only getting 24mb/s:

i formated to exfat but it didnt take
[...]

You've run the correct command, but your next df commands don't show that the newly formatted partition is mounted.

so i dont have that hdparm program not sure how to install it

You can install it with sudo apt install hdparm.

i did get it to format and mount using sudo mkfs.fat
extfat just wouldnt work now is this fat or fat32?

It's neither, exfat is another type of filesystem, just like fat or vfat (which is Linux name for FAT32).

I'm not really sure how you're testing, your df commands don't show the /dev/sda1 partition being mounted so you may actually be testing just the sdcard.

is it maybe retropie cant hand 2TB at the full usb 3.0 speeds? as i didnt have issue with the same brand SSD just at a 250 and 500mbs but now i jumped to a 2TB i get these extreme slow speeds.

It can handle it fine, that's not the issue here - is the filesystem used. Use exfat instead of ntfs and you should get better performance.

comet424

@mitu
ill re ty again and the test i did was on the bottom of the forum page

i ran

pi@retropie:~ $ sudo sh -c "sync && echo 3 > /proc/sys/vm/drop_caches"
pi@retropie:~ $ dd if=./largefile of=/dev/null bs=4k

and ran

dd if=/dev/zero of=./largefile bs=1M count=1024

so i installed it and ran

pi@retropie:~ $ sudo hdparm -Tt /dev/sda1

/dev/sda1:
 Timing cached reads:   1588 MB in  2.00 seconds = 793.80 MB/sec
 Timing buffered disk reads: 882 MB in  3.00 seconds = 293.62 MB/sec

so for doing the exfat how do i change that for the fstab? as it wouldnt mount or if it did it was media/usb0

pi@retropie:~ $ df -h -T
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      117G  4.5G  108G   4% /
devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs     1.9G  9.5M  1.9G   1% /run
tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1 vfat      256M   49M  207M  20% /boot
/dev/sda1      vfat      1.9T  4.2G  1.9T   1% /home/pi/RetroPie
tmpfs          tmpfs     384M     0  384M   0% /run/user/1000

this above formated for fat 32 and i getting copy speeds of 5-10mbs

but when i had ntfs it was least 25 so i dont get it ):

mitu

@comet424 said in how do you get faster write speeds to SSD only getting 24mb/s:

so for doing the exfat how do i change that for the fstab? as it wouldnt mount or if it did it was media/usb0

Did you try to use the fstab line that you previously posted ?

/dev/sda1  /home/pi/RetroPie       exfat   nofail,user,uid=pi,gid=pi 0     2

When you test using dd, make sure you're using an output file that's on the external disc:

dd if=/dev/urandom of=/home/pi/RetroPie/test.file bs=1M count=1024

comet424

@mitu

for the fstab i tried "exfat" "exfat4" "extfat" "ext4"
none of them mounted.. unless i change the fstab uuid
and set the usbmount to Enabled in the menu
then sometimes it mounted

so the results for the tests are

pi@retropie:~ $ sudo hdparm -Tt /dev/sda1

/dev/sda1:
 Timing cached reads:   1588 MB in  2.00 seconds = 793.80 MB/sec
 Timing buffered disk reads: 882 MB in  3.00 seconds = 293.62 MB/sec
pi@retropie:~ $ df -h -T
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      117G  4.5G  108G   4% /
devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs     1.9G  9.5M  1.9G   1% /run
tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1 vfat      256M   49M  207M  20% /boot
/dev/sda1      vfat      1.9T  4.2G  1.9T   1% /home/pi/RetroPie
tmpfs          tmpfs     384M     0  384M   0% /run/user/1000
pi@retropie:~ $ dd if=/dev/urandom of=/home/pi/RetroPie/test.file bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 59.2516 s, 18.1 MB/s

mitu

@comet424 said in how do you get faster write speeds to SSD only getting 24mb/s:

none of them mounted.. unless i change the fstab uuid

My example was without the UUID, just using the partition inode directly. If you re-format the drive, it's label may change, so the UUID may not match.

EDIT: add the entry to /etc/fstab, then try running

mount -a -t exfat

and if the drive is mounted (mount -t exfat).

comet424

@mitu

ok so not sure what to change in my fstab

proc            /proc           proc    defaults          0       0
PARTUUID=e22bcd10-01  /boot           vfat    defaults          0       2
PARTUUID=e22bcd10-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
UUID=3FDD-79EC  /home/pi/RetroPie       vfat    nofail,user,uid=pi,gid=pi 0     2

is my current Fat32 for the 2TB drive
everytime i format it changes the UUID so then i re change it in the fstab

and what i was meaning is /home/pi/RetroPie vfat
i changed the vfat to "ext4" "exfat" "exfat4"
when i had it formated as exFat or whatever the proper wording is for it

to see if it would mount.. it would sometimes mount as /media/usb0

so wehre in the fstab do i add /etc/fstab

and i guess reformat the drive to exfat right?

and i did try where vfat is when i had "ntfs" i tried changing it to "ntfs-3g" but that didnt help either as i read that in a different retropie article someone said worked but didnt work for me
when i had it as ntfs

mitu

Sorry, but it's getting difficult to follow your replies without any phrasing or punctuation. Can you make an effort and organize your replies better ? I understand English may not be your native language, but try to make an effort and put some structure into it.

First, make sure you format your drive as exfat, use the mkfs.exfat command I've mentioned. Use cfdisk /dev/sda or fdisk -l /dev/sda to confirm the drive is properly formatted.
2nd, modify /etc/fstab and replace the last line with the entry I mentioned a couple of replies ago:

/dev/sda1  /home/pi/RetroPie       exfat   nofail,user,uid=pi,gid=pi	0	2

3rd, try the mount commands from my previous reply to make sure that you've correctly modified /etc/fstab and mounting works. If none of the commands fail, then you can disable the usbmount service and reboot to see if the external disc is mounted correctly (to /home/pi/RetroPie).

comet424

@mitu
sorry try my best as i have dislexia and its sounds clear to me

what i was say
in fstab file
vfat line i changed it to "ext4" "exfat" "exfat4" for Exfat
non of these settings "mounted" the drive tp /home/pi/RetroPie
only to /media/usb0 and you had to have the Auto USBmount service Enabled in the Menu.. Disabled wouldnt mount

when it was ntfs i treid "ntfs" and "ntfs-3g"

and i was sayin since its in Fat32 i guess i have to reformat it

and i said i had to change the UUID in the Fstab every single time when i formated the drive

pi@retropie:/etc $ sudo mkfs.exfat /dev/sda1
mkexfatfs 1.3.0
Creating... done.
Flushing... done.
File system created successfully.
pi@retropie:/etc $ cfdisk /dev/sda
cfdisk: cannot open /dev/sda: Permission denied
pi@retropie:/etc $ cfdisk /dev/sda1
cfdisk: cannot open /dev/sda1: Permission denied
pi@retropie:/etc $ fdisk -l /dev/sda
fdisk: cannot open /dev/sda: Permission denied
pi@retropie:/etc $ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 15 Feb 28 16:34 8988cf8f-d459-476f-9a4a-16c56702dc40 -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 10 Feb 28 17:51 B862-4CA4 -> ../../sda1
lrwxrwxrwx 1 root root 15 Feb 28 16:34 C367-3F28 -> ../../mmcblk0p1
pi@retropie:/etc $ sudo nano fstab
pi@retropie:/etc $
pi@retropie:/etc $ fdisk -l /dev/sda1
fdisk: cannot open /dev/sda1: Permission denied

i rebooted and ran your 2 commands again

     `"""'     The RetroPie Project, https://retropie.org.uk

pi@retropie:~ $ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 15 Feb 28 17:58 8988cf8f-d459-476f-9a4a-16c56702dc40 -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 10 Feb 28 17:58 B862-4CA4 -> ../../sda1
lrwxrwxrwx 1 root root 15 Feb 28 17:58 C367-3F28 -> ../../mmcblk0p1
pi@retropie:~ $ df -h -T
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      117G  4.5G  108G   4% /
devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs     1.9G  9.4M  1.9G   1% /run
tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1 vfat      256M   49M  207M  20% /boot
tmpfs          tmpfs     384M     0  384M   0% /run/user/1000
pi@retropie:~ $ cfdisk /dev/sda1
cfdisk: cannot open /dev/sda1: Permission denied
pi@retropie:~ $ fdisk -l /dev/sda
fdisk: cannot open /dev/sda: Permission denied
pi@retropie:~ $ fdisk -l /dev/sda1
fdisk: cannot open /dev/sda1: Permission denied
pi@retropie:~ $

proc            /proc           proc    defaults          0       0
PARTUUID=e22bcd10-01  /boot           vfat    defaults          0       2
PARTUUID=e22bcd10-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
UUID=B862-4CA4  /home/pi/RetroPie       exfat   nofail,user,uid=pi,gid=pi 0     2

pi@retropie:/etc $ mount -a -t exfat
mount: only root can use "--all" option
pi@retropie:/etc $ sudo mount -a -t exfat
FUSE exfat 1.3.0
WARN: volume was not unmounted cleanly.
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
pi@retropie:/etc $

mitu

The errors below

cfdisk: cannot open /dev/sda1: Permission denied
pi@retropie:~ $ fdisk -l /dev/sda
fdisk: cannot open /dev/sda: Permission denied
pi@retropie:~ $ fdisk -l /dev/sda1
fdisk: cannot open /dev/sda1: Permission denied

are because I forgot to prefix them with sudo (they need root access). Try

sudo fdisk -l /dev/sda

and check if the /dev/sda1 partition is reported as NTFS/exFAT.

The same with the mount commands.

sudo mount -a -t exfat
FUSE exfat 1.3.0
WARN: volume was not unmounted cleanly.
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option

The first message is because the fuse variant is installed, though it shouldn't interfere. You can uninstall it first by running

sudo apt remove exfat-fuse

and the mounting will use the native Linux exfat support. Re-try then to run

> sudo mount -a -t exfat -o nonempty

and then check by running mount that /home/pi/RetroPie is mounted using exfat.

comet424

@mitu

pi@retropie:~ $ sudo fdisk -l /dev/sda1
Disk /dev/sda1: 1.8 TiB, 2000397795328 bytes, 3907026944 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
pi@retropie:~ $ sudo apt remove exfat-fuse
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libmicrodns0 rpi-eeprom-images
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  exfat-fuse
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 64.5 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 130482 files and directories currently installed.)
Removing exfat-fuse (1.3.0-1) ...
Processing triggers for man-db (2.8.5-2) ...
pi@retropie:~ $ sudo mount -a -t exfat -o nonempty
pi@retropie:~ $ df -h -T
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      117G  4.5G  108G   4% /
devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs     1.9G   26M  1.9G   2% /run
tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1 vfat      256M   49M  207M  20% /boot
tmpfs          tmpfs     384M     0  384M   0% /run/user/1000
/dev/sda1      exfat     1.9T  2.2M  1.9T   1% /home/pi/RetroPie
pi@retropie:~ $ sudo hdparm -Tt /dev/sda1

/dev/sda1:
 Timing cached reads:   1608 MB in  2.00 seconds = 804.45 MB/sec
 Timing buffered disk reads: 866 MB in  3.00 seconds = 288.37 MB/sec
pi@retropie:~ $

pi@retropie:~ $ dd if=/dev/urandom of=/home/pi/RetroPie/test.file bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 58.2039 s, 18.4 MB/s
pi@retropie:~ $

copying over network from windows still get 2-10mb/s

mitu

OK, so the disc is not formated as exfat and mounted via the native kernel support, without fuse. This should take out the extra CPU consumed by using fuse and ntfs.

If you're copying just a large file over the network, do you get the same - low - speed ? Is this over a wired or wireless conenction ?

I know the NesPi case - the older versions ? - had an issue with the SATA controller used in the case (see this topic), it might be worth checking if the case you're using falls has the SATA controller that is problematic.

Just

comet424

@mitu
i thought the disk got formated as exfat

/dev/sda1      exfat     1.9T  2.2M  1.9T   1% /home/pi/RetroPie

so what do you mean exactly by "OK, so the disc is not formated as exfat and mounted via the native kernel support, without fuse. This should take out the extra CPU consumed by using fuse and ntfs."

explain it for someone that doesnt know the fuse and the native kernel support or the extra cpu consummed etc

network is cabled 1gb connection..
its copying any file.. that i getting only 2-10mbs the dd command shows 18mb/s is that saying its only read writting? large or small just doesnt matter ):

im using the nespi 4 from my how to guide i made
https://retropie.org.uk/forum/topic/28848/guide-how-to-setup-nespi-4?_=1646095106362

so i dont know if it falls under that i thought i solved the slow problem speeds with my article about mounting the sata controller as i think i read that article and thats how i made my article from it

mitu

@comet424 said in how do you get faster write speeds to SSD only getting 24mb/s:

explain it for someone that doesnt know the fuse and the native kernel support or the extra cpu consummed etc

https://en.wikipedia.org/wiki/NTFS-3G#Performance

so i dont know if it falls under that i thought i solved the slow problem speeds with my article about mounting the sata controller as i think i read that article and thats how i made my article from it

Did you add the quirks configuration to /boot/cmdline.txt ? Can you try without it ?

comet424

@mitu
yes and no

i actually still running the same nintendo nesp case you see in the article i made.. all i did for myself was upgrade from 250gb to a 2tb and i use the 250gb in a new system for a friend.. and speeds were fine for it..

so i didnt need to change the cmdline as i made thes changes back in 2020 .. but ill try removing the part in the cmdline to see if it works for a 2tb i had to add it for the 250 and 500gb

comet424

 `*****@@**'  IP Address.........: 192.168.0.52
  `*******'   Temperature........: CPU: 52°C/125°F GPU: 52°C/125°F
    `"""'     The RetroPie Project, https://retropie.org.uk

pi@retropie:~ $ df -h -T
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      117G  4.5G  108G   4% /
devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs     1.9G  9.4M  1.9G   1% /run
tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1 vfat      256M   49M  207M  20% /boot
/dev/sda1      exfat     1.9T   18G  1.9T   1% /home/pi/RetroPie
tmpfs          tmpfs     384M     0  384M   0% /run/user/1000
pi@retropie:~ $ sudo hdparm -Tt /dev/sda1

/dev/sda1:
Timing cached reads:   1546 MB in  2.00 seconds = 772.88 MB/sec
Timing buffered disk reads: 1088 MB in  3.00 seconds = 362.55 MB/sec
pi@retropie:~ $ dd if=/dev/urandom of=/home/pi/RetroPie/test.file bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 58.393 s, 18.4 MB/s
pi@retropie:~ $

so windows shows now anywhere from 2-50mbs its better that way but i dunno... is there a test file size that you copy from windows to find out the actually speed?

i gonna try to copy over some couple gig files

so copying just random files i get 3-5mbs
random files of 159gb i jsut sliding over is going to take 8hours

so removing the cmdline info i added didnt help ):

but its a limit with this usb hot swap and 2TB maybe its only good for 1TB and less?

i dont have a smaller SSD anymore to test it ):

comet424

so selecting 4 files all together 8 gb
it started at 90mbs then dropped down to 2mb then up and down but finished

thats with the removal of the extra for the the cmdline.txt

but the other file copying of smaller more files is at 2.38mb/s saying going to take 15 hours now to copy 156gb

i selected 73 files that equaled 159gb to test to copy.... it peaks at 90mbs/ averages like 60mb and says takes 35min to transfer 159gb not the 15 hours for the smaller files ...

should i re add the info to the cmdline.txt and see what happens ..

is it an issue then for small files? and if so anyway to improve speeds for small files

comet424

had to re add

usb-storage.quirks=152d:0562:u

to the cmdline.txt when you did a reboot was ok.

but when you turned it off.. and turned back on... the usb drive wouldnt not mount

so i re added

usb-storage.quirks=152d:0562:u

then rebooted then the drive mounted..

it can copy big files like 70 meg/s but small files etc slow and takes forever i dont get it.. but if i copy to my windows to unraid server is faster for small files is it possibly not fast cpu?

mitu

Multiple small files will always be slower to copy than larger files. The CPU should be plenty fast, but I/O (disc/network/internal PCI bus) is not the same as on PC, plus the quirks added which disable UAS in favor of the slower USB mas-storage driver have also an impact.

comet424

@mitu

oh so the quirks whatever that is slows the usb port down? or the transfering...

so i wonder if newer nespi4 have it fixed the usb mass storage

does it impact using retropie then loading roms or what not.. or just mainly over network..

and do you guys run the retropie OS on the SD card or do you run it on the SSD

mitu

@comet424 said in how do you get faster write speeds to SSD only getting 24mb/s:

@mitu

oh so the quirks whatever that is slows the usb port down? or the transfering...

Yes, using UAS would allow for faster disc operations, this is noted in the troubleshooting topic on the RPI forums:

UAS is an upgraded transport protocol compared to USB mass-storage - commands and data are separated into different queues and multiple outstanding commands can be in flight at the same time, as opposed to USB mass-storage's lock-step relationship between commands and data. This allows better saturation of the 4GBps USB3.0 transport as there can be a continuous stream of data to and from a device

so i wonder if newer nespi4 have it fixed the usb mass storage

I don't know and their web site doesn't have an easy to find changelog, but you can send them an email and ask about it.

does it impact using retropie then loading roms or what not.. or just mainly over network..

I don't think so, even with the USB storage quirks the disc should be faster than a regular SDcard.