PlayStation and .zip files

IyonUK

@darksavior Loving the Super Famicom BTW. 👍🏻

IyonUK

Hi

I have got a script working to my liking. I'm posting it here in case anyone else wants to use/butcher/edit it.

It creates a .cache directory in /home/pi/RetroPie/roms/psx into which it unzips the games. It keeps the files there until it hits the cache limit (which is set at 5Gb) and then starts deleting older files to recover the space.

There are probably bugs/improvements to be found/made.

As mentioned by @mediamogul I put the following script in /opt/retropie/configs/psx named as lr-pcsx-rearmed-unzip.sh

#!/bin/bash

function centre_string() {
  local LPAD=$(( ( $(tput cols) - ${#1} ) / 2))
  local RPAD=$(( ( $(tput cols) - ${#1} ) - $LPAD ))
  local STRING=$(printf ' %.0s' $(seq 1 $LPAD))${1}$(printf ' %.0s' $(seq 1 $RPAD))
  echo "$STRING"
}

function ticker() {
  STR=""
  B=$(( $P - 1 ))
  while [[ $B -gt 0 ]]; do
    STR="${STR}."
    B=$(( $B - 1 ))
  done
  STR="${STR}:"
  B=$(( $W - $P ))
  while [[ $B -gt 0 ]]; do
    STR="${STR}."
    B=$(( $B - 1 ))
  done
  tput cup $SL3Y 0
  echo -e "${TC_YELLOW}$( centre_string "$STR" )"
  Z=$(( $Z + 1 ))
  P=$(( $P + $D ))
  if [[ $P -gt $W ]]; then
    P=$(( $W - 1 ))
    D=-1
  fi
  if [[ $P -lt 1 ]]; then
    P=2
    D=1
  fi
}

# Clear display and hide cursor
tput clear
tput civis

# Declare terminal colours
TC_CYAN="\e[0;36;49m"
TC_DEFAULT="\e[0;39;49m"
TC_YELLOW="\e[0;33;49m"

# Declare cache variables
CACHE_DIRECTORY="/home/pi/RetroPie/roms/psx/.cache"
CACHE_IDEAL_SIZE=5242880 # 5242880 = 5Gb

# Declare path variables
ZIP_FILE="$(sed '3q;d' /dev/shm/runcommand.info)"
ZIP_FILE_DIRECTORY=$(dirname "$ZIP_FILE")
CUE_FILE="${ZIP_FILE##*/}"
CUE_FILE="${CACHE_DIRECTORY}/${CUE_FILE%.*}.cue"

# Declare status line variables
SL1Y=$(( ($(tput lines) / 2) - 1 ))
SL2Y=$(( $SL1Y + 1 ))
SL3Y=$(( $SL2Y + 1 ))

# Create cache directory (if not exists)
mkdir -p "${CACHE_DIRECTORY}"

# Status update
tput cup $SL2Y 0
STR="Checking cache ..."
echo -e "${TC_CYAN}$( centre_string "$STR" )"
sleep 0.2

# Get current cache size
CACHE_SIZE=$(find "$CACHE_DIRECTORY" -type f \( -iname \*.bin -o -iname \*.cue \) -print0 | du --files0-from=- -c | tail -n1 | cut -f1)

# Does the cache need clearing?
if [[ $CACHE_SIZE -gt $CACHE_IDEAL_SIZE ]]; then

  # Status update
  tput cup $SL1Y 0
  STR="Shrinking cache ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Keep removing files, oldest first, until cache is below maxiumum size
  while [[ $CACHE_SIZE -gt $CACHE_IDEAL_SIZE ]]; do

    # Get oldest file
    FILE_TO_DELETE=$(ls -A1rt "$CACHE_DIRECTORY"/*.bin "$CACHE_DIRECTORY"/*.cue 2>/dev/null | head -n1)

    # Status update
    tput cup $SL2Y 0
    STR=$(basename "$FILE_TO_DELETE")
    echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

    # Delete file
    rm -rf "$FILE_TO_DELETE"

    # Get current cache size
    CACHE_SIZE=$(find "$CACHE_DIRECTORY" -type f \( -iname \*.bin -o -iname \*.cue \) -print0 | du --files0-from=- -c | tail -n1 | cut -f1)

  done

  # Clear
  tput cup $SL1Y 0
  echo -e "$( centre_string "" )"

fi

# Spinner variables
P=1;D=1;W=7

# Declare first regular expression - Redump based
REGEX1="^(.*\(Disc )[0-9]+(\).*)$"

# Single disc or multiple discs?
if [[ ! $ZIP_FILE =~ $REGEX1 ]]; then

  # Output file details
  tput cup $SL1Y 0
  STR="Unzipping disc ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Output file name
  tput cup $SL2Y 0
  STR=$(basename "$ZIP_FILE")
  echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

  # Unzip
  unzip -DDnq "$ZIP_FILE" -d "$CACHE_DIRECTORY" &
  UNZIP_PID=$!

  # Ticker
  while kill -0 $UNZIP_PID 2> /dev/null; do
    ticker
    sleep 0.05
  done

else

  # Status update
  tput cup $SL2Y 0
  STR="Checking for other discs ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Build second regular expression
  REGEX2=${BASH_REMATCH[1]}[0-9]+${BASH_REMATCH[2]}
  REGEX2=$(echo "$REGEX2" | sed -e 's/(/\\(/g' -e 's/)/\\)/g' -e 's/\./\\./g')

  # Unzip discs  
  CURRENT_FILE=1
  find "$ZIP_FILE_DIRECTORY" -maxdepth 1 -regextype posix-extended -regex "$REGEX2" | sort -t '\0' -n | while read LINE; do

    # Output file details
    tput cup $SL1Y 0
    STR="Unzipping disc ${CURRENT_FILE} ..."
    echo -e "${TC_CYAN}$( centre_string "$STR" )"

    # Output file name
    tput cup $SL2Y 0
    STR=$(basename "$LINE")
    echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

    # Unzip
    unzip -DDnq "$LINE" -d "$CACHE_DIRECTORY" &
    UNZIP_PID=$!
  
    # Ticker
    while kill -0 $UNZIP_PID 2> /dev/null; do
      ticker
      sleep 0.05
    done

    # Next file
    CURRENT_FILE=$(( $CURRENT_FILE + 1 ))

  done

fi

# Clear display and restore cursor
tput sgr0
tput clear
tput cnorm

# Launch lr-pcsx-rearmed
/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-pcsx-rearmed/libretro.so --config /opt/retropie/configs/psx/retroarch.cfg "$CUE_FILE" --appendconfig /dev/shm/retroarch.cfg &> /dev/shm/runcommand.log

# Exit
exit

I added the following to /opt/retropie/configs/psx/emulators.cfg and set it to the default emulator

lr-pcsx-rearmed-unzip = "bash /opt/retropie/configs/psx/lr-pcsx-rearmed-unzip.sh %ROM% &>/dev/tty"

NB. The cache directory is on an external HDD so I'm not worried about SD card failure.

Edit: Updated script to not use regex for single disc games.

Clyde

Just a comment to @IyonUK's script: If your machine has enough ram, you might change the CACHE_DIRECTORY to something in the ramdisk /dev/shm (shared memory), e.g. /dev/shm/psx-cache. Most of today's Linux distributions install shm by default. (Does Retropie even work without it, since it uses it for Runcommand?)

Be sure to change CACHE_IDEAL_SIZE accordingly. You can check the size and free amount of shm with the command df -h. But leave some of it for other applications and the system.

As it is with ramdisks, everything stored there is lost on a system shutdown or reboot. So, contrary to on-disk caching, the zip extraction will have to happen every first time a game is started after a system shutdown or restart. Writing to ram is much faster than to disk, though.

el chupacabra

@mediamogul Was that the only way to do it last year? It seems a pretty crazy workaround.

Also pbp files were mentioned but you shouldn't use pbp files they cause headaches.
@IyonUK You can use chd files with the main psx emulator which is far better and easier.
You can get this on the device itself running. chd files also will embed multiple files into the single file which is great for multiple games.

apt install mame-tools

now the program 'chdman' is installed it is very simple to run it:
chdman createcd -i FILENAME.cue -o FILENAME.chd


chdman createcd -i Castlevania\ Symphony\ of\ the\ night.cue -o Castlevania\ Symphony\ of\ the\ night.chd

The output looks like this:

This will put all your files in a single file, like any archive does. Twisted metal 2 for example goes from 12 files and 800mb to 460mb.

cyperghost

@el-chupacabra No, pbp files are not a crazy workaround it the standard format for Playstation portable games. But I like your small tutorial - you should add it to the PSX section.

The best format (imho) is ecm, no (logical) data loss - it removes the error correction from the physical data device. Are there any emulators supporting this format?

el chupacabra

@cyperghost said in PlayStation and .zip files:

@el-chupacabra No, pbp files are not a crazy workaround it the standard format for Playstation portable games. But I like your small tutorial - you should add it to the PSX section.

The best format (imho) is ecm, no (logical) data loss - it removes the error correction from the physical data device. Are there any emulators supporting this format?

I didn't say pbp was a crazy work around.
https://retropie.org.uk/docs/Playstation-1/ Looks like chd is already mentioned briefly here

cyperghost

@el-chupacabra Oh my bad! But then use the cite-function to avoid me from confusion.

But this does not matter at all - I think chd is the better way to store data (lossless with LZMA for data and FLAC for audio) and can be handeled like original iso/bin with m3u/cue.

Again do you know an emulator working with ecm format?

el chupacabra

@cyperghost
By stating "pbp files were mentioned" I am clearly referring to all of the posts talking about it. I have just seen many issues with it over the years causing game issues. But pbp is nicer for multi-disk games because you don't need a m3u file. That is pbp's only benefit.
https://www.reddit.com/r/emulation/comments/714n7g/pbp_vs_chd_for_mednafenbeetle_psx_opinions/
At the time of their posting this was very new and not even available on retropie as far as I know, though.

No I don't know any that use ecm or plan to, probably because ecm is less efficient than chd and offers no other benefits (unlike pbp offers for multi disc games). Therefore I don't think anyone is going to bother to support it. Maybe on the "to do but not important" list. It uses RAR compression by default 15-20% compression vs chd's 35-45%
Also chd does not use a .cue file

https://en.wikipedia.org/wiki/MAME
https://www.mankier.com/1/chdman

MAME is such a massive project, whatever it creates pretty much becomes standard.

cyperghost

@el-chupacabra Thanks for this input! Did not know that ecm is so inefficient. It is used to spread images around but maybe this is outdated.

mediamogul

@el-chupacabra said in PlayStation and .zip files:

Was that the only way to do it last year? It seems a pretty crazy workaround.

It was and still is completely crazy. Like I said above, it was just an interesting exercise. At the time, .pbpwas the only real way to go, but since then the option to use .chd has come about.

OldSchool

@mediamogul So is CHD considered the recommended option now?

mediamogul

@OldSchool

IMO it depends on personal use habits. .chd will likely be the easiest and most widely supported compressed format over time, but if you were someone who also owns a hacked PSP, it'd probably make the most sense to archive .pbp to prevent backing up two copies of each title. Myself, I only use bin/cue, as I have many different emulation setups and that format is the most widely supported across the various platforms and emulation options. In the long run it saves me time and space not to compress at all.

OldSchool

@mediamogul I would probably prefer .bin .cue as well because it's less work, and I use a large SSD anyways, but I don't like how each disk shows up on the emulationstation game list. I suppose the best option is to just keep the original disk images on backup media to always have them, and go ahead with the .chd conversion for space and cleanliness, at least for me. If I could get a multi-disk game to only show up once on the menu then I would certainly go with that though.

mitu

@OldSchool ES (and RetroPie) has long disabled the .bin files showing in the gamelist - https://retropie.org.uk/docs/Playstation-1/#roms. You either have a custom setup or a very old installation.

OldSchool

@mitu
Of course, I just mean that if you have a multi-disk game than you would see something like "final fantasy VII disk 1" "final fantasy VII disk 2" and so on, which I find annoying. It's not the worst thing but it does look nice when you have a single entry in the game list for a game.

steptoe

PBP or CHD are both compressed disc format so should save maximum space. I think CHD is slightly better compression but not by much, with ZIP you will have to uncompress first before being able to run the game

PBP has the added advantage of being able to join multi-disc games into one file. CHD is literally Compressed Hard Disc so each disc is compressed

Have a look at these links for a bit more advice and the last one explains exactly how to create a single PBP from multiple discs

https://retropie.org.uk/forum/topic/8016/psx-multi-disk-combining-bin-files-into-pbp-but-what-about-cue/12
https://www.reddit.com/r/emulation/comments/714n7g/pbp_vs_chd_for_mednafenbeetle_psx_opinions/

I've just checked the download link for PSXtoPSP and its still active. It doesn't matter what emulator or OS you use as long as they can load PBP it should still work

https://www.reddit.com/r/miniSNESmods/comments/7nx6z3/guide_how_to_convert_psx_multiple_discs_game_in

I've found another guide that's a clearer but still using PSXtoPSP and with plenty of pictures to show exactly what you need to do on each step and how a single PBP from multiple discs. The link for PSXtoPSP is still active in this guide as well

https://gbatemp.net/threads/guide-to-converting-ps1-games-for-psp.206195/

el chupacabra

@OldSchool You can just use a m3u file to display a game ONCE no matter how many disks you have. I just use chd for the compression and m3u for multiple disks
tutorial is on the wiki, it's very easy. You just name your file:
final fantasy (disk1).chd
final fantasy (disk2).chd
CHANGE TO:
final fantasy.CD1
final fantasy.CD2
and put that into the:
final fantasy.m3u
file.. that will only show ONCE. again it is on the psx wiki in the docs (of retropie)
I like chd since I can use the retropie install and chdman to make them on the sdcard just fine. I don't need another machine to do it.

IyonUK

Going back on this, I know it’s old...

I like to keep my PSX ROMs aligned to the Redump DATs which means .BIN and .CUE files. Also, I’ve found that converting them to .PBP has thrown errors in the past. Not looked into .CHD files but with both these methods of converting ROMs it can be a pain if you have a large collection.

Since this was posted I still use an unzip script but I use .M3U files for multi-disc games (which seem to use a single .BIN and .CUE in Redump terms) and .CUE for single disc games (which Redump seems to either use single .BIN or multiple .BINs).

This method has both storage benefits and save game benefits as the save files are aligned with the .M3U for multi-disc games.

PS. Last time I checked the entire PSX Redump collection was 3.7TB so there are benefits to unzipping into a cache if storage is tight and you’re not doing so to an SD card.