PlayStation and .zip files

BobHarris

@thedatacereal There is no other advantage than space saving.

mediamogul

@TheDataCereal

The main benefit is compression, but a lot of people seem to like the single file solution for multi-disc games that it offers as well. Myself, I just use bin/cue files and switch them out through the UI. This is mostly to maintain compatibility with all my different emulation setups, as bin/cue is best supported all around.

@IyonUK

Hi, I’m interested in your idea. I like exercises.

Keen beans then. First, you'd want to zip your .bin files, but not the .cue files. You should now have two files for every game, such as gamename.cue and gamename.zip for example. Now, create a script file named lr-pcsx-rearmed-unzip.sh at /opt/retropie/configs/psx/, make it executable and then add the following to it:

#!/bin/bash

##set path to .cue file
CUEPATH="$(sed '3q;d' /dev/shm/runcommand.info)"

##set path to .zip file based on .cue path
ZIP=$(echo "$CUEPATH" | cut -f 1 -d '.' | sed 's/$/.zip/')

##set path to .zip directory based on .zip path
ZDIR=$(dirname "$ZIP")

##unzip .bin file
unzip "$ZIP" "$ZDIR"

##wait for unzip to execute, then wait for it to finish
until [ -n "$UZPID" ]; do
  UZPID="$(pgrep -f unzip)"
  sleep 1
done
while [ -n "$UZPID" ]; do
  UZPID="$(pgrep -f unzip)"
  sleep 1
done

##launch emulator with .cue file, which should now be able to access the unzipped .bin
/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-pcsx-rearmed/libretro.so --config /opt/retropie/configs/psx/retroarch.cfg "$CUEPATH"

Next, add the following to /opt/retropie/configs/all/runcommand-onend.sh:

##if unzip was used, set path to .zip file based on .cue path and delete .bin file
if [ "$2" = "lr-pcsx-rearmed-unzip" ]; then
  BIN=$(echo "$3" | cut -f 1 -d '.' | sed 's/$/.bin/')
  rm >> /dev/shm/runcommand.log 2>&1 "$BIN"
fi

Finally, add the following line to /opt/retropie/configs/psx/emulators.cfg

lr-pcsx-rearmed-unzip = "bash /opt/retropie/configs/psx/lr-pcsx-rearmed-unzip.sh %ROM%"

While you're there, go ahead and set default="lr-pcsx-rearmed-unzip"

Now, in theory, everytime a .cue file is launched, the corresponding .bin file will be unzipped and Retroarch will be launched afterwards, where it will use the .cue to find and run the freshly unzipped .bin. Once exited, the .bin file will be deleted. Again, it should be said that this should work in theory, as I haven't actually tested it. If nothing else, it should give some insight as to how all these pieces can potentially fit together.

IyonUK

@mediamogul Brilliant, I can see how it would work. Launching an script instead of the emulator. I think with that solution you could go full hog and keep the bin and cue in a zip, allow zip as an extension for the emulator then the script could unzip both and launch the cue. I’ve thought of a slight snag which involves multi-disc games as the remaining discs would still be zipped. But I think some cunning regular expressions could unzip those too. I’ll get cracking later. 😊

mediamogul

@iyonuk said in PlayStation and .zip files:

I’ll get cracking later.

Let us know how it turns out.

mitu

@iyonuk Just be aware that every time you'll launch the game, the game will be unzipped. You'll have to wait for the unzip command to finish its job before the game will actually be launched.
If you're doing this on a Raspberry PI, you can wait quite a few minutes and maybe more.

IyonUK

@mitu Hmm, I hadn’t really thought of the unzipping performance. I’ll have to try a few unzips out.

mediamogul

@mitu said in PlayStation and .zip files:

If you're doing this on a Raspberry PI, you can wait quite a few minutes and maybe more.

No problem. Just script another game to launch while you're waiting. ;)

mitu

@mediamogul said in PlayStation and .zip files:

Just script another game to launch while you're waiting. ;)

Just quickly launch a game while you're waiting for the game to start :)

IyonUK

Well... no need for any of this. Added zip to the supported list in es_systems.cfg and tried a game for the sake of it and it works. RetroArch unpacks it to /tmp/retroarch. It takes a while to start, blank screen whilst it unpacks but that solves my storage issue. 😃

IyonUK

Probably spoke too soon. Need to try some with multiple bin files.

IyonUK

Bit of a mixed bag at the moment. Passing a zip file in does indeed extract the zip to /tmp/retroarch but only the first bin file. I’ve adapted your script to work with a zip file and extract everything to /tmp/psx (cue and all bins). This works fine and the game loads, however RetroArch is now ignoring the display settings I had setup for PCSX ReArmed and is launching in 1080p with only the top-left quarter of the screen displaying the full image (I had display for PCSX ReArmed set to CEA1 with custom offsets/width/height to fill my screen). I noticed that an —appendconfig is used when launching PCSX ReArmed normally but this is only to set the refresh rate to 60hz. The other issue is minor in that running from /tmp/psx also saves the srm file there. So I might have to switch back your script to extract to the rom location. Any ideas why my display settings might be borked?

mediamogul

@iyonuk said in PlayStation and .zip files:

Any ideas why my display settings might be borked?

If these are RetroArch settings, then they should have carried over. If the display settings you're referencing were set in the Runcommand menu and if you've changed the launch command in /opt/retropie/configs/psx/emulators.cfg, then your previous settings are tied to the 'lr-pcsx-rearmed' named launch command from that file. You would need to either configure those same settings for 'lr-pcsx-rearmed-zip', or manually change 'lr-pcsx-rearmed' to 'lr-pcsx-rearmed-zip' in /opt/retropie/configs/all/videomodes.cfg.

IyonUK

I’ve set the runcommand resolution for the new unzip emulator/script to CEA-1 to match which works fine but RetroArch still doesn’t play ball. I had set the aspect ratio to “Custom” and manually entered figures for the X and Y offsets, width and height but those have gone. I have set them back but every time I launch those setting have reset. I checked the config at /opt/retropie/configs/psx/retroarch.cfg and my settings are there (same location as where RetroArch says its saved the settings and the same config passed on emulator launch) so I’m a little lost right now. Will do some more digging later. On a positive note the unzipping etc. works fine, will probably tart it up a bit.

dankcushions

@iyonuk unzipping ~400MB files to an SD card every game launch sounds like an efficient way of destroying an SD card ;)

i guess you could unzip to a ram drive, but multi-disk psx games would consume more system ram than an rpi has.

i would definitely recommend the .pbp approach.

Darksavior

Nothing pure about preferring one disc format over another. Pbp saves space, and your "purist" butt won't know the difference.

IyonUK

@dankcushions said in PlayStation and .zip files:

@iyonuk unzipping ~400MB files to an SD card every game launch sounds like an efficient way of destroying an SD card ;)

I’m currently unzipping to /tmp/psx so on the SD card but I’ll most likely move it to the rom location which is mounted on an external HDD. Although it’d be cheaper to replace the SD card. 😉

I could also consider leaving the unzipped files in place until a certain limit is reached and then delete the oldest files, like a cache.

IyonUK

@darksavior said in PlayStation and .zip files:

Nothing pure about preferring one disc format over another. Pbp saves space, and your "purist" butt won't know the difference.

Probably more to do with effort than purism. Also the PSX2PBP thing failed a few times for me.

IyonUK

@darksavior Loving the Super Famicom BTW. 👍🏻

IyonUK

Hi

I have got a script working to my liking. I'm posting it here in case anyone else wants to use/butcher/edit it.

It creates a .cache directory in /home/pi/RetroPie/roms/psx into which it unzips the games. It keeps the files there until it hits the cache limit (which is set at 5Gb) and then starts deleting older files to recover the space.

There are probably bugs/improvements to be found/made.

As mentioned by @mediamogul I put the following script in /opt/retropie/configs/psx named as lr-pcsx-rearmed-unzip.sh

#!/bin/bash

function centre_string() {
  local LPAD=$(( ( $(tput cols) - ${#1} ) / 2))
  local RPAD=$(( ( $(tput cols) - ${#1} ) - $LPAD ))
  local STRING=$(printf ' %.0s' $(seq 1 $LPAD))${1}$(printf ' %.0s' $(seq 1 $RPAD))
  echo "$STRING"
}

function ticker() {
  STR=""
  B=$(( $P - 1 ))
  while [[ $B -gt 0 ]]; do
    STR="${STR}."
    B=$(( $B - 1 ))
  done
  STR="${STR}:"
  B=$(( $W - $P ))
  while [[ $B -gt 0 ]]; do
    STR="${STR}."
    B=$(( $B - 1 ))
  done
  tput cup $SL3Y 0
  echo -e "${TC_YELLOW}$( centre_string "$STR" )"
  Z=$(( $Z + 1 ))
  P=$(( $P + $D ))
  if [[ $P -gt $W ]]; then
    P=$(( $W - 1 ))
    D=-1
  fi
  if [[ $P -lt 1 ]]; then
    P=2
    D=1
  fi
}

# Clear display and hide cursor
tput clear
tput civis

# Declare terminal colours
TC_CYAN="\e[0;36;49m"
TC_DEFAULT="\e[0;39;49m"
TC_YELLOW="\e[0;33;49m"

# Declare cache variables
CACHE_DIRECTORY="/home/pi/RetroPie/roms/psx/.cache"
CACHE_IDEAL_SIZE=5242880 # 5242880 = 5Gb

# Declare path variables
ZIP_FILE="$(sed '3q;d' /dev/shm/runcommand.info)"
ZIP_FILE_DIRECTORY=$(dirname "$ZIP_FILE")
CUE_FILE="${ZIP_FILE##*/}"
CUE_FILE="${CACHE_DIRECTORY}/${CUE_FILE%.*}.cue"

# Declare status line variables
SL1Y=$(( ($(tput lines) / 2) - 1 ))
SL2Y=$(( $SL1Y + 1 ))
SL3Y=$(( $SL2Y + 1 ))

# Create cache directory (if not exists)
mkdir -p "${CACHE_DIRECTORY}"

# Status update
tput cup $SL2Y 0
STR="Checking cache ..."
echo -e "${TC_CYAN}$( centre_string "$STR" )"
sleep 0.2

# Get current cache size
CACHE_SIZE=$(find "$CACHE_DIRECTORY" -type f \( -iname \*.bin -o -iname \*.cue \) -print0 | du --files0-from=- -c | tail -n1 | cut -f1)

# Does the cache need clearing?
if [[ $CACHE_SIZE -gt $CACHE_IDEAL_SIZE ]]; then

  # Status update
  tput cup $SL1Y 0
  STR="Shrinking cache ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Keep removing files, oldest first, until cache is below maxiumum size
  while [[ $CACHE_SIZE -gt $CACHE_IDEAL_SIZE ]]; do

    # Get oldest file
    FILE_TO_DELETE=$(ls -A1rt "$CACHE_DIRECTORY"/*.bin "$CACHE_DIRECTORY"/*.cue 2>/dev/null | head -n1)

    # Status update
    tput cup $SL2Y 0
    STR=$(basename "$FILE_TO_DELETE")
    echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

    # Delete file
    rm -rf "$FILE_TO_DELETE"

    # Get current cache size
    CACHE_SIZE=$(find "$CACHE_DIRECTORY" -type f \( -iname \*.bin -o -iname \*.cue \) -print0 | du --files0-from=- -c | tail -n1 | cut -f1)

  done

  # Clear
  tput cup $SL1Y 0
  echo -e "$( centre_string "" )"

fi

# Spinner variables
P=1;D=1;W=7

# Declare first regular expression - Redump based
REGEX1="^(.*\(Disc )[0-9]+(\).*)$"

# Single disc or multiple discs?
if [[ ! $ZIP_FILE =~ $REGEX1 ]]; then

  # Output file details
  tput cup $SL1Y 0
  STR="Unzipping disc ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Output file name
  tput cup $SL2Y 0
  STR=$(basename "$ZIP_FILE")
  echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

  # Unzip
  unzip -DDnq "$ZIP_FILE" -d "$CACHE_DIRECTORY" &
  UNZIP_PID=$!

  # Ticker
  while kill -0 $UNZIP_PID 2> /dev/null; do
    ticker
    sleep 0.05
  done

else

  # Status update
  tput cup $SL2Y 0
  STR="Checking for other discs ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Build second regular expression
  REGEX2=${BASH_REMATCH[1]}[0-9]+${BASH_REMATCH[2]}
  REGEX2=$(echo "$REGEX2" | sed -e 's/(/\\(/g' -e 's/)/\\)/g' -e 's/\./\\./g')

  # Unzip discs  
  CURRENT_FILE=1
  find "$ZIP_FILE_DIRECTORY" -maxdepth 1 -regextype posix-extended -regex "$REGEX2" | sort -t '\0' -n | while read LINE; do

    # Output file details
    tput cup $SL1Y 0
    STR="Unzipping disc ${CURRENT_FILE} ..."
    echo -e "${TC_CYAN}$( centre_string "$STR" )"

    # Output file name
    tput cup $SL2Y 0
    STR=$(basename "$LINE")
    echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

    # Unzip
    unzip -DDnq "$LINE" -d "$CACHE_DIRECTORY" &
    UNZIP_PID=$!
  
    # Ticker
    while kill -0 $UNZIP_PID 2> /dev/null; do
      ticker
      sleep 0.05
    done

    # Next file
    CURRENT_FILE=$(( $CURRENT_FILE + 1 ))

  done

fi

# Clear display and restore cursor
tput sgr0
tput clear
tput cnorm

# Launch lr-pcsx-rearmed
/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-pcsx-rearmed/libretro.so --config /opt/retropie/configs/psx/retroarch.cfg "$CUE_FILE" --appendconfig /dev/shm/retroarch.cfg &> /dev/shm/runcommand.log

# Exit
exit

I added the following to /opt/retropie/configs/psx/emulators.cfg and set it to the default emulator

lr-pcsx-rearmed-unzip = "bash /opt/retropie/configs/psx/lr-pcsx-rearmed-unzip.sh %ROM% &>/dev/tty"

NB. The cache directory is on an external HDD so I'm not worried about SD card failure.

Edit: Updated script to not use regex for single disc games.

Clyde

Just a comment to @IyonUK's script: If your machine has enough ram, you might change the CACHE_DIRECTORY to something in the ramdisk /dev/shm (shared memory), e.g. /dev/shm/psx-cache. Most of today's Linux distributions install shm by default. (Does Retropie even work without it, since it uses it for Runcommand?)

Be sure to change CACHE_IDEAL_SIZE accordingly. You can check the size and free amount of shm with the command df -h. But leave some of it for other applications and the system.

As it is with ramdisks, everything stored there is lost on a system shutdown or reboot. So, contrary to on-disk caching, the zip extraction will have to happen every first time a game is started after a system shutdown or restart. Writing to ram is much faster than to disk, though.