PlayStation and .zip files

IyonUK

@mitu Hmm, I hadn’t really thought of the unzipping performance. I’ll have to try a few unzips out.

mediamogul

@mitu said in PlayStation and .zip files:

If you're doing this on a Raspberry PI, you can wait quite a few minutes and maybe more.

No problem. Just script another game to launch while you're waiting. ;)

mitu

@mediamogul said in PlayStation and .zip files:

Just script another game to launch while you're waiting. ;)

Just quickly launch a game while you're waiting for the game to start :)

IyonUK

Well... no need for any of this. Added zip to the supported list in es_systems.cfg and tried a game for the sake of it and it works. RetroArch unpacks it to /tmp/retroarch. It takes a while to start, blank screen whilst it unpacks but that solves my storage issue. 😃

IyonUK

Probably spoke too soon. Need to try some with multiple bin files.

IyonUK

Bit of a mixed bag at the moment. Passing a zip file in does indeed extract the zip to /tmp/retroarch but only the first bin file. I’ve adapted your script to work with a zip file and extract everything to /tmp/psx (cue and all bins). This works fine and the game loads, however RetroArch is now ignoring the display settings I had setup for PCSX ReArmed and is launching in 1080p with only the top-left quarter of the screen displaying the full image (I had display for PCSX ReArmed set to CEA1 with custom offsets/width/height to fill my screen). I noticed that an —appendconfig is used when launching PCSX ReArmed normally but this is only to set the refresh rate to 60hz. The other issue is minor in that running from /tmp/psx also saves the srm file there. So I might have to switch back your script to extract to the rom location. Any ideas why my display settings might be borked?

mediamogul

@iyonuk said in PlayStation and .zip files:

Any ideas why my display settings might be borked?

If these are RetroArch settings, then they should have carried over. If the display settings you're referencing were set in the Runcommand menu and if you've changed the launch command in /opt/retropie/configs/psx/emulators.cfg, then your previous settings are tied to the 'lr-pcsx-rearmed' named launch command from that file. You would need to either configure those same settings for 'lr-pcsx-rearmed-zip', or manually change 'lr-pcsx-rearmed' to 'lr-pcsx-rearmed-zip' in /opt/retropie/configs/all/videomodes.cfg.

IyonUK

I’ve set the runcommand resolution for the new unzip emulator/script to CEA-1 to match which works fine but RetroArch still doesn’t play ball. I had set the aspect ratio to “Custom” and manually entered figures for the X and Y offsets, width and height but those have gone. I have set them back but every time I launch those setting have reset. I checked the config at /opt/retropie/configs/psx/retroarch.cfg and my settings are there (same location as where RetroArch says its saved the settings and the same config passed on emulator launch) so I’m a little lost right now. Will do some more digging later. On a positive note the unzipping etc. works fine, will probably tart it up a bit.

dankcushions

@iyonuk unzipping ~400MB files to an SD card every game launch sounds like an efficient way of destroying an SD card ;)

i guess you could unzip to a ram drive, but multi-disk psx games would consume more system ram than an rpi has.

i would definitely recommend the .pbp approach.

Darksavior

Nothing pure about preferring one disc format over another. Pbp saves space, and your "purist" butt won't know the difference.

IyonUK

@dankcushions said in PlayStation and .zip files:

@iyonuk unzipping ~400MB files to an SD card every game launch sounds like an efficient way of destroying an SD card ;)

I’m currently unzipping to /tmp/psx so on the SD card but I’ll most likely move it to the rom location which is mounted on an external HDD. Although it’d be cheaper to replace the SD card. 😉

I could also consider leaving the unzipped files in place until a certain limit is reached and then delete the oldest files, like a cache.

IyonUK

@darksavior said in PlayStation and .zip files:

Nothing pure about preferring one disc format over another. Pbp saves space, and your "purist" butt won't know the difference.

Probably more to do with effort than purism. Also the PSX2PBP thing failed a few times for me.

IyonUK

@darksavior Loving the Super Famicom BTW. 👍🏻

IyonUK

Hi

I have got a script working to my liking. I'm posting it here in case anyone else wants to use/butcher/edit it.

It creates a .cache directory in /home/pi/RetroPie/roms/psx into which it unzips the games. It keeps the files there until it hits the cache limit (which is set at 5Gb) and then starts deleting older files to recover the space.

There are probably bugs/improvements to be found/made.

As mentioned by @mediamogul I put the following script in /opt/retropie/configs/psx named as lr-pcsx-rearmed-unzip.sh

#!/bin/bash

function centre_string() {
  local LPAD=$(( ( $(tput cols) - ${#1} ) / 2))
  local RPAD=$(( ( $(tput cols) - ${#1} ) - $LPAD ))
  local STRING=$(printf ' %.0s' $(seq 1 $LPAD))${1}$(printf ' %.0s' $(seq 1 $RPAD))
  echo "$STRING"
}

function ticker() {
  STR=""
  B=$(( $P - 1 ))
  while [[ $B -gt 0 ]]; do
    STR="${STR}."
    B=$(( $B - 1 ))
  done
  STR="${STR}:"
  B=$(( $W - $P ))
  while [[ $B -gt 0 ]]; do
    STR="${STR}."
    B=$(( $B - 1 ))
  done
  tput cup $SL3Y 0
  echo -e "${TC_YELLOW}$( centre_string "$STR" )"
  Z=$(( $Z + 1 ))
  P=$(( $P + $D ))
  if [[ $P -gt $W ]]; then
    P=$(( $W - 1 ))
    D=-1
  fi
  if [[ $P -lt 1 ]]; then
    P=2
    D=1
  fi
}

# Clear display and hide cursor
tput clear
tput civis

# Declare terminal colours
TC_CYAN="\e[0;36;49m"
TC_DEFAULT="\e[0;39;49m"
TC_YELLOW="\e[0;33;49m"

# Declare cache variables
CACHE_DIRECTORY="/home/pi/RetroPie/roms/psx/.cache"
CACHE_IDEAL_SIZE=5242880 # 5242880 = 5Gb

# Declare path variables
ZIP_FILE="$(sed '3q;d' /dev/shm/runcommand.info)"
ZIP_FILE_DIRECTORY=$(dirname "$ZIP_FILE")
CUE_FILE="${ZIP_FILE##*/}"
CUE_FILE="${CACHE_DIRECTORY}/${CUE_FILE%.*}.cue"

# Declare status line variables
SL1Y=$(( ($(tput lines) / 2) - 1 ))
SL2Y=$(( $SL1Y + 1 ))
SL3Y=$(( $SL2Y + 1 ))

# Create cache directory (if not exists)
mkdir -p "${CACHE_DIRECTORY}"

# Status update
tput cup $SL2Y 0
STR="Checking cache ..."
echo -e "${TC_CYAN}$( centre_string "$STR" )"
sleep 0.2

# Get current cache size
CACHE_SIZE=$(find "$CACHE_DIRECTORY" -type f \( -iname \*.bin -o -iname \*.cue \) -print0 | du --files0-from=- -c | tail -n1 | cut -f1)

# Does the cache need clearing?
if [[ $CACHE_SIZE -gt $CACHE_IDEAL_SIZE ]]; then

  # Status update
  tput cup $SL1Y 0
  STR="Shrinking cache ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Keep removing files, oldest first, until cache is below maxiumum size
  while [[ $CACHE_SIZE -gt $CACHE_IDEAL_SIZE ]]; do

    # Get oldest file
    FILE_TO_DELETE=$(ls -A1rt "$CACHE_DIRECTORY"/*.bin "$CACHE_DIRECTORY"/*.cue 2>/dev/null | head -n1)

    # Status update
    tput cup $SL2Y 0
    STR=$(basename "$FILE_TO_DELETE")
    echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

    # Delete file
    rm -rf "$FILE_TO_DELETE"

    # Get current cache size
    CACHE_SIZE=$(find "$CACHE_DIRECTORY" -type f \( -iname \*.bin -o -iname \*.cue \) -print0 | du --files0-from=- -c | tail -n1 | cut -f1)

  done

  # Clear
  tput cup $SL1Y 0
  echo -e "$( centre_string "" )"

fi

# Spinner variables
P=1;D=1;W=7

# Declare first regular expression - Redump based
REGEX1="^(.*\(Disc )[0-9]+(\).*)$"

# Single disc or multiple discs?
if [[ ! $ZIP_FILE =~ $REGEX1 ]]; then

  # Output file details
  tput cup $SL1Y 0
  STR="Unzipping disc ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Output file name
  tput cup $SL2Y 0
  STR=$(basename "$ZIP_FILE")
  echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

  # Unzip
  unzip -DDnq "$ZIP_FILE" -d "$CACHE_DIRECTORY" &
  UNZIP_PID=$!

  # Ticker
  while kill -0 $UNZIP_PID 2> /dev/null; do
    ticker
    sleep 0.05
  done

else

  # Status update
  tput cup $SL2Y 0
  STR="Checking for other discs ..."
  echo -e "${TC_CYAN}$( centre_string "$STR" )"

  # Build second regular expression
  REGEX2=${BASH_REMATCH[1]}[0-9]+${BASH_REMATCH[2]}
  REGEX2=$(echo "$REGEX2" | sed -e 's/(/\\(/g' -e 's/)/\\)/g' -e 's/\./\\./g')

  # Unzip discs  
  CURRENT_FILE=1
  find "$ZIP_FILE_DIRECTORY" -maxdepth 1 -regextype posix-extended -regex "$REGEX2" | sort -t '\0' -n | while read LINE; do

    # Output file details
    tput cup $SL1Y 0
    STR="Unzipping disc ${CURRENT_FILE} ..."
    echo -e "${TC_CYAN}$( centre_string "$STR" )"

    # Output file name
    tput cup $SL2Y 0
    STR=$(basename "$LINE")
    echo -e "${TC_DEFAULT}$( centre_string "$STR" )"

    # Unzip
    unzip -DDnq "$LINE" -d "$CACHE_DIRECTORY" &
    UNZIP_PID=$!
  
    # Ticker
    while kill -0 $UNZIP_PID 2> /dev/null; do
      ticker
      sleep 0.05
    done

    # Next file
    CURRENT_FILE=$(( $CURRENT_FILE + 1 ))

  done

fi

# Clear display and restore cursor
tput sgr0
tput clear
tput cnorm

# Launch lr-pcsx-rearmed
/opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-pcsx-rearmed/libretro.so --config /opt/retropie/configs/psx/retroarch.cfg "$CUE_FILE" --appendconfig /dev/shm/retroarch.cfg &> /dev/shm/runcommand.log

# Exit
exit

I added the following to /opt/retropie/configs/psx/emulators.cfg and set it to the default emulator

lr-pcsx-rearmed-unzip = "bash /opt/retropie/configs/psx/lr-pcsx-rearmed-unzip.sh %ROM% &>/dev/tty"

NB. The cache directory is on an external HDD so I'm not worried about SD card failure.

Edit: Updated script to not use regex for single disc games.

Clyde

Just a comment to @IyonUK's script: If your machine has enough ram, you might change the CACHE_DIRECTORY to something in the ramdisk /dev/shm (shared memory), e.g. /dev/shm/psx-cache. Most of today's Linux distributions install shm by default. (Does Retropie even work without it, since it uses it for Runcommand?)

Be sure to change CACHE_IDEAL_SIZE accordingly. You can check the size and free amount of shm with the command df -h. But leave some of it for other applications and the system.

As it is with ramdisks, everything stored there is lost on a system shutdown or reboot. So, contrary to on-disk caching, the zip extraction will have to happen every first time a game is started after a system shutdown or restart. Writing to ram is much faster than to disk, though.

el chupacabra

@mediamogul Was that the only way to do it last year? It seems a pretty crazy workaround.

Also pbp files were mentioned but you shouldn't use pbp files they cause headaches.
@IyonUK You can use chd files with the main psx emulator which is far better and easier.
You can get this on the device itself running. chd files also will embed multiple files into the single file which is great for multiple games.

apt install mame-tools

now the program 'chdman' is installed it is very simple to run it:
chdman createcd -i FILENAME.cue -o FILENAME.chd


chdman createcd -i Castlevania\ Symphony\ of\ the\ night.cue -o Castlevania\ Symphony\ of\ the\ night.chd

The output looks like this:

This will put all your files in a single file, like any archive does. Twisted metal 2 for example goes from 12 files and 800mb to 460mb.

cyperghost

@el-chupacabra No, pbp files are not a crazy workaround it the standard format for Playstation portable games. But I like your small tutorial - you should add it to the PSX section.

The best format (imho) is ecm, no (logical) data loss - it removes the error correction from the physical data device. Are there any emulators supporting this format?

el chupacabra

@cyperghost said in PlayStation and .zip files:

@el-chupacabra No, pbp files are not a crazy workaround it the standard format for Playstation portable games. But I like your small tutorial - you should add it to the PSX section.

The best format (imho) is ecm, no (logical) data loss - it removes the error correction from the physical data device. Are there any emulators supporting this format?

I didn't say pbp was a crazy work around.
https://retropie.org.uk/docs/Playstation-1/ Looks like chd is already mentioned briefly here

cyperghost

@el-chupacabra Oh my bad! But then use the cite-function to avoid me from confusion.

But this does not matter at all - I think chd is the better way to store data (lossless with LZMA for data and FLAC for audio) and can be handeled like original iso/bin with m3u/cue.

Again do you know an emulator working with ecm format?

el chupacabra

@cyperghost
By stating "pbp files were mentioned" I am clearly referring to all of the posts talking about it. I have just seen many issues with it over the years causing game issues. But pbp is nicer for multi-disk games because you don't need a m3u file. That is pbp's only benefit.
https://www.reddit.com/r/emulation/comments/714n7g/pbp_vs_chd_for_mednafenbeetle_psx_opinions/
At the time of their posting this was very new and not even available on retropie as far as I know, though.

No I don't know any that use ecm or plan to, probably because ecm is less efficient than chd and offers no other benefits (unlike pbp offers for multi disc games). Therefore I don't think anyone is going to bother to support it. Maybe on the "to do but not important" list. It uses RAR compression by default 15-20% compression vs chd's 35-45%
Also chd does not use a .cue file

https://en.wikipedia.org/wiki/MAME
https://www.mankier.com/1/chdman

MAME is such a massive project, whatever it creates pretty much becomes standard.