Share your collections

cyperghost · 5 Nov 2018, 15:13

@thelostsoul I made several attempts with bash. To strip braces I used this one
${name%% (*}
To get extension you can use this
${name##*.}

Usually the ROM is named like this shema

PATH TO ROM FILE/That is the ROM NAME (Region) [Dump state] !%$.EXTENSION
           |               |                                        |
  We strip this to/    Extract to (                        Save extension
   to get system

Then use find /path to your files for system/ROMNAME without(*.extensionfunction and then we are happy about.
If nothing found then find again but now without extension ;) We have to ignore *.srm and *.state* files for that case :(

cyperghost · 5 Dec 2018, 10:07

@thelostsoul

Here is a bare script! It's bare because it will only operate if there is exactly one match. I don't strip path to roms, so there is no chance to find roms that are located in arcade folder can also be found in mame or fba folders.... So this is a very primitive workaround. Moreover, if you make several runs your gamelist will grow and grow and grow because I don't make crosschecks if file entries that can be modified are already in the lists....

But the idea is clear. Every user can make a run on his game collection and the collection file will increase and increase. With much luck we have a fileset that says. This collection has 21 games in but there are all in all 155 file entries. But this is because someone used zip files, other one got a bad dump.... And if all is good the user got exactly 21 entries in his collection on ES ;)

#!/bin/bash

# >> Create Custom Collection <<
#
# Remember you need ES 2.6.0 at least to use that features
# cyperghost


# Bare script! Please improve!
# Check if custom collections exists
# We can strip-down custom- string later


# INIT
[[ -z $1 ]] && echo "Enter name of custom collection!" && exit
col_file="/opt/retropie/configs/all/emulationstation/collections/$1"
! [[ -f $col_file ]] && echo "File not found!" && exit

# FUNCTION
function find_file () {
    local level="$1"
    local line="$2"
    local strip="$3"
    local match

    case "$level" in
        "-level1")
                match=$(find "$line" -type f 2>/dev/null | wc -l)
                echo $match
        ;;

        "-level2")
                    if [[ $strip == "1" ]]; then
                        match=$(find "${line%% (*}"*."${line##*.}" -type f 2>/dev/null)
                        echo $match
                    else
                        match=$(find "${line%% (*}"*."${line##*.}" -type f 2>/dev/null | wc -l)
                        echo $match
                    fi

         ;;

         "-level3")
                    if [[ $strip == "1" ]]; then
                        match=$(find "${line%% (*}"* -type f 2>/dev/null)
                        echo $match
                    else
                        match=$(find "${line%% (*}"* -type f 2>/dev/null | wc -l)
                        echo $match
                    fi
         ;;               
         
    esac
}

# MAIN
while read line; do

    match_l1="$(find_file "-level1" "$line")"

    if [[ $match_l1 == "1" ]]; then
        echo "File 1 found: $(basename "$line")"
    else 
        match_l2="$(find_file -level2 "$line")"
    fi
        
    if [[ $match_l2 == "1" && $match_l1 == "0" ]]; then 
        line="$(find_file "-level2" "$line" "1")"
        echo "File 2 found: $(basename "$line")"
        echo "$line" >> "$col_file"
    else 
        match_l3="$(find_file "-level3" "$line")"
    fi    

    if [[ $match_l3 == "1" && $match_l2 == "0" && $match_l1 == "0" ]]; then
        line="$(find_file "-level3" "$line" "1")"
        echo "File 3 found: $(basename "$line")"
        echo "$line" >> "$col_file"
    fi
      
done < <(tr -d '\r' < "$col_file")

My first test with the BATMAN Collection ended in 9 "new" added entries ;)

/home/pi/RetroPie/roms/gb/Batman - Return of the Joker (U) [!].gb
/home/pi/RetroPie/roms/gb/Batman - The Animated Series (U).gb
/home/pi/RetroPie/roms/gb/Batman Forever (U) [!].gb
/home/pi/RetroPie/roms/megadrive/Batman - Revenge of the Joker (U) [!].bin
/home/pi/RetroPie/roms/megadrive/Batman Forever (F) [!].bin
/home/pi/RetroPie/roms/megadrive/Batman Returns (U) [!].bin
/home/pi/RetroPie/roms/megadrive/Justice League Task Force (F) [!].bin
/home/pi/RetroPie/roms/snes/Batman Forever (E) [!].smc
/home/pi/RetroPie/roms/snes/Batman Returns (U) [!].smc

Script ouput was:

File 1 found: batman.zip
File 2 found: Batman - Return of the Joker (U) [!].gb
File 2 found: Batman - The Animated Series (U).gb
File 2 found: Batman Forever (U) [!].gb
File 1 found: Batman Beyond - Return of the Joker (USA).zip
File 1 found: New Batman Adventures, The - Chaos in Gotham (USA).zip
File 3 found: Batman - Revenge of the Joker (U) [!].bin
File 3 found: Batman Forever (F) [!].bin
File 3 found: Batman Returns (U) [!].bin
File 3 found: Justice League Task Force (F) [!].bin
File 3 found: Batman Forever (E) [!].smc
File 3 found: Batman Returns (U) [!].smc

Means:
File 1 - 3 ROMs from collection were also located in my ROM folders --> 0 added
File 2 - stripped away all brackets and used same extension - 3 were added to collection
File 3 - stripped away brackets and extension - 6 were added to collection

PittStone · 11 May 2018, 18:12

@thelostsoul Can you post your Collections Lists?

cloudlink · 11 May 2018, 18:25

Here are some of my collections. My rom filenames don’t really adhere to any specific naming convention though.

custom-Warner Bros
custom-TMNT
custom-The Simpsons
custom-Super Mario
custom-Street Fighter II
custom-Star Wars
custom-Sonic
custom-Pokemon
custom-PBS
custom-Pac Man
custom-Nickelodeon
custom-Mortal Kombat
custom-Mega Man
custom-Marvel
custom-Hanna Barbera
custom-Ghost Busters
custom-Final Fantasy
custom-Disney
custom-DC
custom-Castlevania
custom-Cartoon Network

cyperghost · 5 Nov 2018, 19:57

@pittstone

@thelostsoul said in Share your collections:

I have a lot of collections. Can't post them now, as I am not at home.

@cloudlink
Thanks for this huge collection ;)
That's amazing!

Just for an example your Warner Bros Collection was translated by my script to this. So even you naming convention isn't "correct" I was able to get half of your settings on first run. I think that's okay....

But we have to dig deeper into somehow... Maybe checksums ;) But now we could merge your collection and mine in one file ;) The script needs a bit improvment.

It need a first loop with a check if files can in the whole collection are available and put it in an array with array+=("$romfile" "$available") so if file is available the $available will be setted to 1, else to 0 ;)

/home/pi/RetroPie/roms/gb/Tiny Toon Adventures - Wacky Sports (U) [!].gb
/home/pi/RetroPie/roms/gb/Taz-Mania 2 (U) [!].gb
/home/pi/RetroPie/roms/nes/Bugs Bunny Birthday Blowout, The (E) [!].nes
/home/pi/RetroPie/roms/gbc/Tiny Toon Adventures - Dizzy's Candy Quest (Europe) (En,Fr,De,Es,It,Nl).zip
/home/pi/RetroPie/roms/atari2600/Bugs Bunny (Atari) (Prototype) [!].a26
/home/pi/RetroPie/roms/snes/Speedy Gonzales - Los Gatos Bandidos (U) (V1.1) [!].smc
/home/pi/RetroPie/roms/gbc/Looney Tunes - Twouble! (USA) (En,Fr,Es) (GB Compatible).zip
/home/pi/RetroPie/roms/gbc/Looney Tunes - Carrot Crazy (USA) (En,Fr,Es) (GB Compatible).zip
/home/pi/RetroPie/roms/gbc/Speedy Gonzales - Aztec Adventure (USA, Europe) (GB Compatible).zip
/home/pi/RetroPie/roms/snes/Porky Pig's Haunted Holiday (U) [!].smc
/home/pi/RetroPie/roms/megadrive/Taz-Mania (U) [!].bin
/home/pi/RetroPie/roms/snes/Tiny Toon Adventures - Buster Busts Loose! (E) [!].smc
/home/pi/RetroPie/roms/snes/Daffy Duck - The Marvin Missions (U) [!].smc
/home/pi/RetroPie/roms/snes/Animaniacs (U) [!].smc
/home/pi/RetroPie/roms/gbc/Marvin Strikes Back! (USA) (En,Fr,Es).zip
/home/pi/RetroPie/roms/megadrive/Animaniacs (E).bin
/home/pi/RetroPie/roms/megadrive/Sylvester and Tweety in Cagey Capers (UEJ) [!].bin
/home/pi/RetroPie/roms/snes/ACME Animation Factory (E) [!].smc
/home/pi/RetroPie/roms/gb/Looney Tunes (U) [!].gb
/home/pi/RetroPie/roms/gbc/Daffy Duck - Fowl Play (USA, Europe) (GB Compatible).zip
/home/pi/RetroPie/roms/gb/Animaniacs (U) [S][!].gb
/home/pi/RetroPie/roms/snes/Road Runner's Death Valley Rally (U) [!].smc
/home/pi/RetroPie/roms/snes/Bugs Bunny - Rabbit Rampage (U) [!].smc
/home/pi/RetroPie/roms/megadrive/Tiny Toon Adventures - Buster's Hidden Treasure (E) [!].bin
/home/pi/RetroPie/roms/gb/Daffy Duck - The Marvin Missions (UE) [!].gb
/home/pi/RetroPie/roms/megadrive/Daffy Duck in Hollywood (JUE) [!].bin
/home/pi/RetroPie/roms/snes/Taz-Mania (U) [!].smc
/home/pi/RetroPie/roms/megadrive/Bugs Bunny in Double Trouble (4) [!].bin
/home/pi/RetroPie/roms/gbc/Looney Tunes Collector - Alert! (USA) (En,Fr,Es).zip
/home/pi/RetroPie/roms/gb/Tiny Toon Adventures - Babs' Big Break (E) [!].gb
/home/pi/RetroPie/roms/gb/Speedy Gonzales (U) [!].gb

thelostsoul · 5 Dec 2018, 10:29

@cyperghost This is great!
I didn't test it out yet, but now we are talking. :D First, I want thank you for doing/starting this. I was thinking about this "problem" a while back and didn't know how to do it properly. I was more thinking of a Python script, but Bash is fine too. ;-) Currently I am doing other stuff. I'll test this later and will have a look into the script. From a different script I have, I use this to strip down the clutter from filename:

sed -i s/"(.*$"//g "$file"

which deletes everything after (including) the first brace. I use this to get a clean name of the filename for a quick list. But thats not gonna help for your script, as you use those different parts. What do you think about doing it like this: Strip down on both sides (custom collection and files) everything after brace and compare. Found identical entry, use original filename from drive and add to new collection. So, you don't deal with the clutter. Not sure about this approach, if it would work better. It just sounds more simple. What do you think about it?

Custom Collections with max. of 13 elements per collection:
The entries are like showcases for a specific genre or what they are for. I try to get different type of games for different systems. The systems in use are following: atari2600, mame2003, fba, neogeo, gamegear, gb, gba, gbc, megadrive, sega32x, mastersystem, nes, snes, pcengine, psx, n64. The SNES have two extensions, sfc and smc. PC Engine have two extensions, pce and cue.

Genre:

Other:

Insider Tips:

Custom Collections From Theme, for main front page without a limit in elements per collection:

Just a note here. I did a pre selection of games. That means, I often have only one or two best versions of games, with some exception. In example:

I have Earthworm Jim for SNES, because I grew up with it. Found out, the Mega Drive/Genesis version is better, but excluded it for not having duplicates, even if there are differences. Finally added the Mega CD/Sega CD version of the game, as definitive version.
I have Street Fighter 2 versions (Super, Turbo) for SNES and Arcade. Because I play Console versions with gamepad and Arcade versions with arcade stick. And there are duplicates for some personal reason.
I often have only one version of a game, in example the Arcade versions. In example of Bubble Bobble, later I added the NES version to be able to build the correct NES Mini collection.

cyperghost · 12 May 2018, 09:56

@thelostsoul said in Share your collections:

which deletes everything after (including) the first brace. I use this to get a clean name of the filename for a quick list. But thats not gonna help for your script, as you use those different parts. What do you think about doing it like this: Strip down on both sides (custom collection and files) everything after brace and compare. Found identical entry, use original filename from drive and add to new collection. So, you don't deal with the clutter. Not sure about this approach, if it would work better. It just sounds more simple. What do you think about it?

The collection needs to fit exactly with file pathes. So the way how to strip down bash internal, sed, python isn't important. Important is, how the files can be retrieved. The bash script here works in 3 levels:

Just compare file name, is 1 is returned everything is fine if 0 (=no file found) then go to level 2
Compare filename with stripped brackets and original file extension. If one file os found (returns 1) then add the file to collection. If no file is found(0 returned) go to level 3
Compare filename with stripped bracktes and ignore file extension. If one file is found then add to collection, if 0 is returned then make nothing else.

You see fail by design. If 2 entries are found then the script also fails ;) As I said: I can live with that becasue I have no savegames or SRM-states in my ROM folder.

Annother (maybe better) approach would be following.

We use a tool like md5tree and all roms will be scanned for md5 checksum. So we have filename and it's checksum stored in one huge file. We call it md5_romcollection.dat

But this won't save us if someone uses zipped files. But let us be independent of file pathes. I must think about it a bit. I can work with the script I coded ... it covers 75% of my usecase but it's not coded in a good manner and uses unnecessary uses cases and selections.

I would be glad if someone else would step in...

thelostsoul · 5 Dec 2018, 12:06

@cyperghost My idea was using wildcards to get full filename. So strip the clutter path and find file with wildcard. In example "find Bik*" gives me "Biker Mice From Mars (U) [!].smc". Ok, the save files have to be filtered out. Maybe this can be used? This could be used as fallback, if exact filename of different variations aren't found. The md5checksum approach would be good too, but custom collections don't have a md5checksum. And what do you think about stripping down the "-" between Batman and its subtitle and all spaces, plus making everything lower case while comparing?

cyperghost · 5 Dec 2018, 17:39

@thelostsoul That's exactly how it's working
Look: match=$(find "${line%% (*}"*) cuts all charcters till the braket appears and set a wildcard * - that's the searchstring for level 3. So it fits exact your example.

In level 2: match=$(find "${line%% (*}"*."${line##*.}")this cuts all charcater till the braket and set a wildcard * but with extension retrieved from the custom collection.

Your example would be proccesed in my script like following
level 1 - find Biker Mice From Mars (U) [!].smc
level 2 - find Biker Mice From Mars*.smc
level 3 - find Biker Mice From Mars*

I think to filter out all possibles filenames will be a good choice. And the more people share the collections the more the chance to get a full working collection. But as I said I'm satisfied how the script is working. But there is space for lot's of improvements.

See... level 3 is very critical

Because it will also find
Biker Mice From Mars II (U) [!].smc or Biker Mice From Mars (U) [!].smr or Biker Mice From Mars (U) [b].smc or Biker Mice From Mars (J) [!].smc or even worse Biker Mice From Mars (U) [!].autostate

But now I've an idea...... ;)
Why not ask the user for right choice?
So if more then one entry appears open a dialog and ask the user for input.

Annother thing that has to be solved is.... if we process the custom collection.
Should we just add the new found entry? Or should we overwrite the existing one? Because if we just add a new entry we have to check the whole custom collection before we write anything because we will append data on every run. So the overwriting should be more solid! And that's a real easy sed command ;)

thelostsoul · 5 Dec 2018, 17:56

@cyperghost OH man, sorry I was just blind. I only did a brief look into the script. And I forgot about all the other files, so a pure find Bik* would be really a bit greedy. I think asking the user wouldn't be good, its the job of the script to do this. After while with different collections, it could get annoying or the user chose wrong decision, say a noob did it.

Snes roms have mostly 2 different types, .sfc and .smc. Also .zip files are supported too. Maybe we could do it with ls and grep. I did this with a script of mine and it worked.

output="$(ls "$1" | grep -E ".$2")"

Variable $1 is the base directory where all files are. The final output of the command is saved in variable $output. I used this in another script as a function. Sorry for the German comments, it never was intended to release. But now, it helps explaining how I used it.

#!/bin/bash
# Important, this script was intended to be used under Linux / Ubuntu. Not on Raspberry Pi!
base="smb://retropie/roms/"
temp="./games.txt"

cd $base
rm "$temp" 2> /dev/null

# Listet und filtert Verzeichnisse nach Dateityp.
# Argumente: 
#   1 = Verzeichnis
#   2 = Dateityp
function lsfiles ()
{    
    # Führt ls Kommando aus und filtert jede Zeile durch grep.
    # "$()" Konstrukt ermöglicht es als Kommando auszuführen.
    # -E ist nötig, um mehrere Dateitypen zu ermöglichen.
    output="$(ls "$1" | grep -E ".$2")"
    # Speichert die Ausgabe der Variablen in der Datei.
    echo -e "$1\n$output\n" >> "$temp"
}

lsfiles "atari2600" "bin"
lsfiles "mame-libretro" "zip"
lsfiles "fba" "zip"
lsfiles "neogeo" "zip"
lsfiles "gamegear" "gg"
lsfiles "gb" "gb"
lsfiles "gba" "gba"
lsfiles "gbc" "gbc"
lsfiles "megadrive" "md"
lsfiles "sega32x" "32x"
lsfiles "mastersystem" "sms"
lsfiles "nes" "nes"
lsfiles "snes" "sfc|smc"
lsfiles "pcengine" "pce|cue"
lsfiles "psx" "cue"
lsfiles "n64" "z64"

# Pro Zeile, alles nach erstem Klammer löschen.
# -i ist nötig, damit es in der Datei gespeichert wird.
sed -i s/"(.*$"//g "$temp"

Try it manually with:

ls "./" | grep -E ".sfc|smc"

And then we have the filtered list.

cyperghost · 5 Dec 2018, 19:25

@thelostsoul I understand the comments ;) And this will create a list of ROMs without brackets and extension.

But I would stick to find this supports RegEx so wie can include/exclude filetypes ;) and more...

I would also use the level model
I would use sed to directly modify custom collection file rather to add a new entry
Then there are special folder like neogeo/cps/fab/aracade/mame ;) These should be investigated step by step ;)

thelostsoul · 12 May 2018, 19:07

Ok, back to plan A. :D

cyperghost · 13 May 2018, 14:03

@thelostsoul Definitly use find it outmatches ls easily. I think I can provide a working script for all usecases. But thank you for posting your script. It gives me inspiration.

cyperghost · 14 May 2018, 20:35

@thelostsoul I trying to write a script ...
Followup is here Pathfinder for custom collections

cyperghost · 23 May 2018, 18:10

Updated main topic ... We can now use filefinder script to use collections from other users with the personal rom-set
Thanks to @thelostsoul for testing and @pjft for developing the custom collections feature for ES ;)

thelostsoul · 6 Apr 2018, 21:17

New stuff for you geeks:

Sega Mega Drive Collection based on newest iteration of Mega Drive/Genesis game collection for upcoming PS4 (and others release).
Source: denofgeek.com
Neo Geo Mini based on the announced Neo Geo Mini, something like the NES and SNES Mini hardware.
Source: wccftech.com

I hope, the lists are complete. But be careful, this is a personal list and so not everything is correct for you. For the Mega Drive collection in example I used Story of Thor, The (Germany) version and the US version have a different title. So the script will not find your game, it should be impossible. I think the US version is named Beyond Oasis. And about Neo Geo games, I don't know how the script will work for you. The games I have are all under neogeo folder, but you could have them in fba or Arcade too. I know some of you use Mame to play Neo Geo games, so it could be under Mame folders too!

cyperghost · 6 Oct 2018, 12:25

@thelostsoul Thx ;)
As small info... I missed 7 only 2 roms out of your MegaDrive Collection with the filefind.sh script ;) (since version 1.10_060918) It was forced to change every file extension from .md to .bin

DEcapAttack = Decap Attack, so no chance as it seems to be a typing error in your ROMslist
Sonic 3D Blast ~ Sonic 3D Flickies' Island (USA, Europe).md ;), I don't have this game, so no chance for the script.

So the result is 49 of 51 ROMs detected
One ROM was added manually (Decap Attack) and so the result is 50 out of 51
The missing ROM isn't in my list.... but I left the entry into the list.

Thx for sharing

thelostsoul · 5 Jun 2018, 18:18

No problem. Your script does well and helps to minimize the manual work. I did a lot of changing regional roms and the only thing I needed to do was using your script. ;-) But one thing, shouldn't the script do the change of Roman numericals automatically?

Did you try the Neo Geo Mini list?

cyperghost · 5 Jun 2018, 18:36

@thelostsoul said in Share your collections:

No problem. Your script does well and helps to minimize the manual work. I did a lot of changing regional roms and the only thing I needed to do was using your script. ;-) But one thing, shouldn't the script do the change of Roman numericals automatically?

No... but I think I will try annother search level that just cuts the last space. Then you might get a big list of selections. But this would help to get all ROMs very quick

Did you try the Neo Geo Mini list?

Yes! 100% hit

jamesnj · 9 Jun 2018, 02:25

I love this thread, thanks.

I've started making some collections, my favorite right now are pinball games.

If you are interested, my pinball and other lists can be gotten from my retropie github repo.

I have about 20 games listed there ... if anyone has some other hidden pinball gems please send them my way.

Thanks!