Create custom-collections from txt file [v1.3, final]

cyperghost

v1.2
TRIM CR from windows files
1.2 = Final version

meleu

I don't know (yet) how the custom collections work, but I would like to make some suggestions on the script...

Suggestion 1

Use $HOME instead of hardcoded /home/pi. It makes the script useful for those who install RetroPie on other platform (example: me).

Suggestion 2

Considering the filenames can't have spaces (I'm not sure if it's valid for custom collections, but assuming that it's not), I would like to suggest the use of this regex to check valid filenames: ^[^\ /]+$. This regex means "no spaces and no slashes".

Suggestion 3

Create the $colpath directory if it doesn't exist.

Suggestion 4

Trim the CR character in done redirection.

result:

#!/bin/bash

txtfile="$HOME/RetroPie/roms/collections.txt"
colpath="/opt/retropie/configs/all/emulationstation/collections"

mkdir -p "$colpath"

while read line; do
    if [[ "$line" =~ ^[^\ /]+$ ]]; then
        file="$colpath/custom-$line.cfg"
        echo "Creating \"$file\"."
        touch "$file"
    else
        echo "Ignoring \"$line\": invalid filename." >&2
    fi
done < <(tr -d '\r' < "$txtfile")

cyperghost

@meleu Hi mate ;) Didn't know you are not using custom-collections;)
Maybe now? Btw: Do you know a good manual for RetroAchivments?

@pjft Wonderfull work ;) One question: Are spaces in custom-collections allowed?

I think this tool can be quite usefull because you can create collections:

No need of keyboard to Pie
No need for custom-.cfg - I always forget this to add
You can create an "overview" of collections because file content will not be changed, so no need to "maintain" the list! Just add a new entry and run the script!

Suggestion 1: Thank you ;) That is usefull for multiuser mashines

Suggestion 2: I think @pjft removed empty characters vie ES keyboard input but filename with space work. So this can be well... explained as "expect the unexpected". Your solution is surly better but will remove spaces ;) So what to do?

Suggestion 3: No, it's already created by default if ES 2.6.0 is updated via RetroPie_setup

Suggestion 4: Seems to me just for ... speed reasons? Because I read file in variable and then rewrite it with trim command. You do this in one slash ;)

So everything is clear for me ... but:
Is the CAT command usefull here? Because it will extract single strings in file?

meleu

@cyperghost said in Create custom-collections from txt file:

@meleu Hi mate ;) Didn't know you are not using custom-collections;)
Maybe now?

Yes. I wanna use it!

Suggestion 3: No, it's already created by default if ES 2.6.0 is updated via RetroPie_setup

But if the user try the script without updating ES he/she will get a bunch of "no such file or directory" errors.

Is the CAT command usefull here? Because it will extract single strings in file?

Well, I used it because tr can't read from a file name. It reads only the stdin. But we can avoid the cat using tr -d '\r' < "$txtfile".

Btw: Do you know a good manual for RetroAchivments?

There's a FAQ on their site. I think it's simple, but I agree the info is a little scattered. I'll try to write two or three paragraphs about it and post here in the forum.

But just to let you start I suggest you to:

Cretate an account at RetroAchievements.org
Configure your RetroArch following the instructions on the docs.
Play something. :-)

pjft

This is the regular expression of things I allow/exclude:

[^A-Za-z0-9-[]()\s']

So, we should allow:

letters (lower and upper case)
numbers
spaces
'

I believe none of these would pose problems with any file system. That was the main criteria.

cyperghost

@meleu As whitespaces are working ~~(even if they are not allowed!)~~ I would suggest following

if [[ "$line" =~ ^[^\/]+$ ]] && [[ -n "$line" ]]; then

remove the whitespace in the regex but check $line if empty or not.

Maybe pjft can tell more ;)
Goodnight mate

EDIT: Was to late
So we should allow whitespaces ;)

pjft

@cyperghost I think in concept everything should work as long as it's accepted by the file system. I just wanted to play it safe and create a minimum common denominator set of characters.

cyperghost

@pjft I think so but as I'm not a C++ coder I can't exactly tell. Thank you for taking part!

The touch command is a smooth way to leave files unchanged so this script seems to be save even if we update with new collections. The exclusion that @meleu made ... the two slashes are correct. But I wan't avoid that LF entries create custom-.cfg files therefore a check of $line parameter is needed.

I will update first post and it's codes as @meleu agrees to changes - maybe he gots a better idea ;)

meleu

@pjft said in Create custom-collections from txt file:

This is the regular expression of things I allow/exclude:

[^A-Za-z0-9-[]()\s']

So, we should allow:

letters (lower and upper case)

numbers

spaces

'

Are you saying that this is what currently works with your custom collections feature? Or is it your personal opinion?

meleu

@cyperghost said in Create custom-collections from txt file:

@meleu As whitespaces are working ~~(even if they are not allowed!)~~ I would suggest following
if [[ "$line" =~ ^[^\/]+$ ]] && [[ -n "$line" ]]; then
remove the whitespace in the regex but check $line if empty or not.

looks like you didn't test that... :-) that regex don't match empty strings. Try it and see.

prompt$ [[ "" =~ ^[^\/]+$ ]] && echo valid || echo invalid
invalid

Translating that regex:

^ - start of the line
[^\ /] - any character that is NOT a space '\ ' or a slash '/'.
+ - one or more occurences (here is the trick for not allowing empty strings)
$ - end of the line

pjft

@meleu those are the checks I implemented for when the user can type a collection name.

cyperghost

@meleu You did forget the timeshift ;)
I updated first posting with script!
It works flawless...
Whitespaces in filenames of collections were never the problem

@pjft I think the simple RegEx method inside ES is okay and covers all BE/AE usecases

meleu

@cyperghost just to match the same limits @pjft imposed to what the user can type for a collection name, I suggest this regex in the script: ^[][()\ \'A-Za-z0-9-]+$.

Then that if inside the while becomes this:

if [[ "$line" =~ ^[][()\ \'A-Za-z0-9-]+$ ]]; then

@pjft out of curiosity: is the underscore _ invalid?

pjft

@meleu I suppose it is, by omission, not deliberately. As I said, I tried to get a regular expression that would match the majority of names people would try to enter - it was not extensive, but had the main concern of not allowing people to enter characters that would fail miserably in some OSs ('/', '*', '?', etc).

Underscores could certainly be added. My recommendation is that if you manage to try it out and confirm it works via your script in ES, then by all means keep it. As I mentioned, these checks are only used at text entry time, so they shouldn't affect in any way how ES would work if you create them elsewhere. But if they fail, don't hold me against it :)

cyperghost

@meleu I updated!
You restrict underscore with this but well ... I think that enough

Thx for your help and in creating generic solutions ;)

@pjft

But if they fail, don't hold me against it :)

Always ;)

cyperghost

@meleu @pjft
I updated previous post to give tributes.
Well I think there are other characters that can be included. But I think space and "-" and () are good way.
The underscore works btw ;)
What do you both think.... Is v1.3 with RegEx ^[][()\ \'A-Za-z0-9-]+$ final? or should it be extended. For my personal usage it's enough

We can add \_ and then the underscore works, what's next? #,&,!,?,~.....

pjft

@cyperghost No, please stay away from most of the characters you just mentioned, as some of them are special characters in different file systems (~, ?, etc).

I'd rather keep the REGEXP exactly the same for compatibility purposes. I don't like to give people tools for them to easily shoot themselves in the foot :)

cyperghost

Okay... I update script (hopefully last time) and add the underscore.

meleu

@cyperghost said in Create custom-collections from txt file [v1.3, final]:

We can add _and then the underscore works

There's no need to scape underscore. Then the regex is ^[][()\ \'_A-Za-z0-9-]+$

w0lv3r1nix

Hi

Really nice idea ! Thanks

Suggestion :
Instead of using if/else, just sanitize the string with the regex used in the if:
sline=${$line//^[][()\ '_A-Za-z0-9-]/}