Pathfinder for Custom Collections - correct pathes for custom collections

thelostsoul

Very nice! Can't wait to ~~test~~ use it. :-) Will report then. Thank you for all the work you done so far.

cyperghost

@thelostsoul Do that.
You can try to improve search routines. The logic in in case selection starting in line 118. Then you see each search case ends with a statement like [[ -z $filefind ]] && file_search 2 this means if the string is empty then try next search case.

I ended it in case 3 with statement [[ -z $filefind ]] && unset array this aborts the search and switches back to next line in custom collection. You can extend as much cases as you want. But one warning. I tried the usecase you told and the result was that for example a game for gamegear was setted to mastersystem .... I don't know if this is a good behaviour ;)

So I disabled this possibility! Maybe I add it later to get called by a command-line filefinder.sh --advanced or maybe selectable with a third option button integreated into custom collection select mask.... I don't know ;)

So far I just change entries but I think the user can add ROMs later. Because I store output of every run backup folders. There the results are listed ;)

thelostsoul

@cyperghost don't forget to update initial posting. ;-)

thelostsoul

Now, I tested the v99. Sorry for trying it so late! It could have been found earlier. I was just doing other stuff (input lag investigation, playing and so on).

First something not a bug, but very annoying on the console. Backup filesnames shouldn't start with numbers. Especially if all of them have similiar numbers. That makes autocomplete a little bit confusing and I have to be very careful. In example, I have 2 collections: Street Fighter and Mario. I can't just "nano 15" TAB, it won't autocomplete. I have to type whole number and be very careful, which next number is the right one, as it autocompletes only the half. If they start with the original name, its much easier and the number on the end of filename would be unimportant.

And then an error happened while importing the Street Fighter 2 collection, but I am not sure. It was asking me which game to choose from and none of them was right. So I decided to select "Don't change !". and it stopped with a message from sed. The original collection changed, I got 8 games found from original 1 game. I think, the games after sfiii aren't checked then.

And I would say don't change or touch anything if the script isn't terminated successfully or is about to. Everything should be done on a temporary file and if script reaches the end, it just asks the user if its ok to replace. Or even if it does not asks, it should only do the change, if it reaches the end of script and copy the temp file over.

Here is the log:

Found write with sed: /home/pi/RetroPie/roms/fba/spf2t.zip -- /home/pi/RetroPie/roms/mame-libretro/spf2t.zip
File not found: /home/pi/RetroPie/roms/ngpc/SNK vs. Capcom - The Match of the Millennium (World) (En,Ja).zip
File not found: /home/pi/RetroPie/roms/psx/Street Fighter Alpha 3 (USA).PBP
Found write with sed: /home/pi/RetroPie/roms/fba/sf2.zip -- /home/pi/RetroPie/roms/fba/hsf2.zip
File not found: /home/pi/RetroPie/roms/gba/Street Fighter Alpha 3 (USA).zip
File not found: /home/pi/RetroPie/roms/psx/Street Fighter EX2 Plus (USA).PBP
Found write with sed: /home/pi/RetroPie/roms/gb/Street Fighter II (USA, Europe) (Rev A).zip -- /home/pi/RetroPie/roms/gb/Street Fighter II (UE) [S][!].gb
File not found: /home/pi/RetroPie/roms/psx/Street Fighter EX Plus Alpha (USA).PBP
Dialog hold 2 files: /home/pi/RetroPie/roms/snes/Street Fighter II (USA).zip -- /home/pi/RetroPie/roms/snes/Street Fighter II Turbo - Hyper Fighting (U) [!].smc /home/pi/RetroPie/roms/snes/Street Fighter II - The World Warrior (E) [!].smc
Found write with sed: /home/pi/RetroPie/roms/snes/Street Fighter II Turbo (USA).zip -- /home/pi/RetroPie/roms/snes/Street Fighter II Turbo - Hyper Fighting (U) [!].smc
File not found: /home/pi/RetroPie/roms/megadrive/Super Street Fighter II - The New Challengers (USA).zip
File not found: /home/pi/RetroPie/roms/fba/sf2ce.zip
File not found: /home/pi/RetroPie/roms/fba/ssf2.zip
File not found: /home/pi/RetroPie/roms/psx/Marvel vs. Capcom - Clash of Super Heroes (USA).PBP
File not found: /home/pi/RetroPie/roms/gba/Super Street Fighter II Turbo - Revival (USA).zip
File not found: /home/pi/RetroPie/roms/megadrive-japan/Street Fighter II' Plus - Champion Edition (Japan).zip
File not found: /home/pi/RetroPie/roms/fba/xmvsf.zip
File not found: /home/pi/RetroPie/roms/megadrive/Street Fighter II' - Special Champion Edition (USA).zip
Found level 1: /home/pi/RetroPie/roms/neogeo/svc.zip
File not found: /home/pi/RetroPie/roms/mame-libretro/sfex2p.zip
File not found: /home/pi/RetroPie/roms/fba/sf2hf.zip
Found level 1: /home/pi/RetroPie/roms/fba/sfiii2.zip
File not found: /home/pi/RetroPie/roms/gba/Super Puzzle Fighter II Turbo (USA).zip
File not found: /home/pi/RetroPie/roms/psx/Street Fighter Alpha - Warriors' Dreams (USA).PBP
File not found: /home/pi/RetroPie/roms/fba/svcpcb.zip
File not found: /home/pi/RetroPie/roms/pcengine/Street Fighter II' - Champion Edition (Japan).zip
File not found: /home/pi/RetroPie/roms/gbc/Street Fighter Alpha - Warriors' Dreams (USA).zip
File not found: /home/pi/RetroPie/roms/snes/Street Fighter Alpha 2 (USA).zip
Found level 1: /home/pi/RetroPie/roms/fba/sfiii3.zip
Found level 1: /home/pi/RetroPie/roms/fba/hsf2.zip
File not found: /home/pi/RetroPie/roms/fba/ssf2t.zip
File not found: /home/pi/RetroPie/roms/mame-libretro/sftm.zip
Found level 1: /home/pi/RetroPie/roms/fba/sfa3.zip
File not found: /home/pi/RetroPie/roms/fba/sfz2al.zip
File not found: /home/pi/RetroPie/roms/fba/sfzch.zip
Found write with sed: /home/pi/RetroPie/roms/snes/Super Street Fighter II (USA).zip -- /home/pi/RetroPie/roms/snes/Super Street Fighter II - The New Challengers (J).smc
Found write with sed: /home/pi/RetroPie/roms/fba/sfa.zip -- /home/pi/RetroPie/roms/fba/sfa3.zip
File not found: /home/pi/RetroPie/roms/psx/Marvel Super Heroes vs. Street Fighter (USA).PBP
File not found: /home/pi/RetroPie/roms/fba/mvsc.zip
File not found: /home/pi/RetroPie/roms/fba/sfa2.zip
Dialog hold 2 files: /home/pi/RetroPie/roms/fba/sfiii.zip -- /home/pi/RetroPie/roms/fba/sfiii2.zip /home/pi/RetroPie/roms/fba/sfiii3.zip

0_1526924825975_Bildschirmfoto von 2018-05-21 19-30-50.png

cyperghost

@thelostsoul Ahh... I know what you mean....

the find romname* won't find romname.extension name, but it will find romname1.extension romname2.extension and so on. But I don't get it why it does not find sfiii.zip because this is searched in first level.... Need to investigat.

About the dialog....
There is nothing implented.

case "$status" in

        2) # Help
           echo "help" 
        ;;

        3) # Extra 
           exit
        ;;
  
        *)  echo "$choices"
            break 
;;

If you choose "Don't change it's the CANCEL-button" and would be case 1....
You see I don't have implented this now ;) Also HELP button would just show "help" message. Not more not less.

Well I think that's the great advantage of inline changing. It's fast and it's automatic.

About the numbering and the autocomplete.... Well... There is no difference if you autocomplete 123456-filename or autocomplete filename-123456

You type first the numbers (indeed very carefully) and later got the filename which is faster to type. But I think it's better to see which file is newer/older

The highest number is the newest file. The filenumber is just the elapesed time in seconds since ... 1970? And I usw WINSCP (a graphical filemanager) to open files ;)

Thank you very much for your report - I appreciate this. But let me investigate abit the sfiii thing. That's weird.... Later I will implent a correct button action.

Things to do:

Correct Button for Cancel
HELP item will view full path of file
Show counter how much dialogs the user should process....
Optional: temp files instead of direct write to file (I prefer direct write)

It's still a work in progress ;) and the sed command failed because there was no paramter parsed to it.

thelostsoul

@cyperghost said in Pathfinder for Custom Collections - correct pathes for custom collections:

But I don't get it why it does not find sfiii.zip because this is searched in first level.... Need to investigat.

I don't have sfiii in my collection, so it can't find it. ;-)
After all, I misunderstood the "Don't change" button. Then everything is fine.
But what about sfiii, if I don't have this game? What if I don't want it to recognize as sfiii2 or sfiii3. What should have i selected then?

About the numbering and the autocomplete.... Well... There is no difference if you autocomplete 123456-filename or autocomplete filename-123456

Look at the image I uploaded. If I do "ls", I get this:

1526922783-custom-super_mario.cfg
1526922783-filefinder.log
1526923279-custom-street_fighter_ii.cfg
1526923279-filefinder.log

Now I type "nano 1" and do TAB, I get "nano 152692". Now what to do? I have to be very careful, next one is a 2 or a 3? Ok, type 3 and TAB. No problem. Now imagine there are more files with a lot of different numbers. They aren't sorted by name, just by random numbers. It would be much easier if I just type "nano c" TAB to get "nano custom-" and then "nano custom-s" TAB. Now imagine different collections with similiar titles and so many numbers. Here is how it could look like

1526922783-custom-super_mario.cfg
1526922783-filefinder.log
1526923279-custom-street_fighter_ii.cfg
1526923279-filefinder.log
1526924521-custom-street_of_rage.cfg
1526924521-filefinder.log
1526924573-custom-snes_collection.cfg
1526924573-filefinder.log

Ok, to be fair, I do most stuff on my pc connected to the raspberry pi. But I just wanted to point it out. Still not big deal.

cyperghost

@thelostsoul said in Pathfinder for Custom Collections - correct pathes for custom collections:

But what about sfiii, if I don't have this game? What if I don't want it to recognize as sfiii2 or sfiii3. What should have i selected then?

Okay ... I should implent a working "Do nothing" or "Proceed to next selection" button ;)

If you don't have a sfiii file then it's clear what happens.
It does not find the right file. Next Step is search for sfiii* files and guess... tada it will show sfiii2 and and sfiii3 ;) So there are several things that I can do...
I can add a working button or just add an "Ignore this entry" into the selection itself ;)

Again about the inline writing. It took me hours to find out how to use the sed command. It's easy to search for some words to replace but file pathes can be a bit different because there are signs in like the slash that are control characters ;) So the easiest way was - instead to replace the string by occourance - to replace the line were the string occours ;) Is the more effictive way because I wanted to avoid using temporary files!

About the filenaming. Maybe real dates would be better ;)
like "2018-05-21--21-50-10-custom-super-mario.cfg"
Would be more clearer but I was in hurry as I did a backup solution, so I used just the time gone in seconds since 1970.

thelostsoul

@cyperghost Yes, sed isn't easy to work, because of all the RegEx rules. I was once working on something different with heavy RegEx manipulations, so RegEx got my favorite function (back then, it was AutoHotkey on Windows :D). But why don't you want work with temporary files? In case something happens or I abort the process, nothing should have changed. Wait a second. And what is "Don't change !" doing? Shouldn't the file stay original afterwards? I think it changed the original file.

To be more clear, imho the options should be named to <Select>, <Stop now>, <Revert changes> and <Ignore this entry>.

cyperghost

@thelostsoul Made new updates....
You can select between advanced and simple search (I recommend simple in first try and then use advanced search in second try)

In ROM selection screen, Exit works, Skip works, Select works
Help does not work ... will display just a message box ;)

Newest version is here

thelostsoul

I have a real world use for your script now. I decided to change a lot of roms at NES and SNES, mostly from European to US version, because of 60 Hz. And now, many of my collections aren't up to date. But here comes your script and now they are. I have updated some of my collections now and I don't see any problem. Using Simple search. It works so far and it saved me some time, instead of manually checking every single file!! So a big thank you.

Edit this line: I forgot about the executable bit. To use the script without typing bash in front, you need to give the executable right for the script file with "chmod +x filefind.sh". Then for the starting script you need to type "./filefind.sh". Maybe add this info.

If the script is done, the screen don't change on my setup. It still shows the last menu entry like this:
As a sidenote, I use PuTTY client.

Ok, here is something else in my Action Adventure collection:

Found write with sed: /home/pi/RetroPie/roms/gbc/Legend of Zelda, The - Link's Awakening DX (G) [C][!].gbc -- /home/pi/RetroPie/roms/gbc/Legend of Zelda, The - Link's Awakening DX (G) [C][!].gbc

It is the same path, so no need for sed right? Is this a "BUG" or do I miss something? Same thing happened with "Kirby's Block Ball (U) [S][!].gb". Oh and with two other game boy games too. So it must be a game boy thing here, maybe because of the file endings. Or maybe not, it happened with snes entries too. Here entries when sed was used, while it was exactly the same:

Clock Tower (J) ]T+Eng1.01_AGTP].smc
Live A Live (Japan) [En by Aeon Genesis v2.0Deluxe].sfc
Alcahest (J) [T+Eng1.0_FH].smc

and some more. All entries like these above.

Another thing. It did not find alternative for

Mario Bros. Classic (E) [!].nes

while I have

Mario Bros. (JU) [!].nes

It is simple mode, I know, but after using simple mode, the entry is gone. Afterwards using advanced mode does not help. So user never knows when to use advanced mode. Maybe a function to restore a backup then?

cyperghost

@thelostsoul said in Pathfinder for Custom Collections - correct pathes for custom collections:

It is the same path, so no need for sed right? Is this a "BUG" or do I miss something? Same thing happened with "Kirby's Block Ball (U) [S][!].gb". Oh and with two other game boy games too. So it must be a game boy thing here, maybe because of the file endings. Or maybe not, it happened with snes entries too. Here entries when sed was used, while it was exactly the same:

Clock Tower (J) ]T+Eng1.01_AGTP].smc
Live A Live (Japan) [En by Aeon Genesis v2.0Deluxe].sfc
 Alcahest (J) [T+Eng1.0_FH].smc

I hunted that down. Please update the script... it's an error in find routine! The find command seems to be unable to hunt ] chars. So I escaped that.

You can try manually. Go to that directory and use find -name "Clock Tower (J) ]T+Eng1.01_AGTP].smc" and you will get no result. Type find-name "Clock Tower (J) ]T+Eng1.01_AGTP*" will get the result. Now type "Clock Tower (J) ]T+Eng1.01_AGTP/].smc"` and you will get a result. Don't ask me why!

and some more. All entries like these above.

Another thing. It did not find alternative for

Mario Bros. Classic (E) [!].nes
and 
Mario Bros. (JU) [!].nes

No chance!
The string compare is *Mario*Bros.* and *Mario*Bros.*Classic*
You can edit the ignorelist in line 13 and add Classic, then Classic will be escaped with a star and the search string is *Mario*Bros.* vs *Mario*Bros.*** and this will be a hit!

PS: Please tag my name, I just searched the forum today and saw an update done to this post. So the bug was very easy to resolve but took some time because I don't checked that post ;)

EDIT:::
Reverted the changes.... they don't work escaping /] is a failure

cyperghost

Final release

Fixed error in search level 1 @thelostsoul reported

So there is no sed if file is found (I just escaped opening square bracket)
Added Region settings mode inspired by @thelostsoul .... This is simply a search with all brackets removed. It will not work very good on arcade games!
Added Info text in the beginning (therefore it's a final release because the script explains what it does now ;))
All textoutput is hidden now (except if file is not found), so please let the program take it's time, you can enable logging if you edit calls to record() "MESSAGE" "#" the # is a number....
- 0 enables text log only
- 1 is text log and screen log
- 2 is screen log only
- last entry is -1, the disables all logging
- Guys you will find out

Examples of it's work

I used @jamesnj entry for a custom collection: ARCADE COLLECTION and processed with my script

log output is a 100% conversion!

Found write with sed: /home/pi/RetroPie/roms/fba/timeplt.zip -- /home/pi/RetroPie/roms/arcade/timeplt.zip
Found write with sed: /home/pi/RetroPie/roms/fba/atetris.zip -- /home/pi/RetroPie/roms/arcade/atetris.zip
Found write with sed: /home/pi/RetroPie/roms/fba/qbert.zip -- /home/pi/RetroPie/roms/arcade/qbert.zip
Found write with sed: /home/pi/RetroPie/roms/fba/jrpacman.zip -- /home/pi/RetroPie/roms/arcade/jrpacman.zip
Found write with sed: /home/pi/RetroPie/roms/fba/btime.zip -- /home/pi/RetroPie/roms/arcade/btime.zip
Found write with sed: /home/pi/RetroPie/roms/fba/altbeast.zip -- /home/pi/RetroPie/roms/arcade/altbeast.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/superpac.zip -- /home/pi/RetroPie/roms/arcade/superpac.zip
Found write with sed: /home/pi/RetroPie/roms/fba/dkong3.zip -- /home/pi/RetroPie/roms/arcade/dkong3.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/pacman.zip -- /home/pi/RetroPie/roms/arcade/pacman.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/astdelux.zip -- /home/pi/RetroPie/roms/arcade/astdelux.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/galaxian.zip -- /home/pi/RetroPie/roms/arcade/galaxian.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/mrdo.zip -- /home/pi/RetroPie/roms/arcade/mrdo.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/mspacman.zip -- /home/pi/RetroPie/roms/arcade/mspacman.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/mpatrol.zip -- /home/pi/RetroPie/roms/arcade/mpatrol.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/digdug.zip -- /home/pi/RetroPie/roms/arcade/digdug.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/kangaroo.zip -- /home/pi/RetroPie/roms/arcade/kangaroo.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/qix.zip -- /home/pi/RetroPie/roms/arcade/qix.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/gravitar.zip -- /home/pi/RetroPie/roms/arcade/gravitar.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/junglek.zip -- /home/pi/RetroPie/roms/arcade/junglek.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/ddragon.zip -- /home/pi/RetroPie/roms/arcade/ddragon.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/spyhunt2.zip -- /home/pi/RetroPie/roms/arcade/spyhunt2.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/ladybug.zip -- /home/pi/RetroPie/roms/arcade/ladybug.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/paperboy.zip -- /home/pi/RetroPie/roms/arcade/paperboy.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/phoenix.zip -- /home/pi/RetroPie/roms/arcade/phoenix.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/sinistar.zip -- /home/pi/RetroPie/roms/arcade/sinistar.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/asteroid.zip -- /home/pi/RetroPie/roms/arcade/asteroid.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/spacedx.zip -- /home/pi/RetroPie/roms/arcade/spacedx.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/defender.zip -- /home/pi/RetroPie/roms/arcade/defender.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/robotron.zip -- /home/pi/RetroPie/roms/arcade/robotron.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/frogger.zip -- /home/pi/RetroPie/roms/arcade/frogger.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/joust.zip -- /home/pi/RetroPie/roms/arcade/joust.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/joust2.zip -- /home/pi/RetroPie/roms/arcade/joust2.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/congo.zip -- /home/pi/RetroPie/roms/arcade/congo.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/dkong.zip -- /home/pi/RetroPie/roms/arcade/dkong.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/dkongjr.zip -- /home/pi/RetroPie/roms/arcade/dkongjr.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/galaga3.zip -- /home/pi/RetroPie/roms/arcade/galaga3.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/galaga.zip -- /home/pi/RetroPie/roms/arcade/galaga.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/gorf.zip -- /home/pi/RetroPie/roms/arcade/gorf.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/invaders.zip -- /home/pi/RetroPie/roms/arcade/invaders.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/vanguard.zip -- /home/pi/RetroPie/roms/arcade/vanguard.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/xevious.zip -- /home/pi/RetroPie/roms/arcade/xevious.zip
Found write with sed: /home/pi/RetroPie/roms/mame-libretro/zaxxon.zip -- /home/pi/RetroPie/roms/arcade/zaxxon.zip

thelostsoul

@cyperghost Thank you for the update. I am currently rebuilding all of my collections. This may take some time. I will test your script someday later and report then.

cyperghost

@thelostsoul I used the scrip for your genesis custom collection....
2 files are missing....

Sonic Blast .... This file is not in my collection, so impossible to find
DECapAttack ... That's an error in writing, so no chance to discover

oUTPUT OF LOG
https://pastebin.com/9L9D3iU6