ScummVM / lr-scummvm Launcher and Interoperability Improvements

mitu

@lolonois said in EmulationStation, ScummVM, and .svm files:

Instead I vote for a freely choosable gamefolder name (f.i. DoTT (1993)/ for Day of the Tentacle) and an *.svm with the same basename as the gamefolder (thus DoTT (1993).svm) sibling to the game folder.
While the game detection of scummvm works fairly well, it still leaves the user with the burden of creating the *.svm file manually. But with the file / folder pattern above, the matching *.svm file creation can be automated. Plus it can be automated to fill in the matching game section into the file (e.g., tentacle-de).

This is already supported with the current implementation - the .svm file is created from the ScummVM list of detected games. The name of the game folder doesn't matter as long as the detection works. Of course, the name of the .svm is not intuitive, but if the game name (instead of the short id) is used, as proposed by @ctrueden, then the .svm will be created with a friendly name.
This leaves the manual adding to be handled or configured/explained.

Behind the scenes the game will be detected, added to scummvm.ini and the matching game section will be put into the Myst.svm file. Subsequent launchs will directly pass the game section id (myst-win) from the Myst.svm to ScummVM.

How will the game detection happen ? Is this included in your python script you reference ? I don't see it from a cursory glance at it.

I haven't tested the python script, but couldn't that functionality be implemented as shell script that would look at the detected games and names and writes the .svm files (as the current +Start ScummVM.sh attempts) ?

Lolonois

This is already supported with the current implementation - the .svm file is created from the ScummVM list of detected games. The name of the game folder doesn't matter as long as the detection works. Of course, the name of the .svm is not intuitive, but if the game name (instead of the short id) is used, as proposed by @ctrueden, then the .svm will be created with a friendly name.

I'd wish I had a whiteboard or similar instead of only text, but anyhow. I will try to illustrate with the "Myst" example:

Current implementation: .svm file will be myst-win.svm, containing the basename from 2.
User's ctrueden submission: .svm will be Myst (Masterpiece Edition/Windows/English).svm, containing the basename from 1. (which references to the section in the scummvm.ini).

I felt this naming constraints too limiting, it also creates a logical gap between the svm-filename and the game folder, as the most users won't name the game folder to one of the both basename options above when they deploy their gamefiles and even if they would do it imposes some research in the ScummVM wiki to get the right names.

Furthermore, it forces the user to enter ScummVM GUI everytime he/she deployed additional game(s).

Last but not least: In both above cases these files are written everytime when ScummVM exits. The user may decide to remove additional demos found by the Mass Add... to have a clean presentation in ES, but with the always-recreate-approach they will keep reappearing after each ScummVM exit and ES restart, unless the scummvm.ini is cleaned from such demos ("Castle of Dr. Brain" is such an example).

These limitations made me come up with my approach: Let the user choose game folder and svm-file where the only rules are: a) They use the same basename from the svm-file and b) the svm-file is sibling to the game folder. With these rules it is possible to add games on-the-fly without having to enter ScummVM GUI.

How will the game detection happen ? Is this included in your python script you reference ? I don't see it from a cursory glance at it.

Not easily to spot as it is encapsulated in the scriptmodule, I must admit. It is here

I haven't tested the python script, but couldn't that functionality be implemented as shell script

Yes, could be if these functions can be done without much ado (the inifuncs.sh does not cover all AFAICT):

Create svm-files (e.g. after Mass Add... in ScummVM GUI). Used here
Copy a [section] with all keys from scummvm.ini to lr-scummvm inifile (and optional vice versa). Used in lr-scummvm/romdir-launcher, here
Test if a [section] exists in scummvm.ini or lr-scummvm inifile. Used in lr-scummvm currently here
Find matching [section](s) when a game folder is provided. Used here. Note: There may be more than one match if an game / game folder provides for example different languages (e.g., "Little Big Adventure", which detects lba-gb,lba-de,lba-fr)

Lolonois

Most importantly: If the rule a) is not applied it is impossible to use libretro ScummVM without having beforehand the native ScummVM fully configured while sticking to the one svm-file for libretro and native ScummVM at the same time. Why?

libetro.cpp expects an absolute path to a .scummvm within the game folder file which has to contain the game identifier pointing to the section in the scummvm.ini (in this case for libretro, which is not the same as the native scummvm.ini file). While the extension is easy to modify for the romdir-launcher.sh it is impossible to know which game folder to choose as there is no scummvm.ini section. Also it is not possible to fallback to libetro.cpps autodetect mechanism as the path is not known and the path is crucial for successful autodetect. The only chance to get a game / the games running in libretro is to have the user run Mass Add... again in for lr-scummvm. Not very user friendly IMHO.

If a game section is configured/present in scummvm.ini for the native ScummVM the logic is still likely to fail when rule a) is not applied:

User launches .svm file and has advised to start this game with lr-scummvm.
romdir-launcher has to consult native ScummVM inifile for game section and pick the value of key path=
If found: Absolute path to .scummvm can be built and passed into libretro. (Optionally copy section from native scummvm.ini to libretro's scummvm.ini for customization).
If not found: Game will not start and no chance to advise libretro to the game folder for autodetect. (same case as above).

My suggested approach eliminates the failure at 4.) (in other words: lr-scummvm can be used without ScummVM) and also does not require to re-run Mass Add... for lr-scummvm to get games going.

mitu

@Lolonois Forgot to reply here, apologies for the delay.

I understand the intention to give more flexibility and allow the user to create their SVM files, but to me it looks like your only issue is the naming of the .svm files by ScummVM. What if we remove the offending chars (:) from the name ?
One thing I haven't tested is the impact on scraping when using a custom .svm name (in ES, only ScreenScraper has Scummvm games support).

It's fine allowing the user to specify the .svm file name, but I think the primary method to add the game to ScummVM should still be the ScummVM gui - I'm not keen on detecting the games during startup, then telling the user to run scummvm from the commandline if the game is not detected. Your code assumes the .svm is named after the game folder, correct ?

I'm not sure how this will reconcile with the ability to run multiple games from the same folder, though, I think that would require the user to know beforehand the full id/name of the game variant to add it to the .svm file.

Lolonois

@mitu Nvm. There is no SLA (service level agreement) attached to open source. :) But thanks for being polite.

From a Linux only viewpoint I don't care much about the filename of the *.svm file. Only / seems to be problematic when using the description= entry from the scummvm.ini as filename (which also the OP proposed script uses and it filters / already out.).

What I dislike in the current implementation is that the *.svm files are generated everytime, after ScummVM has been run. This might overwrite changes in the *.svm file the user did (e.g. instead of using lba-fr instead of lba-gb (the default/generated by the script) for "Little Big Adventure")

The scraping with a used defined *.svm filename should also be successful as long as the majority of the game title is used in the filename, but I have not tested it with the builtin scraper of ES.

I agree that the initial "bulk" add should happen from the ScummVM GUI, although it may add some demos/teasers which are not really a game.
The on-the-fly-add would be handy for later occasional adding of a game.
The errormessage I added is the last option, usually something else has been wrong beforehand. If all gamefiles are present autodetect works well.

I did not invent the "contract" that the *.svm and folder must share the same name. It has been done by other contributors for a good reason: To be able to run games with lr-scummvm without having to consult the native ScummVM scummvm.ini file for identifying the path= of a game.
In other words: If this contract is not granted a user must install and configure ScummVM before being able to use retroarch ScummVM. In turn, when running lr-scummvm must do an indirection via the native scummvm.ini to find the gamefolder (which is an annoyance from a decoupling perspective/voiding the orthogonality of each emulator).
However, a user may decide to also run "bulk" add in lr-scummvm to avoid this step, but then the annoyance is transfered to the user (as he or she has to rerun Add Game also for lr-scummvm while she or he did it already in the native ScummVM). This is also contrary to all other systems sharing the same romfiles and is not needed there when changing the emulator.

PS: Maybe it is advisable to move this to "Ideas and Development", regardless if/how it will be implemented, to avoid getting covered by a pile of support questions.

Lolonois

FTR here [1] are some other observations to consider on the Mass add... via GUI (first paragraph). So it seems advisable to review/clean the list of detected games after such run.

[1] github.com/libretro/scummvm/issues/179#issuecomment-1013292476

mitu

NOTE: I've split the conversation from https://retropie.org.uk/forum/topic/29349/ to a new topic.

What I dislike in the current implementation is that the *.svm files are generated everytime, after ScummVM has been run. This might overwrite changes in the *.svm file the user did (e.g. instead of using lba-fr instead of lba-gb (the default/generated by the script) for "Little Big Adventure")

Yes, I agree w.r.t. auto-generation running each time. Maybe checking for added games in scummvm.ini (file timestamp?) and cross-checking with existing .svm file so any additions will not overwrite user edits (checking for game id maybe).

The scraping with a used defined *.svm filename should also be successful as long as the majority of the game title is used in the filename, but I have not tested it with the builtin scraper of ES.

I've briefly tested with a few manually created .svm files and it worked with ScreenScraper. TGDB (the other scraping source) doesn't have a ScummVM platform, so it doesn't apply to it.

I agree that the initial "bulk" add should happen from the ScummVM GUI, although it may add some demos/teasers which are not really a game.
The on-the-fly-add would be handy for later occasional adding of a game.
The errormessage I added is the last option, usually something else has been wrong beforehand. If all gamefiles are present autodetect works well.

I did not invent the "contract" that the *.svm and folder must share the same name. It has been done by other contributors for a good reason: To be able to run games with lr-scummvm without having to consult the native ScummVM scummvm.ini file for identifying the path= of a game.
[..]

I haven't looked in deep at how lr-scummvm works, but the docs on the libretro site describe roughly the same steps as the standalone - start the core, then scan for games [1]. I can only guess that this modifies the core's scummvm.ini.

Alternatively, I see it uses the a .scummvm file, which is created in the game's folder. My idea was to generate this file similar to how .svm files are generated, just to appease both emulators. I guess the launcher can check both scummvm.ini files and - if something new is added - try to generate a .svm/.scummvm file depending on which emulator is installed. They can be basically identical: GameName as the filename and gameid as content.

[1] https://docs.libretro.com/library/scummvm/#adding-a-game-to-the-scummvm-launcher.

Lolonois

Re: auto-generation run every time

Using time on a system with an undisciplined clock by default was too shaky for me. I check if hashes over all sections before and after the launch of ScummVM differ (see my branch in +Start ScummVM.sh).

Re: Scraping

Thanks for testing. Good to know.

Re: lr-scummvm

What I have noticed that lr-scummvm only keeps scummvm.ini entries if they are done via the UI. Manual adds or edits are silently removed. The UI puts the games with dashes into its ini, e.g. [dw2-gb] for Discworld II.
This effect makes my python helper script not usable for the lr-scummvm ini file. :(
I only changed the launcher (romdir-launcher.sh) to create a symlink with extension .scummvm of the .svm in the parent folder. If the .svm does not contain any game id a dummy (.autodetect) file is given to lr-scummvm to trigger autodetection.
(On a side note: I was not able to persist any game specific changes in the libretro scummvm.ini, e.g., setting different volumes for music, sfx and speech - guess I will have to table that to the RA guys...)

However, I had a look at the RA documentation [1, from above] and changed the +Start ScummVM.sh accordingly, which moved the implementation after running Mass Add... / Add Game in a more concise direction:

Re: Adding games via UI and creation/update of *.svm files

First, I could remove all lr-scummvm ini logic from the scummvm_helper.py script.
Second, as retroarch follows the "no dashes" policy for game identifiers I changed the logic in the scummvm_helper.py to act the same way. This means any .svm file should only contain the game id (as in the compability chart of https://www.scummvm.org/compatibility/) and as it is currently documented in the RetroPie Wiki.

The Add Game prompts the user which game to add when variants of one game are found (variants will end up having dashes in their identifier). Retroarch documents that one should remove the dashes and retain only the game short name, but as documentation is rarly consulted IMHO my Python script does the same.
When running Mass Add... this is even more handy, as there is no interaction with the user. Eventually, after applying the Python script, the scummvm.ini has exactly one section for every game which has no dashes and thus can be seamlessly used also for lr-scummvm. But more importantly: When a game id from the .svm is not exactly matched no game specific settings are saved (e.g., volume settings). Additionally it is an annoyance for the user to make the match exact between game id and scummvm.ini section by editing either file and even error prone. IMHO identifying the game id including variant is nearly impossible if the user just deploys the gamefiles in a folder. And -as I wrote earlier- getting the folder name right during game file deployment to match the description= value of the scummvm.ini is also impossible (Chicken and egg problem).
TL;DR: No game id with dashes either in .svm file or in scummvm.ini`s [section].

ctrueden

Thanks @lolonois @mitu and everyone for discussing this issue further!

I wrote a comment on my PR which for convenience I will reproduce here:

Reading over the forum topic, this is a bigger can of worms than I realized. I don't think I'm going to have enough time any time soon to think more carefully about this problem—especially the dichotomy between the two ScummVM flavors.

Off the top of my head, my instinct for solving that issue would be first figure out how to discern which of the two flavors of ScummVM is in use within that installation, and then have the behavior of the shell script contingent on that information. Of course, one must still make decisions about how aggressively to generate .svm files. Best is probably to add some flags to the top of the shell script letting people control this, and then in the documentation on the website explain how to configure these things, so that everyone can decide for themselves how they want it to work. Finally, for default behavior, I think it should come as close as possible to "just working" out of the box as possible, which to me means trying to stay in sync with the ScummVM UI, i.e. autogenerating .svm files always to match what the UI has configured. To accommodate manually created or tuned .svm files, there could be a comment in the file indicating it was autogenerated, which users can remove if they want that .svm to become "sticky" i.e. not get deleted by the auto-syncing logic.

I'll leave this PR open as is, in case anyone else has time to work further on this.

Thanks again to all for everything you do to make RetroPie awesome. ☺️

Lolonois

@ctrueden Glad you followed-up on the "can of worms" sorting. IMO there is nothing wrong to pause a suggested change. However, you are right the ScummVM launcher is an odd landmark and contradicts with the documentation atm.

Luckily we dug up most (if not all) the details. Surprsingly, it turned out there is more to consider as I assumed as I went down the rabbithole. Anyway, the information is in thread and can be picked up at some time. I made my suggestions in a branch [1], all what's needed is some consent with/between the maintainers on which logic to integrate it into the mainline.

Maybe @hhromic and @pjft want to add here, as they were discussing the lr-scummvm [2] module initailly.

[1] https://github.com/Gemba/RetroPie-Setup/tree/fb_scummvm_lrscummvm_scriptmodules (see scriptmodules scummvm.sh and lr-scummvm.sh)
[2] https://retropie.org.uk/forum/topic/19693/lr-scummvm-request-for-comments-and-testing

Lolonois

Did some updates as it turned out that a more recent ScummVM core (2.5+) in lr-scummvm does use the scummvm.ini.

My scummvm_helper.py does apply also again for the lr-scummvm, and:
When a Add Game or Mass Add... is done in lr-scummvm UI it follows the same logic as the native ScummVM:
- Make unique game id entries in scummvm.ini by chopping of any variant suffix.
- It creates *.svm files, if missing. The *.scummvm extension should only be used by the rom-launcher.sh internally as it might produce doubles in the gamelist if these would be enabled in es_systems.cfg.

Update Jan 2024
Tested ok with RetroPie 4.8.6 (ad3ca8c) and ScummVM 2.8 / lr-scummvm 2.8

For those who want to testdrive, ssh to your Pi, then:

git clone --depth 1 --branch fb_scummvm_lrscummvm_scriptmodules https://github.com/Gemba/RetroPie-Setup/ rp_setup_scummvm
cd rp_setup_scummvm
sudo ./retropie_packages.sh scummvm install_bin 
sudo ./retropie_packages.sh scummvm configure 
sudo ./retropie_packages.sh lr-scummvm install_bin 
sudo ./retropie_packages.sh lr-scummvm configure

To return to the original scriptmodules do run the retropie_package.sh commands in /home/pi/RetroPie-Setup folder. No need to run those with install_bin again.

windg

Thanks for the trying to improve the function of scummvm. I test it but it doesn't work very well for me or i did something wrong.

I have the game in the scummvm folder with name : Broken Sword - The Shadow of the Templars . Inside i put the empty file Broken Sword - The Shadow of the Templars.svm. When i choose to launch the game an error from scummvm appear Error running game:Game data didn't found, here is runcommand.log:

Parameters: 
Executing: bash /home/pi/RetroPie/roms/scummvm/+Start\ ScummVM.sh "Broken Sword - The Shadow of the Templars"
/home/pi/RetroPie/roms/scummvm/+Start ScummVM.sh: line 55: /home/pi/RetroPie/roms/scummvm/scummvm_helper.py: Permission denied
Using game controller: Logitech F710 Gamepad (XInput)
Virtual keyboard pack 'vkeybd_default' loaded successfully
WARNING: No path was provided. Assuming the data files are in the current directory!
WARNING: Broken Sword: The Shadow of the Templars failed to instantiate engine: Game data not found (target 'sword1', path '/home/pi/RetroPie/roms/scummvm')!
User picked target 'sword1' (engine ID 'sword1', game ID 'sword1')...
   Looking for a plugin supporting this target... Broken Sword: The Shadow of the Templars
Using game controller: Logitech F710 Gamepad (XInput)
WARNING: initWithU32String: Fribidi not available, will use input strings as fallback.!

Thanks

Lolonois

@windg said in ScummVM / lr-scummvm Launcher and Interoperability Improvements:

/home/pi/RetroPie/roms/scummvm/+Start ScummVM.sh: line 55: /home/pi/RetroPie/roms/scummvm/scummvm_helper.py: Permission denied

This one is on me, I will fix it in the branch. In the meantime you can do:

sudo chmod a+x /home/pi/RetroPie/roms/scummvm/scummvm_helper.py

The other one is that the .svm should be alongside/sibling to the folder thus:

Broken Sword - The Shadow of the Templars/
Broken Sword - The Shadow of the Templars.svm

and not:

Broken Sword - The Shadow of the Templars/Broken Sword - The Shadow of the Templars.svm

Thanks for having it tested on your setup in either case.

mitu

@Lolonois said in ScummVM / lr-scummvm Launcher and Interoperability Improvements:

sudo chmod a+x /home/pi/RetroPie/roms/scummvm/scummvm_helper.py

You don't need sudo for that, but the reason it might not work is an USB mountpoint for roms, which gets mounted by default with noexec. It's safer to use python3 /path/to/script.py to work around this.

windg

Permissions issue fixed and i put .svm
file into the scummvm folder but the scummvm shows the same error: about no game data found. Game is working if i choose to start it from the gui.

I looked in the scummvm Gui and the game is successfully added and svm file have the correct short name.

I saw mitu comment and i have to say that i am running roms from a usb stick.

Update: I reinstall again the cores, i move the games in the sd-card and they are running, Hurray!!

Lolonois

Thanks both for testing and highlighting the culprit. The Windows/USB mount use case constantly escapes me, @mitu .

Anyhow, I just pushed changes to allow also USB/Windows formatted romfolders. Tested ok on my side. Happy for feedback from your side @windg when you can spare some time.

mitu

@Lolonois said in ScummVM / lr-scummvm Launcher and Interoperability Improvements:

The Windows/USB mount use case constantly escapes me, @mitu .

That's easy to fix - buy an USB stick and store your ROMs on it.

windg

@Lolonois I tested it but it's failing with the same error. Also the permission issue is still here.

Parameters: 
Executing: bash /home/pi/RetroPie/roms/scummvm/+Start\ ScummVM.sh "Return to Zork (CD DOS)"
/home/pi/RetroPie/roms/scummvm/+Start ScummVM.sh: line 52: python3 "/home/pi/RetroPie/roms/scummvm/scummvm_helper.py": No such file or directory
Using game controller: Logitech F710 Gamepad (XInput)
Virtual keyboard pack 'vkeybd_default' loaded successfully
WARNING: No path was provided. Assuming the data files are in the current directory!
WARNING: MADE failed to instantiate engine: Game data not found (target 'rtz', path '/home/pi/RetroPie/roms/scummvm')!
User picked target 'rtz' (engine ID 'made', game ID 'rtz')...
   Looking for a plugin supporting this target... MADE
Using game controller: Logitech F710 Gamepad (XInput)
WARNING: initWithU32String: Fribidi not available, will use input strings as fallback.!

Something I noticed, the helper.py file is on the usb stick, into the rom folder, alongside with the +scummvm sh that is normally here.

Lolonois

@windg The location of the helper script is intended, as it can also be used without the launcher. That has not changed with my update.

Just to cross this out: Did you run git pull in the folder with my branch before running configure again?

windg

@Lolonois I didn't did that. THe steps i follow was :

uninstall both scummvm cores
Delete the folder
Reinstall again with the above commands
Run the permission fix command

Thanks!