Trying to make script module for my application please help

ziggurat

I can't find any documentation on how to write my own script module. I have only read the pre existing once. Please check if this one should work. Give feedback, I want to follow best practice.

I am working on a GOG client for RetroPie, and would very much like it to be a click to install installation.

#!/usr/bin/env bash

# This file is part of The RetroPie Project
#
# The RetroPie Project is the legal property of its developers, whose names are
# too numerous to list here. Please refer to the COPYRIGHT.md file distributed with this source.
#
# See the LICENSE.md file at the top-level directory of this distribution and
# at https://raw.githubusercontent.com/RetroPie/RetroPie-Setup/master/LICENSE.md
#

rp_module_id="piegalaxy"
rp_module_desc="Pie Galaxy - Downloand and install GOG.com games in native emulators or ports"
rp_module_licence="GPL https://github.com/sigboe/pie-galaxy/blob/master/LICENSE"
rp_module_section="exp"

function depends_piegalaxy() {
    getDepends jq html2text unar
}

function sources_piegalaxy() {
    gitPullOrClone "$md_build" https://github.com/sigboe/pie-galaxy.git master
}

function build_piegalaxy() {
    local innoversion="1.8-dev-2019-01-13"

    if uname -m | grep -q arm; then
        local arch="armv6j-hardfloat"
    elif uname -m | grep -q x86; then
        local arch="amd64"
    fi

    # Get Innoextract dev build
    wget "http://constexpr.org/innoextract/files/snapshots/innoextract-${innoversion}/innoextract-${innoversion}-linux.tar.xz"
    tar xf innoextract-${innoversion}-linux.tar.xz innoextract-${innoversion}-linux/bin/${arch}/innoextract
    cp "innoextract-${innoversion}-linux/bin/${arch}/innoextract" .
    rm -rf innoextract-${innoversion}-linux innoextract-${innoversion}-linux.tar.xz

    # Get wyvern binary
    if [[ "${arch}" == "amd64" ]]; then
        local wyvernurl="https://demenses.net/wyvern-nightly"
    elif [[ "${arch}" == "armv6j-hardfloat" ]]; then
        local wyvernurl="https://demenses.net/wyvern-1.3.0-armv7"
    fi

    wget -O wyvern "${wyvernurl}"
    chmod +x wyvern

}

function install_piegalaxy() {

    mkdir -p /opt/piegalaxy
    cp ./* /opt/piegalaxy/
}

function configure_piegalaxy() {
    cat <<'_EOF_' >>"${romdir}/ports/Pie Galaxy.sh"
#!/usr/bin/env bash
bash -x /opt/piegalaxy/pie-galaxy.sh
_EOF_
    chmod +x "${romdir}/ports/Pie Galaxy.sh"
}

Before you say innoextract is in the debain repos, I depend on the newer version.

mitu

Not sure what exactly your module does and how it works, but here are some comments

You can test what architecture your using the isPlatform helper function. Here is an example on how it's used.
In the install_piegalaxy function, you're supposed to list the output of your build process in the md_ret_files array, so the setup script knows what to copy to the install folder.
In the build_ function, you should also return something in the md_ret_require array, so that the setup script knows the build completed. Looks like right now your build function is producing nothing, and the Github repo you've added in the source_ function is non-existent (!).
Completing 2 and 3, the setup script will make sure your files (and/or folders) are copied to the /opt/retropie/ports/piegalaxy folder, based on the md_ret_ variables. So you don't need to actually create any folders during install_ or copy anything.
The configure_ part is called both when the module is installed or uninstalled - there's a $mode variable you can consult to know which phase you're in. See here.
Look also how other ports script modules are installed, you don't need to create manually the script itself in the ports rom folder, it's also handled automatically by the setup script with the addPort helper function (see here for an example).

Leaving aside the RetroPie specific function - really only a few - there are a few things missing from your script

you don't build anything during the build_ phase.
what's the wyvern binary functionality ? Note that RetroPie can run on more than ARM7 or amd64 archs, so it's best to compile it if the sources are available, instead of trying to add a generic one.
the same applies to innoextract.

If you intend to have a full client for GoG from your Github, you can build both those binaries from you repo, instead of downloading them from the internet, thus making sure of their availability and usability.

ziggurat

Hello @mitu I am making a GOG client for RetroPie reddit post
This is what I wanted to do:

apt install the dependencies that I can use from the repos
clone my github repo
wget innoextract (not the one in the apt repos), and wget wyvern
copy them to an appropriate folder
have a shortcut in the ports folder.

My needed files are

md_ret_files=( 
pie-galaxy.sh    #main executable
exceptions       #functions for games require their own install method
innoextract      #executable I use to extract setup files
wyvern           #executable that I use for things in gog that require you to log in
)

I dont want all of these to be listed in the Ports menu in EmulationStation, only the .sh file is usable with a gamepad. So I thought to place a shortcut (two liner cat >> to .sh file) Maybe this is not needed to make it show up in EmulationStation as just an app like KODI?

I wanted to do a binary installation because compiling requires getting rust from outside the debian repos. The Wyvern executables, there is one for x86 which works great on several linux machines I have tried it on. And one that work on Raspberry Pi Zero and Pi2 and Pi3.

isPlatform Thanks I will read up on this and use it
md_ret_files see example array above, I thought all will be listed as shortcuts in EmulationStation?
md_ret_requiremy install is more a binary install, maybe I am doing this here wrong?
2 and 3: again I just wanted one shortcut in EmulationStations Port section
configure_ should I delete the files my self on uninstall?

you don't build anything during the build_ phase.
- Yeah its more of a binary install
what's the wyvern binary functionality ?
- Its a rust executable that I use to communicate with GOG
- Both x86 and ARM executables are hosted online
the same applies to innoextract
- Its a program to extract setup files
- the zip file downloaded provide both x86 and ARM
If you intend to have a full client for GoG from your Github, you can build both those binaries from you repo, instead of downloading them from the internet, thus making sure of their availability and usability.
- What is the legality of rehosting compiled files?
  - The guy making wyvern seams really committed to providing support for it
  - innoextract seam to have been hosting these files for a long time, do I need to rehost them?

I think if I have to rehost them, I also need to rehost the source, both of them are on github, I could fork them and compile them and host them, and update them sometimes. Are you saying I need to do this to get a pull-request accepted into RetroPie?

mitu

@ziggurat said in Trying to make script module for my application please help:

Are you saying I need to do this to get a pull-request accepted into RetroPie?

No, that's not a pre-requisite and it's not for me to decide. I was thinking more on the lines of having a submodule from the upstream repositories and to compile them locally when you install. Also, what is their license - do they allow re-distribution, do they have source available, etc. ?
Generally, source code should be used instead of opaque binaries - especially if they're hosted outside of your control - but that's just my opinion.

md_ret_requiremy install is more a binary install, maybe I am doing this here wrong?

If you don't have a source part, then don't provide the source_ function. List your modules files so the setup script knows to install them to the destination folder.

md_ret_files see example array above, I thought all will be listed as shortcuts in EmulationStation?

No, they're just copied to the install folder, which is not the ROMs folder.

configure_ should I delete the files my self on uninstall?

No, the uninstallation will remove automatically your script's files, but it was a suggestion to not run configuration actions during the uninstallation.

ziggurat

@mitu

Generally, source code should be used instead of opaque binaries - especially if they're hosted outside of your control - but that's just my opinion.

wyvern is open source on github with GPL 3, but it does require installation of binaries from the internet again. Because the binaries in the debian repo are not new enough. So its just shifting the problem one step further.
innoextract is also on github with zlib license, I have not tried to compile this yet (trying now). innoextract is also in the debian repos, but fails to extract many GOG setup files, the one online works much better.

If you don't have a source part, then don't provide the source_ function. List your modules files so the setup script knows to install them to the destination folder.

I do have the source of my main executable, it is bash script. I depend on two other binaries not in the debian repo.

ziggurat

@mitu So I got it to work almost I got a few issues left.

You see my configure section I need to run addPort to make sure the Ports folder is made and available in emulation station. But my script is incompatible with the runcommand.sh Maybe it is very rude of me to ping @BuZz if so my sincere apologies.

I get things to show if the runcommand launches with the parameter bellow "$md_inst/pie-galaxy.sh >$(tty)" the backslash escaping the $ will input a litteral $(tty) into the runcommand, and not resolve it during the configuration method. Which gives me some output on my app. My app is ncurses/dialog based. And I have a lot of dialogs that are done in subshells like so:

 selected=$(dialog --menu "Choose one" 22 77 16 "${menuOptions[@]}"  3>&2 2>&1 1>&3)
"_${selected:-exit}"

And the commands in the subshell are not output when launched via the runcommand. They are put as binary data to the runcommand log, which is funny, because if you tail that log i will put output partially intact to the screen and mess up the terminal session.

I tried redirecting the dialog command using > in different ways, but I can't get it to work. Using the >$(tty) inside the runcommand will only show dialogs that are not run in subshells.

So bellow I put in a workaround, cat a two liner that overwrites the .sh file launching the runcommand. But that is gonna break systems where pi is not the main user, and I didn't find a way to resolve the romdir, $HOME will resolve to roots home, and $romdir was not set. Also maybe you guys don't like such a workaround, maybe there is a way to make it comaptible with the runcommand. Below I put some images of my app. (there has been visual improvements since I took those pictures)

#!/usr/bin/env bash

# This file is part of The RetroPie Project
#
# The RetroPie Project is the legal property of its developers, whose names are
# too numerous to list here. Please refer to the COPYRIGHT.md file distributed with this source.
#
# See the LICENSE.md file at the top-level directory of this distribution and
# at https://raw.githubusercontent.com/RetroPie/RetroPie-Setup/master/LICENSE.md
#

rp_module_id="piegalaxy"
rp_module_desc="Pie Galaxy - Downloand and install GOG.com games in RetroPie"
rp_module_licence="GPL https://github.com/sigboe/pie-galaxy/blob/master/LICENSE"
rp_module_section="exp"

function depends_piegalaxy() {
	getDepends jq html2text unar
}

function install_bin_piegalaxy() {
	local innoversion="1.8-dev-2019-01-13"
	gitPullOrClone "$md_inst" https://github.com/sigboe/pie-galaxy.git master
	isPlatform "x86" && (cd "$md_inst" && curl -o wyvern -O https://demenses.net/wyvern-nightly)
	isPlatform "arm" && (cd "$md_inst" && curl -o wyvern -O https://demenses.net/wyvern-1.3.0-armv7)
	isPlatform "x86" && downloadAndExtract "http://constexpr.org/innoextract/files/snapshots/innoextract-${innoversion}/innoextract-${innoversion}-linux.tar.xz" "$md_inst" 3 innoextract-${innoversion}-linux/bin/amd64/innoextract
	isPlatform "arm" && downloadAndExtract "http://constexpr.org/innoextract/files/snapshots/innoextract-${innoversion}/innoextract-${innoversion}-linux.tar.xz" "$md_inst" 3 innoextract-${innoversion}-linux/bin/armv6j-hardfloat/innoextract
	chmod +x "$md_inst"/wyvern "$md_inst"/innoextract "$md_inst"/pie-galaxy.sh
}

function configure_piegalaxy() {
	addPort "$md_id" "piegalaxy" "Pie Galaxy" "$md_inst/pie-galaxy.sh >\$(tty)"
	cat >"/home/pi/RetroPie/roms/ports/Pie Galaxy.sh" <<_EOF_
#!/usr/bin/env bash
/opt/retropie/ports/piegalaxy/pie-galaxy.sh
_EOF_
	chmod +x "/home/pi/RetroPie/roms/ports/Pie Galaxy.sh"
	[[ "$mode" == "remove" ]] && rm "/home/pi/RetroPie/roms/ports/Pie Galaxy.sh"
}

main menu
game list
installer

mitu

@ziggurat said in Trying to make script module for my application please help:

You see my configure section I need to run addPort to make sure the Ports folder is made and available in emulation station. But my script is incompatible with the runcommand.sh Maybe it is very rude of me to ping @BuZz if so my sincere apologies.

You'll probably be hearing from him when you'll submit the PR :).

I can take a look over your module script - do you have a source for the main script ? The Github repo you listed is not available.

One thought that pops to mind - since you don't actually run any games, maybe it's ok to not have it appear as a game - so you don't need runcommand to work - and just expose your GUI by implementing gui_piegalaxy (which will run your main script). Similar to how the scrapers included in RetroPie are doing.

mitu

@ziggurat One trick to make sure that the output/input is not 'swalled' when running your script through RetroPie-Setup is to use see >/dev/tty </dev/tty after the command, see here how alsamixer is run.

ziggurat

@mitu said in Trying to make script module for my application please help:

I can take a look over your module script - do you have a source for the main script ? The Github repo you listed is not available.

Yes I have the source, I can give you access. I intend to make it public as soon as I can get this packaging to work properly. There is also a partial blocker for release that it requires you to log into the GOG account via SSH at the time of writing. What is your github account?

Also please look at the main script at this commit https://github.com/sigboe/pie-galaxy/blob/679e80c51fb8f422143ba824fd3dbabc065ed4b4/pie-galaxy.sh
Because I need to roll back the following commit, where I tried to fix this but made it worse.

One thought that pops to mind - since you don't actually run any games, maybe it's ok to not have it appear as a game - so you don't need runcommand to work - and just expose your GUI by implementing gui_piegalaxy (which will run your main script). Similar to how the scrapers included in RetroPie are doing.

Yeah, I could put it in the path, but then I would rather but my script dir in the path because I find the downloaded binaries by doing this scriptdir="$(dirname "$(readlink -f "${0}")")"

mitu

@ziggurat said in Trying to make script module for my application please help:

Yes I have the source, I can give you access. I intend to make it public as soon as I can get this packaging to work properly. There is also a partial blocker for release that it requires you to log into the GOG account via SSH at the time of writing. What is your github account?

I prefer for you to release it publicly for review, so others may chime in. When you're confortable with the functionality and you feel it's ready, submit the PR or post in this topic.

Unfortunately I don't think I have a GoG account :(.

Yeah, I could put it in the path, but then I would rather but my script dir in the path because I find the downloaded binaries by doing this scriptdir="$(dirname "$(readlink -f "${0}")")"

If you use the standard RetroPie installation conventions, your binaries will be copied to the $md_inst dir (expanded as /opt/retropie/ports/piegalaxy automatically), so you can safely assume you can find them there.

ziggurat

@mitu when I get home I will experiment with using >$(tty) <$(tty)

You can register for a free GOG account and "purchase" a few free game.
I support DOSBox and ScummVM games, which there are a number of free games for.

I also support automatically Teenagent (Free) will install as a ScummVM even though it ships with DOSBox on GOG.

There are more free games like Akalabeth, and other Ultima games,
Beneath a Steel Sky, Jill of the Jungle, lure of the temptress, shadow warrior, tyrian, flight of the amazon queen.

All these will install with my program.

Notably, Doom is now on sale like $2 (i dont know, I have another currency), and it ships with DOSBox, but my script will install the WAD as a port. :)

I am working on supporting more native games, maybe I will get X server games working, similar to how steamlink works, then maybe Heroes 3, Jedi Outcast, Jedi Academy and Morrowind might be possible. Also I am furiously reading up on trying to get NEO-GEO games and Amiga games working, but I haven't gotten that working.

mitu

@ziggurat said in Trying to make script module for my application please help:

Notably, Doom is now on sale like $2 (i dont know, I have another currency), and it ships with DOSBox, but my script will install the WAD as a port. :)

I already have DOOM :). It's possible I already have an account, just not using it actively.

hiulit

@ziggurat I'm interested in this project :) I have a GoG account and I'll be happy to test the script when you release it to the public.

ziggurat

@hiulit I am very much looking forward to public, first I need to wrestle with a few more bugs that only really are there when using the run command. Which takes time due to me wrestling with a cold.

@mitu The >$(tty) <$(tty) did work to get all the ncurses based menus on the screen, but there are more stdin/stdout issues, I think the runcommand is designed to steal them for them selves because all the other programs that the runcommand use just use stdin/out/err for logging and errors, I use them for example, taking output of commands into variables, and that seams to break.

When I get it working using the runcommand, it seams to also work without the runcommand, but just in case, I will write in a way to only use the fixes while running retropie. What are the proper way to detect retropie? cat /etc/*releasejust shows raspbian, maybe it is better to try to detect being launched by the runcommand instead.

mitu

@ziggurat I would then recommend to not use runcommand at the moment and design your script to work for now without considering the runcommand interference.
For the other question - RetroPie is not an OS, it can be installed on Ubuntu/Debian/etc. on PC, so don't rely on OS versions. I think it's better to detect if you're starting from runcommand if you want to check if you're on a RetroPie system.

ziggurat

@mitu I got it working with the runcommand now. Only a small visual issue (no progress bar while downloading a game).

But I found a bug in downloadAndExtract, and I opened a pull request :D
https://github.com/RetroPie/RetroPie-Setup/pull/2628