Retropie Installation on Ubuntu Server x64 18.04.1

[Clyde]

@johnodon @mitu Ahh, that's the synergy which I love about open source communies. Thank both of you!

[etheling]

@Clyde you sure sent me down the rabbit hole here! :)

It appears that when using latest stable mainline kernel 5.8.10 (as of 2020/09) it no longer gives out initramfs unpacking failed: Decoding failed error. So finding working kernel may be the path of least resistance here. Or spending ~1s longer booting when using gzip compression.

That said, based on info in that Ubuntu bug report it looks like 'initramfs unpacking...' error is sent as KERN_CRIT (loglevel=2), so I thought I'll be done with it by simply
dropping kernel loglevel to KERN_EMERG (loglevel=0). And sure enough, it's hidden, but after boot I thought I'll check that the error is still in kernel logs and that it got hidden and that loglevel got properly set:

$ pi@retropie ~ $ dmesg | grep -i 'initramfs unpacking'
[    0.834312] Initramfs unpacking failed: Decoding failed
pi@retropie ~ $ uname -a
Linux retropie 5.7.19-050719-generic #202008270830 SMP Thu Aug 27 08:35:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
pi@retropie ~ $ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.7.19-050719-generic root=UUID=639b8c73-24be-4c14-858b-4e4c017dcc8e ro loglevel=0 splash vt.global_cursor_default=0 vt.handoff=7
$ cat /proc/sys/kernel/printk
4       4       1       7

All as expected, except that kernel is now back to KERN_WARNING (loglevel=4).

It turns out that modern Linux distributions set kernel log level during boot to distro specific defaults. In Ubuntu this is controlled by /etc/sysctl.d/10-console-messages.conf where kernel.printk = 4 4 1 7is set (on Raspberry OS / Raspbian these are /etc/sysctl.d/98-rpi.conf, kernel.printk = 3 4 1 3). In 'old' days, with pre-systemd Linuxes, which is when I looked at this direction last time, when you set loglevel boot parameter, that's what you had when the OS was up. So this was a bit confusing at first.

This also lead me to look into boot parameters. As it turns out, quiet is actually a shorthand for loglevel=4 (KERN_WARNING). Both Ubuntu (source code) and Raspberry Pi documentation state this. Thus quiet and loglevel= should never be used at the same time to boot the kernel since one or the other will override the other. Also note that while Ubuntu defaults to 'quiet' (loglevel=4) log level after boot, Raspberry OS switches to loglevel=3 after boot (e.g. lower than quiet).

edit #2: alas, quiet appears to be more than just a shorthand; e.g. if used, it's made available as an environment variable to /usr/share/initramfs-tools/init and other scripts which will limit their output when 'quiet' is specified. So the 'correct' way to limit output during boot is that both quiet and log_level have to be used, in exactly that order (e.g. quiet first so it get's set, and then loglevel= to override the loglevel=4 set by quiet).

So it looks like most documentation online is plain wrong or at least misleading regarding silently booting Linux and recommends mixed use of 'quiet' and 'loglevel' without consideration that their order matters, or does not account for the fact that quiet really means loglevel=4. Or mention that Linuxes will during boot switch to distro specific loglevels, which for example in case of Raspbian OS is more silent than 'quiet' boot parameter.

Based on this, a couple of recommendations:

• Update RetroPie documentation to mention that with quiet and loglevel, order matters. Was: to not recommend using quiet (loglevel=4), and only recommend to use loglevel=3 (which is also where RaspberryOS will switch after boot). And maybe link to RaspberryPi docs about 'quiet' as a reminder to avoid future confusion about the matter.
• Update Ubuntu setup script to use both 'quiet' and 'loglevel=0' in that order instead of 'quiet' @MisterB to actually be silent edit #3: after some consideration; maybe just apply quiet - e.g. no changes needed (as it is silent enough alread, unless using a buggy kernel for which workarounds exist)
• Consider if for truly silent system, also /etc/sysctl.d/{10-console-messages|98-rpi}.conf' should be changed to set lower loglevels?
• Consider if loglevel=0 (or some other loglevel < 4) is a way to go all around? After all this is console loglevel, and everything will still get written to kernel logs if needed for debugging (regardless of the fact that messages to console would get suppressed).

Personally for my RetroPie installation I am going with loglevel=0 for both boot and post-boot (I set /etc/sysctl.d/{10-console-messages|98-rpi}.conf to kernel.printk = 0 4 0 7). That said, I don't think I've fully thought out what negative impacts this might have so I have some hesitation to yet recommend this as a go-to tactic to silence the system. Or if the kernel.printk should be set to e.g. '0 4 8 7' instead - especially I am not confident about 3rd parameter (minimum_console_loglevel) and what it really does, or even if loglevel=0 is really valid, and I'm not ready to read the kernel source yet. Food for thought though.

Fun times.

edit: added link to /proc/sys/kernel/printk documentation.

edit: see edit#2 above and strikeouts (e.g. quiet appears to do more than just set the log level after all).

[Clyde]

@etheling said in Retropie Installation on Ubuntu Server x64 18.04.1:

@Clyde you sure sent me down the rabbit hole here! :)

It seems so, my pleasure. What a hell of an elaborate report! 🧐

So finding working kernel may be the path of least resistance here. Or spending ~1s longer booting when using gzip compression.

Decisions, decisions … 😉

As it turns out, quiet is actually a shorthand for loglevel=4 (KERN_WARNING).

Even after more than 13 years of using Linux as my main OS on many machines, I didn't know that, thanks!

Raspberry OS switches to loglevel=3 after boot (e.g. lower than quiet).

Or quieter than quiet. 😎

So it looks like most documentation online is plain wrong or at least misleading

As it is, sadly, for many documentations ans how-tos on the net, which are either outdated and/or circulate wild combinations from other sources. After all, technical information tends to not aging well and has to be maintained, which requires ongoing time, effort, and dedication.

Based on this, a couple of recommendations:

Which may be better placed in a new thread, preferably in https://retropie.org.uk/forum/category/10/ideas-and-development, or even as issues for the corresponding files on Github?

Fun times.

Indeed. Thanks for all the effort, as OCD-driven it may have been. 🤓

[etheling]

@Clyde Thanks! Although it looks like I jumped the gun a bit (see Edit #2 / strikethroughs in previous post). E.g. I just realized that I still got console message(s) during boot after removing quiet and adding loglevel=0:

Begin: Loading essential drivers ... done
...

Which comes from here:

pi@retropie ~ $ grep -ri 'essential drivers' /usr
/usr/share/initramfs-tools/init:[ "$quiet" != "y" ] && log_begin_msg "Loading essential drivers"

So the 'correct' way appears to be after all to use quiet followed by loglevel= for a desired loglevel to override loglevel=4 set by quiet. I should have just read the Archlinux documentation for silent boot right a way! ;-)

Which may be better placed in a new thread, preferably in https://retropie.org.uk/forum/category/10/ideas-and-development, or even as issues for the corresponding files on Github?

I'll start a new thread later. I think this is not as clear cut as I initially thought it was, so I think some discussion might be useful before creating an issue/PR.

[MisterB]

Lots of great discussion here, so I figured I'd check in. Just wanted to acknowledge that I am working on script updates now, and hope to have something published in the next few days. In addition to incorporating some of the great ideas above into the 'core' install script, I also plan to allow for pre and post-install script folders, which will allow for further customization. My vision is to include a library of optional add-on script files in the repository, which can then be copied/moved/linked into these pre/post folders and be automatically executed by the master script. Still playing with some implementation options.

[johnodon]

On a side note...

I'll go on record as saying that I have MUCH better results using the mini ISO than I do using server. So much less crap getting in the way!

[MisterB]

Latest release is published:
https://github.com/MizterB/RetroPie-Setup-Ubuntu/tree/LTS-20.04

Some of the feedback from above has been put in the "master" script, while others have been setup in the Optional Script Library. Lots of additional information is in the README.

I may include the unattended upgrade and WiFi scripts into the master script at a later time, but want to test those out further before committing.

In the meantime, hopefully some of you will test out these changes and let me know how it goes for you. Thanks!

[johnodon]

@MisterB said in Retropie Installation on Ubuntu Server x64 18.04.1:

Latest release is published:
https://github.com/MizterB/RetroPie-Setup-Ubuntu/tree/LTS-20.04

I'll take a snapshot of my system today and rebuild from scratch using the new scripts.

John

[johnodon]

@johnodon said in Retropie Installation on Ubuntu Server x64 18.04.1:

@MisterB said in Retropie Installation on Ubuntu Server x64 18.04.1:

Latest release is published:
https://github.com/MizterB/RetroPie-Setup-Ubuntu/tree/LTS-20.04

I'll take a snapshot of my system today and rebuild from scratch using the new scripts.

John

OK...just ran my first test and submitted my first issue. :)

https://github.com/MizterB/RetroPie-Setup-Ubuntu/issues/7

I'm combing through the logfile now to look for issues.

After the script finished running, I was dropped at a black screen. I had to ALT+F1 to get a prompt and startx manually. I'll keep investigating but this could be the kernel issue referenced earlier.

John

[MisterB]

I responded to your issue on GitHub.

Not sure when you started your install today, but I made some edits earlier, so it's possible we crossed paths and that is causing your issue. However, I can confirm that I have a working setup built from the mini.iso as of the latest commit - 12794d0.

[johnodon]

@MisterB said in Retropie Installation on Ubuntu Server x64 18.04.1:

I responded to your issue on GitHub.

Not sure when you started your install today, but I made some edits earlier, so it's possible we crossed paths and that is causing your issue. However, I can confirm that I have a working setup built from the mini.iso as of the latest commit - 12794d0.

The issue is only for people who fork your repo. The change to the bootstrap.sh is easy enough...just wondering if there is a way to avoid it programmatically.

John

[johnodon]

@johnodon said in Retropie Installation on Ubuntu Server x64 18.04.1:

@johnodon said in Retropie Installation on Ubuntu Server x64 18.04.1:

@MisterB said in Retropie Installation on Ubuntu Server x64 18.04.1:

Latest release is published:
https://github.com/MizterB/RetroPie-Setup-Ubuntu/tree/LTS-20.04

I'll take a snapshot of my system today and rebuild from scratch using the new scripts.

John

OK...just ran my first test and submitted my first issue. :)

https://github.com/MizterB/RetroPie-Setup-Ubuntu/issues/7

I'm combing through the logfile now to look for issues.

After the script finished running, I was dropped at a black screen. I had to ALT+F1 to get a prompt and startx manually. I'll keep investigating but this could be the kernel issue referenced earlier.

John

FYI...logfile is nice and clean. Also, upgrading kernel to 5.8.10 fixes my issue with X not starting.

[MisterB]

FYI - I have promoted all of the 20.04 code to the master branch, and have archived 18.04 to its own branch.

[johnodon]

@etheling @MisterB @movisman @ObiEric

I think I may have found another fix for the issue of the screen going black after the Plymouth theme (Pacman) and eventually dropping to a prompt. This one doesn't require mucking with kernel updates and is VERY simple...

All I did was add 'exec' before the 'startx' command in the .bash_profile file:

if [[ -z $DISPLAY ]] && [[ $(tty) = /dev/tty1 ]]; then
    exec startx -- >/dev/null 2>&1
fi

This works for me on a vanilla install of Ubuntu 20.04 mini + MisterB's installer script. The only ill effect I have seen so far is that it take 15 - 20 seconds for ES to finally load once past the Pacman theme. I'll continue to investigate to see what is causing the delay.

John

[johnodon]

OK...figured out what is causing the boot process to take longer than expected.

It is the Plymouth theme.

If I remove 'splash' from /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet vt.global_cursor_default=0"

boot time takes about 14 seconds from the last BIOS post message I see.

If I leave 'splash' in...

GRUB_CMDLINE_LINUX_DEFAULT="quiet vt.global_cursor_default=0"

...my boot time goes up to 30 seconds...even though the Pacman theme is gone in like 3 seconds. I think he eats about 10 dots before disappearing.

I can certainly do without the plymouth theme but when I disable it, I am now seeing 2 boot errors that I would need to figure out how to handle.

One is the typical 'clean' message on /dev/sda5 (like below):

dev/sda1: clean, 552599/6111232 files, 7119295/24414464 blocks

The other is a message I am not familiar with:

pnp 00:01 can’t evaluate _crs 12311

Any ideas?

John

[MisterB]

Take a look at the suggestions here:

https://askubuntu.com/questions/772874/how-to-turn-off-the-filesystem-check-message-which-occures-while-booting

I'm wondering if a line like this might suppress both messages:
GRUB_CMDLINE_LINUX_DEFAULT="quiet loglevel=3 fsck.mode=skip splash vt.global_cursor_default=0"

[johnodon]

@MisterB said in Retropie Installation on Ubuntu Server x64 18.04.1:

Take a look at the suggestions here:

https://askubuntu.com/questions/772874/how-to-turn-off-the-filesystem-check-message-which-occures-while-booting

I'm wondering if a line like this might suppress both messages:
GRUB_CMDLINE_LINUX_DEFAULT="quiet loglevel=3 fsck.mode=skip splash vt.global_cursor_default=0"

Thanks. I'll give that a try in a few mins.

[MisterB]

Bad copy/paste above...of course, remove the splash in the above line for your use case.

[johnodon]

@MisterB said in Retropie Installation on Ubuntu Server x64 18.04.1:

Bad copy/paste above...of course, remove the splash in the above line for your use case.

I saw that. :)

So, this works well. The only thing I had to do was set loglevel=0. If I set it to 3, I would see something flash quickly (in red)...to fast to read. Setting to 0 I see nothing.

Thanks for pointing me in the right direction!

John

[MisterB]

How about loglevel 2 (CRITICAL)? Curious if there is a minimum threshold that hides everything for 'normal' use, but can still show a truly critical system error if it occurs.