CRASH TO TERMINAL, keyboard stops working, rebuilding doesn't fix

Dochartaigh

BACKGROUND: This is a recurring problem for months now. I've seen TONS of posts about this happening to other people but NEVER a solution to fix the problem. This has happened on nearly every conceivable hardware/software combination possible: multiple Raspberry Pi's (both Pi 3 and Pi Zero 1.3), on multiple power supplies, multiple Samsung EVO+ SD cards (and regular EVO, and SanDisk ultra), with every conceivable type of controller out there...it likewise happens no matter how I install RetroPie: it can be from known-working images I put onto the identical brand/size SD card (or even the same exact card I just backed up), happens to my BerryBoot installs I used to use back in 4.0.2, happens from my scratch installs from the image off the RetroPie website. I can wipe the SD card and start over and use that card for months without a problem so we can count out the individual cards as being bad....so basically it's DEFINITELY A SOFTWARE PROBLEM and NOT a hardware problem.

MOST RECENT INSTANCE: The most recent instance of this problem happened on RetroPie 4.1.8, on a Raspberry Pi Zero 1.3, installed from a working SD card image (which was installed from scratch from the Zero-specific .IMG from the RetroPie site).

WHAT ACTUALLY HAPPENS: The system crashes to terminal. It can crash on initial boot, or after booting up and sitting idle for 20 minutes, or after/during a quick gaming session – even just messing around in the menus it can crash. There is no repeatable action(s) I can do to make it crash every time - it seems to be random. Never saw any log in /dev/shm/runcommand.log. When it crashes to terminal my 2.4Ghz wireless keyboard stops working (hard-wired keyboard doesn't work either), but the cursor still blinks in terminal. If the Zero happened to have a wifi dongle plugged in before the crash I can access the Zero through SSH (they usually don't have that dongle, but when this happens to my Pi 3's I can always SSH in).

Also, once I get this screen of death on a machine, that machine is bricked and unusable until I reimage the card, or do a fresh install from scratch. I might get 10 minutes out of it, maybe 20 minutes, but I can usually get it to crash again in ~5-10 minutes no matter what. The worst cases only boot into Terminal (I assume after EmulationStation crashes) - with I believe no keyboard access again per usual (but again, can still access via SSH).

THINGS I TRIED TO FIX THE PROBLEM: Running sudo apt-get update/upgrade/dist-upgrade doesn't do anything. I also use SSH to get to the RetroPie setup menu by using these commands (stolen from SSelph scraper directions)

cd RetroPie-Setup
chmod +x retropie_setup.sh
sudo ./retropie_setup.sh

Then using the GUI I run through all the packages and install them from binary, rebooting between categories (i.e. install all core packages from binary, reboot; then install all main packages, reboot, etc.). When I have the processing power of the Pi 3 I have tried all this but instead used the "install from source" option as well (don't think I've ever successful gotten the Pi Zero to update everything from source....would literally take days I think).

Dochartaigh

Yet another topic about a nearly new setup (exactly like mine) crashing to terminal...yet still silence from all you guys...(and I've honestly been too busy lately to even follow the couple forums I'm on...)

edmaul69

@Dochartaigh are you using a proper shutdown or are you just killing the power or resetting it not through the emulationstation options menu? If you dont have a proper shutdown it is most likely why you keep bricking the system because improper shutdown causes sd card corruption.

Dochartaigh

Hello, proper shutdown every single time. This can also happen as soon as maybe the 3rd or 4th boot.

I would like to re-iterate that I have been seeing posts like mine popping up with frequency for the past month maybe so I know I'm not alone.

Also want to re-iterate that I've literally re-built every conceivable software this system runs on, down to Jessie itself (or at least that's what I think I'm doing ;) with no solution in sight (besides re-formatting and re-installing).

Last, I've tried this on so many different pieces of hardware that the culprit most definitely is something in the code/program itself. There's just no other possible explanation.

BuZz

I have not seen many posts like this and the few I have seen have mostly been down to people using the wrong image or a bad sdcard.

After first install you can try updating retropie-setup, and updating emulationstation from binary. The most recent ES launch doesn't clear the screen on exit so any problem will hopefully be more visible.

If it was a common issue, there would be hundreds of people reporting it - bearing in mind the last images have had over 500,000 downloads

It must be something specific to your set-up.

Just sounds like ES is crashing anyway - runcommand won't log anything - it's not used for launching ES. You might find some data in /home/pi/.emulationstation/es_log.txt but updating ES first would be the best option.

Dochartaigh

@BuZz
You are completely, totally, correct, that crashing to terminal soon after installing is indicative of people using the wrong image (i.e. RP2/3 image on a RP0/1) - and is very common, and is why everybody (including myself) always asks first and foremost the OP which image they used and how many raspberries they see upon booting up. But I'm talking about the others I've been reading about (which as time permits -which is scarce of late- I will link here as I see them) – specifically when the keyboard no longer works when it crashes to terminal and/or typing 'emulationstation' does nothing which is weird and exactly what I've experienced numerous times.

The weird part, and what I hate the most, is there doesn't seem to be ANY fix besides reformatting and starting over. And that's quite extreme don't you think?

You would think that if you run all the sudo apt-get/update/upgrade/dist, isn't that updating/upgrading (essentially rebuilding) the base Jessie linux distro RetroPie is built upon?

And if you then go into the RetroPie setup menu and rebuild everything there as well (from the main packages, to the main and optional and experimental cores, to the drivers, etc) - isn't that essentially rebuilding every single file RetroPie itself uses? ....you would think that does the trick, correct? Not for me, and not for others - and this has happened to me personally maybe 10 times. I'm FAR and away not an expert so these are the only types of things I know to try.

What made me make this topic is I was at my cousin's house where he imaged his own system (i.e. totally different computer, card reader, SD card, and Pi 3 computer than my own setups) and it crashed to terminal. Rebooted and crashed again (x3). We plugged in a keyboard and couldn't type. Same exact thing I've experienced multiple times before...so in the middle of a family party (with tons of kids sitting by looking for something to do ;) I had to reimage the entire card and start over. Then I read about several other people with the same exact weird problem (one last night), and think to myself "there HAS to be a solution to this"...

BuZz

@Dochartaigh don't update everything from source etc. You should also avoid trying to install everything. You should only install things you use (eg don't install all drivers). Updating Jessie packages is unlikely to help too. Please just follow my steps above so we can see why ES is crashing. Thanks.

Despite what you say there are very posts I have seen with this issue unresolved. I will help debug it if you go through the steps.

Dochartaigh

@BuZz
I definitely will Buzz. Thank you as always.

To your steps above, should I enter retropie-setup using this below code? (which will have to be done via SSH since my keyboard never works when it crashes)

cd RetroPie-Setup
chmod +x retropie_setup.sh
sudo ./retropie_setup.sh

Then once I'm in there ONLY update the MAIN packages from binary, correct? - then I'll see the actual error up on screen (since it will no longer clear the screen on crash) - and I can post a photo of my screen, and then we (I mean you ;) can hopefully troubleshoot from there.

I will also go into /home/pi/.emulationstation/ and look for es_log.txt.

I just hope I still have this Samsung 16gb card. I'm pretty sure I markered a big "X" on the back of it in silver Sharpy marker just for troubleshooting purposes as this has been irking me for quite some time now.

BuZz

@Dochartaigh

Nope. That's not what I said :)

You don't need to do chmod +x retropie_setup.sh but yes, the other commands will launch retropie-setup

Then you need to update the RetroPie-Setup script from the main menu, then update just Emulation Station from binary (Manage Packages -> Core Packages)

BuZz

Once you have done that, let me know what happens and I will give you some additional steps.

Dochartaigh

So this is horrible....but I think I either lost that card or more likely re-formatted it and used it for another project. Thankfully (in a sick kinda way), I'm doing my last batch of these (which better be the last friggin batch since I'm getting tired of making these and after this there will be no friends/family left who don't have one!), my last once I made crashed to terminal, the keyboard doesn't work, but I'm SSH'd into it now.

This one that just crashes was made from an ApplePi-Baker backup I put onto a Samsung EVO 16gb card. I know that would make you automatically think that either A.) the backup is bad (which isn't the case as it just worked for me 9 times in a row), or B.) This particular backup didn't copy over to the card correctly this 1 time (which could be the case, but keep in mind this same exact kind of error - to the "t" - has happened to me multiple times from a brand new fresh install from scratch from the image on the RetroPie site).

Here's the Pastebin of /home/pi/.emulationstation/es_log.txt

My 2.4ghz wireless seems to be down (works for SSH, but updating emulationstation keeps on timing out on the download, and I'm getting like 3kbps on my desktop on it (5ghz works weirdly), AND my Realtek RTL8152 ethernet USB hub seems to need special drivers to work on RetroPie so I'm going to have to get that all sorted before I can update and report back.

MWGemini

@Dochartaigh How are you powering the Pis you've seen this problem with? I know when I first started, I had an adequate power supply but what turned out to be really cheap cables, and it was causing brownouts and other problems. Make sure you've got a good power supply AND good cables. That seems like it could be a common denominator in your case, since you've tried so many different hardware components.

Dochartaigh

@MWGemini
For my Pi Zero's I have a couple of the Anker 2-4-port USB wall chargers around the house. They output up to 2.1 amps (might be 2.4) per channel, and I've switched everything over to Amazon Basics micro USB cables which seem very nice. The power supply/cables hopefully aren't the problem as they power all the other working Zero's I have just fine.

@BuZz
I have another topic open about getting my ethernet adapter working since my 2.4ghz wireless has been flaky. Will RetroPie (like version 4.0.2. or maybe 4.1.0) give me a warning if it fails to compile or update something like EmulationStation for example? I know you mentioned RetroPie was JUST updated to NOT clear error messages for better troubleshooting.

I'm wondering because I'm now actually watching my other Pi (not this one I'm having problems with) run all the updates and I don't think I've ever seen that much code flash across the terminal window on the ones which have had issues (it's up to like 15+ pages of code, and it's been doing the update for maybe an hour so far and I don't know how much longer it has to go). I can never remember the problematic ones taking longer than maybe 20-30 minutes to update something which has me wondering if they're really failing to update, and I missed any error message which flashed up, and that's been why I could never fix them.

--once I'm done updating this working Pi Zero I'll see if the malfunctioning one can use the Ethernet adapter (but I'm pretty sure only newer versions of RetroPie can use that ethernet adapter since I could never get it working before - so I might be stuck without internet access to test anything out on the malfunctioning Pi Zero).