New bug/crash from 15 days with new ES release

Hex

@pjft This is the first time that PS enabled solved a problem :)

@darknior The problem is present in ES. PS enabled (default) might just be hiding it due to the wake ups and sleeps. Try reproducing it using PS disabled. If you are able to reproduce it consistently then we have a bug.

ttusoulfan

so i have been having a similar issue,except that when i go to install the roms it gives the a similar error.

darknior

Ok @Hex . Tonight i let it with the option enable, and if it not crash, tomorrow i will disabling it again.
But how to help you to repair it ? Can i have some logs ?

Hex

Do not connect any bluetooth devices while testing. I will not need logs as the error is an assert failing. So I can study it based on if it fails due to PS.

Can you tell me if you have screensaver enabled? and of what type?

ChauncyBlack

I can confirm this freezing during idle is caused by a recent emulation station update. I'm running on a combination of RPi3, Retropie 4.2, a known good SD card, and 2 Logitech wireless Rumblepad 2 joysticks. The system had been running flawlessly until my most recent updates of installed base packages (via apt-get update/upgrade), kernel (via rpi-update, currently at 4.9.45), and Retropie packages (via retropie_setup.sh), about a week ago (around August 20 or so). Previous full system update took place around end of June or beginning of July, and had been running fine. Once this freezing behavior started, I thought maybe the update process had corrupted the file system (as it has happened to me before), so I did a completely clean install using the 7/5/17 raspian lite image (last official Jesse release) and a manual install of the retropie script (git clone, etc.). The problem immediately reappeared.

I am using a wired network connection; wifi is on but unused. Bluetooth is on, but no devices are connected. If left idle after booting into emulation station, the system will soft-hang in about an hour, with video and joysticks becoming unresponsive. If I attempt to log in via ssh, the terminal appears to hang, but I can Ctrl-C out of the hung logon process and get a prompt. A dmesg query returns a number of repeated warning about a hung kworker thread being unresponsive for over 120 seconds. That repeats in the log 4 or 5 times, and then the log ends. The process id of the hung kworker thread changes every time. A query with "journalctl -p 3 -xb" gives similar error messages. I haven't been able to find anything else in the logs. At this point in the process, if I attempt a "sudo reboot", my terminal session terminates, but the RPi never actually reboots.

I have found that if I restart the system and quit out of emulation station to the command line, the system remains responsive and functions as expected (e.g. I can login via terminal normally, I can reboot, and dmesg and journalctl behave and expected) and has done so for about six hours since my last reboot.

Hope that helps.

pjft

The file in question hasn't been changed since April, and it has been only for minor maintenance issues. :/

Unless it'd be related to something with Power Savings (which I'm not clear that it would, given the reports from @darknior ), we might be on the wrong track here.

@ChauncyBlack does this also happen if you disable Power Savings?

Sano

FYI , someone had similar errors 7 months ago (no PS at the time) :
https://retropie.org.uk/forum/topic/7744/repeatable-crash-w-bluetooth-error

Could it be related to SDL ?

pjft

@sano My bet is that it is indeed related to ES input management via SDL or the underlying OS drivers, and has been there for a while now. But without hard evidence it's hard to narrow down what may be affecting this.

Sano

According to @bmn001 post, it was only happening when he powered off his controllers after exiting a game, but before full ES reload.

Just a guess, could be totally wrong but that seems logical :
Without knowing the code, we could assume that the function crashing (AddJostckByDeviceIndex ?) has problem if a BT device disappears right before the call.
If it's right, that could at least tend to prove that 7 months ago this function only was called during ES launch, and is now called on a more regular basis.

Edit : of course this won't narrow the bug reasons, but would prove it was already existing, and explain why it seems to happen more frequently...

pjft

This is only called in Window::init(), specifically in InputManager::getInstance()->init();.

This function is called at launch, and every time returning from a game back into ES (where we re-initialize the Window object).

The code is:

int numJoysticks = SDL_NumJoysticks();
for(int i = 0; i < numJoysticks; i++)
{
	addJoystickByDeviceIndex(i);
}

and then the assertion is

assert(id >= 0 && id < SDL_NumJoysticks());

So, reading the code:

id is always >= 0 (it is the iin the cycle)
so this must be failing because, at some point, id is >= to the SDL_NumJoysticks() returned.

Now, this is an awful thing to test or review, as the cycle by definition on runs for i = 0to i < SDL_NumJoysticks() so it means that sometime during the cycle the number of joysticks being returned by SDL is being reduced.

An extra layer of logging there would certainly help troubleshoot this.

Also, if one of you experiencing this could SSH into the Pi after it crashes and retrieve the content of ~/.emulationstation/es_log.txt (or, actually, after restarting, ~/.emulationstation/es_log.txt.bak should be the backup of the previous session if you're on 2.6.0) it may be helpful.

Hex

@pjft I think this is a one time thing. The bluetooth controller must have disconnected at the exact time when it was processing this loop and caused a crash. If @darknior can reproduce this even once then we might be able to bisect it.

pjft

@hex my suggestion is that this is not really new, though, but certainly keen on hearing more about it.

Hex

ES mostly runs on a single thread and this entire process runs sequentially. So PS actually doesnt interrupt this

ChauncyBlack

@pjft Power savings doesn't seem to have an effect either way. The hang state repeats regardless.

In response to sano's suspicion that it may be SDL related, I do recall seeing some sort of SDL-related error when installing the retropie setup script. From the install logs:

dpkg: dependency problems prevent configuration of libsdl2-2.0-0:armhf:
libsdl2-2.0-0:armhf depends on libxcursor1 (>> 1.1.2); however:
Package libxcursor1 is not installed.
libsdl2-2.0-0:armhf depends on libxi6 (>= 2:1.2.99.4); however:
Package libxi6 is not installed.
libsdl2-2.0-0:armhf depends on libxinerama1; however:
Package libxinerama1 is not installed.
libsdl2-2.0-0:armhf depends on libxrandr2 (>= 2:1.2.99.3); however:
Package libxrandr2 is not installed.
libsdl2-2.0-0:armhf depends on libxss1; however:
Package libxss1 is not installed.
libsdl2-2.0-0:armhf depends on libxxf86vm1; however:
Package libxxf86vm1 is not installed.

dpkg: error processing package libsdl2-2.0-0:armhf (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libsdl2-dev:armhf:
libsdl2-dev:armhf depends on libsdl2-2.0-0 (= 2.0.5+5rpi); however:
Package libsdl2-2.0-0:armhf is not configured yet.
libsdl2-dev:armhf depends on libgl1-mesa-dev; however:
Package libgl1-mesa-dev is not installed.
libsdl2-dev:armhf depends on libx11-dev; however:
Package libx11-dev is not installed.

dpkg: error processing package libsdl2-dev:armhf (--install):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for man-db (2.7.5-1~bpo8+1) ...
Errors were encountered while processing:
libsdl2-2.0-0:armhf
libsdl2-dev:armhf

Could these be at all related?

Hex

@ChauncyBlack Your bugs are not related please open a new thread

ChauncyBlack

Contents of es_log.txt.bak (es_log.txt was identical)

lvl2: EmulationStation - v2.6.1rp, built Aug 25 2017 - 23:15:46
lvl2: Creating surface...
lvl2: Created window successfully.
lvl2: Added known joystick Logitech Logitech Cordless RumblePad 2 (instance ID: 0, device index: 0)
lvl2: Added known joystick Logitech Logitech Cordless RumblePad 2 (instance ID: 1, device index: 1)
lvl2: Checking available OpenGL extensions...
lvl2: ARB_texture_non_power_of_two: MISSING
lvl2: Loading system config file /etc/emulationstation/es_systems.cfg...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/amstradcpc/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/amstradcpc/../carbon.xml")
could not find file ""
lvl1: System "amstradcpc" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/arcade/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/arcade/../carbon.xml")
could not find file ""
lvl1: System "arcade" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/atari2600/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/atari2600/../carbon.xml")
could not find file ""
lvl1: System "atari2600" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/atari7800/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/atari7800/../carbon.xml")
could not find file ""
lvl1: System "atari7800" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/atarilynx/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/atarilynx/../carbon.xml")
could not find file ""
lvl1: System "atarilynx" has no games! Ignoring it.
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/fba/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/fba/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/fba/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/fds/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/fds/../carbon.xml")
could not find file ""
lvl1: System "fds" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/gamegear/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/gamegear/../carbon.xml")
could not find file ""
lvl1: System "gamegear" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/gb/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/gb/../carbon.xml")
could not find file ""
lvl1: System "gb" has no games! Ignoring it.
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/gba/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/gba/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/gba/../carbon.xml")
could not find file ""
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/gbc/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/gbc/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/gbc/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/mame/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/mame/../carbon.xml")
could not find file ""
lvl1: System "mame-libretro" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/mame/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/mame/../carbon.xml")
could not find file ""
lvl1: System "mame-mame4all" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/mastersystem/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/mastersystem/../carbon.xml")
could not find file ""
lvl1: System "mastersystem" has no games! Ignoring it.
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/megadrive/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/megadrive/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/megadrive/../carbon.xml")
could not find file ""
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/n64/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/n64/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/n64/../carbon.xml")
could not find file ""
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/neogeo/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/neogeo/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/neogeo/../carbon.xml")
could not find file ""
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/nes/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/nes/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/nes/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/ngp/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/ngp/../carbon.xml")
could not find file ""
lvl1: System "ngp" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/ngpc/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/ngpc/../carbon.xml")
could not find file ""
lvl1: System "ngpc" has no games! Ignoring it.
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/pcengine/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/pcengine/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/pcengine/../carbon.xml")
could not find file ""
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/psx/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/psx/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/psx/../carbon.xml")
could not find file ""
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/retropie/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/retropie/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/retropie/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/sega32x/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/sega32x/../carbon.xml")
could not find file ""
lvl1: System "sega32x" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/segacd/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/segacd/../carbon.xml")
could not find file ""
lvl1: System "segacd" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/sg-1000/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/sg-1000/../carbon.xml")
could not find file ""
lvl1: System "sg-1000" has no games! Ignoring it.
lvl2: Parsing XML file "/home/pi/.emulationstation/gamelists/snes/gamelist.xml"...
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/snes/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/snes/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/vectrex/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/vectrex/../carbon.xml")
could not find file ""
lvl1: System "vectrex" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/zxspectrum/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/zxspectrum/../carbon.xml")
could not find file ""
lvl1: System "zxspectrum" has no games! Ignoring it.
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/auto-allgames/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/auto-allgames/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/auto-favorites/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/auto-favorites/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/auto-lastplayed/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/auto-lastplayed/../carbon.xml")
could not find file ""
lvl1: Warning from theme "/etc/emulationstation/themes/carbon/custom-collections/theme.xml"
(from included file "/etc/emulationstation/themes/carbon/custom-collections/../carbon.xml")
could not find file ""
lvl2: EmulationStation cleanly shutting down.

pjft

@chauncyblack Thanks. Could you share this log when it has frozen? I'd expect to find some log entries on joysticks disconnecting or connecting, and their IDs, towards the end before it crashes. This log seems to pertain to a session that successfully finished.

Similar to the ones at the beginning:

lvl2: Added known joystick Logitech Logitech Cordless RumblePad 2 (instance ID: 0, device index: 0)
lvl2: Added known joystick Logitech Logitech Cordless RumblePad 2 (instance ID: 1, device index: 1)

ChauncyBlack

@pjft Will do. I had already rebooted it after the last hang, so it might be a few hours (or tomorrow) before I can get the logs from a currently hung system.

@Hex New thread opened per your suggestion, on the chance that my errors are totally unrelated.

darknior

@hex said in New bug/crash from 15 days with new ES release:

Do not connect any bluetooth devices while testing. I will not need logs as the error is an assert failing. So I can study it based on if it fails due to PS.

You want i launch ES without Power Saver like before and no BT Keyboard and no x360 Pad ?

Can you tell me if you have screensaver enabled? and of what type?

Yes i have the @pjft Video Screensaver.

Hex

Ok set the screensaver to dim and no bluetooth devices connected.