DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory

[roslof]

[Acknowledged that RetroPie for Rpi-4B is not currently supported.]
Banging my head on this one and hoping others have stumbled upon a solution for this.

TLDR: After using the Pi for a number of minutes, launching various games and such, suddenly, I can no longer launch games and receive one of these messages in runcommand.log:

DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory
failed to export dumb buffer: Permission denied

or

DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory
Failed to create scanout resource

When these messages appear, it seems I can no longer launch anything and have to reboot the Pi. After rebooting, all seems normal for a limited time until it happens again (guaranteed over time).

Configuration:
Raspberry Pi 4B
RetroPie 4.5.7 (97ee6b5)
RetroArch 1.7.9.2 <--tried multiple
Emulation Station 2.9.0 RP <--tried multiple
Kernel: 4.19.83-v7l+
Raspbian upgraded to latest
Fixed permissions in Retropie/rom folders
Ruled out overclocking -- happens regardless.
Running on HD TV, but happens on 4K too
Tested image on four (4) different devices (three Pi4B 1GB RAM and one Pi4B 4GB RAM)

I don't suspect the issue is directly related to either RetroPie configuration or anything with RetroArch. Seems more like a Raspbian/Buster configuration issue. Possibly a boot/config.txt configuration issue. Hoping somebody here has seen this and know of a solution.

I've searched all over the place for people experiencing this, and only found a few instances (video related, GL related, 4K related -- and even Pegasus related...) but can't find any solution for anything I can do/change to resolve.

Have successfully forced video to HD via boot/config.txt, but that didn't solve the problem.

Any advice would be appreciated.

-Ros

[roslof]

Resolved the issue, but not clear why the solution works.

The problem was [rationally] self-inflicted. I started with a clean Raspbian Buster build and then installed RetroPie manually. So I added the memory split in boot/config.txt as documented here:
https://github.com/RetroPie/RetroPie-Setup/wiki/Memory-Split

Prior to adding the split, I was having intermittent Emulation Station crashes, which reminded me that I might need to set this up like on builds in the past. The ES crashes went away, so I felt this was all good.

But intermittent freezes and memory allocation issues (as noted in OP) were the result.

For now, I commented out all memory split lines:
#gpu_mem_256=128
#gpu_mem_512=256
#gpu_mem_1024=256

Also, per recommendation of some internet threads, I did add cma=384M to cmdline.txt. Granted, I don't fully understand if this step was needed, but for now -- no more crashes/freezes and Emulation Station appears stable.

Will update this thread when I get this all locked down.

[mitu]

[..]

DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory
failed to export dumb buffer: Permission denied

or

DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory
Failed to create scanout resource

This is an indication there's another program that uses (exclusively) GPU and RetroArch cannot get initialise the KMSDRM video context. When you get this error, you should list the running processes (ps -ef) and see which one is 'locking' the DRM crtc.

[...]
https://github.com/RetroPie/RetroPie-Setup/wiki/Memory-Split

On a Pi4, these memory settings don't have an impact on the 3D GPU memory, so it's normal they won't affect EmulationStation's memory consumption/performance.

Also, per recommendation of some internet threads, I did add cma=384M to cmdline.txt. Granted, I don't fully understand if this step was needed, but for now -- no more crashes/freezes and Emulation Station appears stable.

That's interesting, what theme are you using with EmulationStation ? Have you modified the default GPU Memory allocated from the settings (I think the default is 80) ?

[roslof]

Thanks as always, @mitu.

When you get this error, you should list the running processes (ps -ef) and see which one is 'locking' the DRM crtc.

I can revert my changes and do exactly that, but so far, it's been very stable.

That's interesting, what theme are you using with EmulationStation ?

Hursty's Blu-Ray. Been using it since he created it on a Pi3B+. It's graphically intensive (scrolling could stutter a bit as assets load) but has otherwise been stable.

Have you modified the default GPU Memory allocated from the settings (I think the default is 80) ?
No, I left it alone. I did add the cma=384M parameter to cmdline.txt, but was unclear exactly how it works. Only that other non-RetroPie folks recommended this when others experienced the allocation errors.

Do you have a recommendation for the 4B cmdline and/or config.txt? I'm very close to stock Raspbian/Buster, except OC and forcing HD-res (1920x1080 60p). I looked for a sample boot/config.txt, but couldn't find the path on github. When the final RetroPie build is released, I'm sure you guys will have something on the microSD image, but unclear what is optimal for a Pi 4B.

Between the heavy theme, flycast launching, MAME (latest) launching and such, I'm really pushing the device. 3B+ was stable, and I believe 4B will get there. So far -- progress.

[roslof]

Going to set GPU to 256 (gpu_mem=256) in config.txt and remove the cma line from cmdline.txt. If the DRM error reappears, I'll check ps -ef as advised.

Thanks again, mitu!

[mitu]

Hursty's Blu-Ray. Been using it since he created it on a Pi3B+. It's graphically intensive (scrolling could stutter a bit as assets load) but has otherwise been stable.

Try upping the VRAM allocation in EmulationStation, but don't go beyond 256.

If the DRM error reappears, I'll check ps -ef as advised.

Thanks for testing.

For the record, the gpu_mem settings are still needed for the hardware video (mpeg4/mpeg5) decoders in the PI's GPU. If you don't use Kodi or intend to play 4k videos, you can lower the limit or even omit that setting. It would be useful for the 1G model.

[herb_fargus]

Can also confirm similar results when I upped my gpu split to 256

[roslof]

Tested for a bit on a 4GB Pi 4B. More to test, but for now, some results.

Removing cma=384 from cmdline and adding gpu_mem=256 to config.txt actually destabilized the system. Re-adding cma=384 and retaining gpu_mem=256 destabilized even faster. This surprised me with a 4GB Pi. Perhaps only the 1st GB is utilized. Had constant game freezes and ES crashes after launching and exiting 3-5 beefy games (MAME 0.209 & flycast). Not catching anything in logs that would shed light on what led to freezing or crashing. Even the screensaver (set for random video) was freezing in between videos without warning.

So I commented-out the gpu increase and re-added the cma line... Rebooted. Stable again.

In parallel, I increased the ES VRAM Limit to 200Mb (guessed value). This improved the overall ES navigation experience. I'm no longer seeing jumpy emulator system images when scrolling left/right. Change did not yield perceivable negative side effects.

Summarizing that Pi4B 4GB seems VERY stable with:

• No memory split in boot/config.txt
• No change to default GPU in boot/config.txt
• Add cma=384M to boot/cmdline.txt
• ES VRAM increased to 200Mb to support art-intensive theme
  (system may be even MORE stable with this ES change)
• Wholly running in HD (4K forced off in boot/config.txt)

To force HD, and prevent/ rule-out 4K memory drain, I added these lines:

hdmi_group=1
hdmi_mode=16
[hdmi:0]
hdmi_max_pixel_freq=200000000
[hdmi:1]
hdmi_max_pixel_freq=200000000

Now, I'm on hour 3 of launching/exiting games and letting the video screensaver run in between... Launched Kodi and watched a few videos in HD. Stable. Other than a few expected moments of framerate hiccups during intro sequences (eg. Skies of Arcadia), stable.

All of this is anecdotal, of course, so will try to learn more. A few hours of testing isn't enough. But this has been been bugging me for weeks and thrilled I can finally run stable. I'm not technical enough to understand why the cma change (allocating a large contiguous block of memory for GPU) is working, but it seems key here.

Next, I'll try to repro the "OP" description issue and try to learn what is making the gpu starve.

[roslof]

In testing, I found significant memory growth (leak?) in ES. Probably core to my troubles. Basically, I'm able to rapidly and reliably repro a freeze or forced exit of EmulationStation by simply navigating/watching videos in a memory-intensive theme. No game launch is required. I'm not getting the "DRM_IOCTL..." error, rather EmulationStation eventually either freezes or crashes. I would have thought others would have seen this by now, but... maybe I'm more active in ES.

SETUP:

• Tested with both ES v2.10.0RP-DEV and ES v2.9.0RP
• Issue occurs regardless of setting cma=384, or boosting ES RAM
• Install and Set Hursty's "Magazine Madness" theme (although any theme will demonstrate memory bloating over time)
• Ensure videos exist in the various systems and that they play when selecting them
• Use the top command to watch memory (constantly)

SIMPLE TEST

• Enter a system. When a video plays memory dramatically increases. After existing, memory isn't fully restored
• Exit/enter the same system and watch any video (including the same video) and memory will incease, but after existing, won't fully be restored.
• Repeat until VIRT/RES memory fills

RESULT
Eventually, EmulationStation will become unstable and either freeze or exit.

Interesting to note that the Video Screen Saver (displaying random videos) also grows memory constantly.

Running ps axeuf shows this specific command is growing, while all other commands are stable:

\_ /opt/retropie/supplementary/emulationstation-dev/emulationstation MAIL=/var/mail/pi LANGUAGE=en_US.UTF-8 USER=pi TTY=1 XDG_SEAT=seat0 XDG_SESSION_TYPE=tty SHLVL=3 HOME=/home/pi HUSHLOGIN=FALSE DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus LOGNAME=pi _=/opt/retropie/supplementary/emulationstation-dev/emulationstation.sh XDG_SESSION_CLASS=user TERM=linux XDG_SESSION_ID=c1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games XDG_RUNTIME_DIR=/run/user/1000 LANG=en_US.UTF-8 SHELL=/bin/bash XDG_VTNR=1 PWD=/home/pi LC_ALL=en_US.UTF-8

I hope this information is helpful. Happy to provide more information if desired.

[mitu]

Is this issue occuring withe the omxplayer used as video player (the HW accelerated player) or with the built in video player (VLC) ?

[roslof]

@mitu the setting for "Use OMX Player (HW Accelerated)" is off.
Is there any other setting I should be looking at to confirm?

[mitu]

@roslof said in DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory:

Is there any other setting I should be looking at to confirm?

No, that's the only one - On = using omxplayer, Off = using libvlc.

[roslof]

@mitu okay. Will run additional checks with OMX turned on.

[roslof]

That didn't take long... ES memory is completely stable with the OMX Player enabled. omxplayer.bin is releasing memory as expected. Safe to say that libvlc is leaking memory?

[roslof]

I'm a bit too junior to know if this is the same issue:
https://code.videolan.org/videolan/LibVLCSharp/issues/252

[mitu]

@roslof said in DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory:

Safe to say that libvlc is leaking memory?

Hard to say, but my guess it's not libvlc, the error you're reporting comes from the DRM layer, libvlc is used just to decode the video and not to render the video.

[roslof]

I let my system "soak" for 5 hours and it eventually froze on a black screen. I looked into it and learned that the video screensaver has a separate option for using OMX. Since it was using libVLC, memory was maxed in ES -- thus the freeze.

I set the screensaver to use OMX for videos, then started a new soak test. I noted the screensaver videos don't render w/OMX enabled. Perhaps I'm missing something in configuration.

Not clear is this a known issue and if there is a work-around...

[roslof]

Ah... Looks like if OMX is set, and "Show Game Info On Screensaver" is set to anything other than Never, the videos won't render. So for now, I've disabled overlaying game info.

Re-running soak test with OMX enabled...

[mitu]

@roslof on the Pi4, omxplayer can't render the subtitles - it's not supported. Maybe we need that taken care of in EmulationStation.

I've set the EmulationStation screensaver since my last reply, so it's been running for about 7 hours, but there's no crash and the memory has stayed the same (from what top tells me).

I'll see if using the theme and doing the switch as you mention would make EmulationStation crash.

[roslof]

@mitu for me without omx, memory creeps regardless of theme. Even Carbon. Maybe there is something out of date with my setup. Can't imagine what. Will keep looking.