ensuring ES gracefully finish and save metadata in every system shutdown

dankcushions

@stoo said in ensuring ES gracefully finish and save metadata in every system shutdown:

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

most stuff in retropie only saves on exit - game saves will only save to disk on exit of a game. retroarch settings only save on exit of retroarch, etc.

when you're dealing with SD card storage, you generally don't want to do multiple writes; it will kill your card quicker, especially with something as intense as an ES xml write.

stet

I just wanted to put in my thanks for putting this together. I use a RemotePi board to power down my RetroPie setup and this solved the problem of ES not saving the favorites list. :)

meleu

@dankcushions You just gave me an idea for a feature request: an option on the ES menu to Save Current Metadata (something like the "Save Current Configuration" on RetroArch) Then a careful user can do it to prevent loss of data.

My suggestion for such option: in the OPTIONS menu (the one that appears when the users press Select).

@Zigurana @Hex @pjft @jdrassa and other ES hackers, do you guys think it's feasible?

stoo

@dankcushions Hey Dank, thanks for your input!

Before I begin, I'm not 100% sure about the behaviour of ES (and many other things!). I don't actually know if scrapers write the XML files directly at runtime or if they let ES handle it. The following assumes that ES does it:

A change like I suggested wouldn't make much difference in terms of number of writes. If it were up to me, ES would write out metadata on a - let's say 10 mins for argument - schedule unless a scraper is running (no point writing mid-scrape) and only if there are changes to write.

I don't think ES records things like duration spent in a game or number of game launches, so the only data written would be:

Scraped data (which is going to be written on shutdown anyway)
Data the user has explicitly changed e.g. added favourites (which is going to written on shutdown anyway)

In this situation, writing before shutdown and writing at shutdown are the same, the only difference is that we mitigate the case where user metadata is lost when ES is killed before it gets a chance to write it. From a quick glance at the source it appears to me (an almost total non-coder) that ES only writes if there are changes anyway.

(Incidentally, I wrote a tiny change for ScummVM to help scrapers ID ScummVM titles by hash. The PR I submitted had .svm files write only if they didn't exist or were empty. The PR was changed to write them all out every time ScummVM launches. Why? No idea.)

@meleu I had a similar thought :)

Hex

@meleu Cant the MD be saved while exiting from any game? That is the only time MD needs to be updated. The other part would be when MD is manually editted.

Delaying the MD saving can help with minimal writes to the SD card but this problem happens with GPIO shutdown. Would it be more useful to have the script that handles safe shutdown signal ES that the System is going down and it should perform cleanup. Another option would be to have a service that handles the MD saving only and let it handle the clean shutdown.

My recommendation is to update the safe shutdown script to first send sig_term to ES and thus enabling safe storage of MD.

meleu

@hex said in ensuring ES gracefully finish and save metadata in every system shutdown:

Would it be more useful to have the script that handles safe shutdown signal ES that the System is going down and it should perform cleanup. Another option would be to have a service that handles the MD saving only and let it handle the clean shutdown.

eh... pretty much what this whole thread is all about. :D

G30FF

I have some new information for you.

After doing a bit of research, I tried to test out the killes service. If I SSH into a terminal session and run "sudo systemctl stop killes" while the killes service is running, EmulationStation will quit. The next time I start up EmulationStation, the metadata was saved correctly.

So the service does work, and IS invoking the shell script to kill EmulationStation when the service stops. However, it seems that the service is stopping AFTER EmulationStation is already killed. For reference, I updated my RetroPie install along with the kernel tonight to see if it would help.

dankcushions

@meleu said in ensuring ES gracefully finish and save metadata in every system shutdown:

@dankcushions You just gave me an idea for a feature request: an option on the ES menu to Save Current Metadata (something like the "Save Current Configuration" on RetroArch) Then a careful user can do it to prevent loss of data.

My suggestion for such option: in the OPTIONS menu (the one that appears when the users press Select).

@Zigurana @Hex @pjft @jdrassa and other ES hackers, do you guys think it's feasible?

doesn't restarting emulationstation accomplish the same goal? seems like a redundant option to me.

cyperghost

@dankcushions
I totally agree, most of these "errors" seems to be somehow user made. The service should work and if somehow the power is cut then it's a hardware error and not software.

@meleu
The only thing that can be improved imho is the emulator detection routine. The detection of ES PID is done 100% and I think there is no more improvement to do in ES itself. The ES restart via options menu is exactly the feature you mentioned.

mediamogul

@cyperghost said in ensuring ES gracefully finish and save metadata in every system shutdown:

The detection of ES PID is done 100%

This is as good a time as any to thank you guys for putting this together. I dual boot between RetroPie and OSMC and a while back I used part of this to create a script that launches from the ES Kodi system menu that allows ES to shut down and save all MD properly before rebooting into OSMC. This was a very welcome addition after the update to 'favorites', 'last played', etc. Very keen and thanks again.

caver01

@mediamogul quick side note here if I may: what were the broad steps you followed to build your OSMC dual boot? I have Kodi setup, but I really do prefer the OSMC flavor. Always have.

mediamogul

@caver01

NOOBS was used for the setup I'm currently using. Since both sides will need to be redone for Stretch, I plan to use PINN to replace it in the months to come, as it offers RetroPie as one of it's default install options. As you may be aware, using NOOBS for this involves converting a RetroPie SD image to be used. After that, it's just a matter of booting it up and selecting RetroPie to install alongside the already available OSMC option.

When it's all said and done, you either have the option of using a dual boot selection screen every time the Pi starts, or you can take a page out of the Multi Boot Pi book like I do and use a custom addon for Kodi that launches RetroPie, along with a custom script in RetroPie to return to OSMC. All that said, besides mentioning PINN as an option, I'm probably burying the lead in that you could just get an install image from the Multiboot Pi page with OSMC and RetroPie already configured for this. While it will install older versions of both, they can of course be updated independently as they would normally.

Fitz2380

@meleu I am still not able to get the metadata to save from my process. Is there something I could look at to help troubleshoot the issue.

I have verified that the killes.service is enabled and therefore running.

I can execute the killes.sh script from the command line and emulation station processes do terminate. After I power down and backup, the metadata has been saved.

My issue is when I execute the 'shutdown' command from within emulation station as a menu item. The command performed is the 'shutdown now' command, but the meta data is never saved when I execute it this way.

Any thoughts that would help me debug would be great.

Semper 5

Any updates on the MD not saving?

I've made no progress. MD only saves when performing shutdowns or restarts within ES menu....not button :/

Fitz2380

I just wanted to update my previous post as I did not include the proper information before. I am running Retropie v4.3, RPi3 Model B, 5v @2.5A PSU, 32GB SandDisk Microsd.

I have installed the killes service and installed the killes.sh script. I can successfully save metadata by running the script from the command line. I cannot save metadata if I run a script that does the 'shutdown now' command which is executed from the main retropie menu as was posted here: https://retropie.org.uk/forum/topic/10736/power-on-off-option-in-main-menu/3

Additionally how do you put your system information into a footer in your post

Bananu

@yahmez

Hi! I´m on your previous situation: Modded Nespi Case soft shutdown, following PDF (working GREAT) and after some iso backups, I installed the killes.sh, but is not working... adding some favs and pressing power down button, and after power it up, favs didn´t save.

I can see killes.service enabled from :~ $ systemctl status killes
● killes.service - Kill EmulationStation
Loaded: loaded (/etc/systemd/system/killes.service; enabled)
Active: active (exited) since Sat 2018-03-24 10:05:18 UTC; 3min 46s ago

Is the fresh install your only way to go? For me, it would be a nightmare re-install 30gb of roms and ports, gamepad config, overclock... I´m making backups every now and then just to not repeat what it´s done (like 6 months of work here)

So... how can I emulate the "fresh install" on my scripts? is something I should wipe from systemctl or something?

Thank you!

Yahmez

@bananu
Sorry man, I really don't know what is causing this issue for people. Especially because the killes script worked for me both on a system I had already set up completely, and with a fresh install. Wish I could be more of a help here...

julenvitoria

@meleu HI! I'm trying to make the gracefully finish to work but I can't and I don't know why... I follow all the steps even create the script in the same path than you (/etc) but it does not work. I'm using a raspberry pi 3 with the last version of retropie and shutdown script with a button connected to GPIO 3. Could you guide me about this?
Thanks

Quackwalks

When using this with the Kintaro 9000 case, the power switch doesn't respond but the reset switch does. Using the reset switch won't save metadata. If anyone has experience with this, I'd love some help.

Pi Model or other hardware: Pi 3b
Power Supply used: 5.1v 2.5a
RetroPie Version Used: 4.3.17
Built From: Pre made SD Image on RetroPie website, updated to 4.3.17
USB Devices connected: dongle for wireless keyboard
Controller used: Official PS3 Controller
How to replicate the problem: Followed install instructions from Kintaro 9000, then followed instructions in this post

meleu

Hello fellows,

I've just updated the OP with the following warning:

UPDATE: many guys are reporting that this trick isn't working anymore. Maybe something has changed after an OS update, but I'm not sure. I don't use GPIO buttons to shutdown my RetroPie myself.

I'm currently very busy on other projects and unable to provide support here. Sorry mates..