ensuring ES gracefully finish and save metadata in every system shutdown

caver01 · 9 Jan 2018, 22:15

@cyperghost That's fantastic! So, you found the definitive answer to the shutdown vs. reboot problem. Way to go!

cyperghost · 9 Jan 2018, 22:20

@caver01 Yes, that's the solution for this problem so far. But don't thank me (directly), it were the people from PIMORONI and theirclean SHIM shutdown daemon here that give me hints how this work. The solution is nested into the setup script ;)

meleu · 10 Jan 2018, 21:58

@cyperghost said in ensuring ES gracefully finish and save metadata in every system shutdown:

Can you link this posting in your first posting please?

Done!

Thanks for sharing. ;)

Semper 5 · 2 Jan 2018, 21:43

this script had been working great until about 2 days ago. I noticed it stopped saving my favorites.

i add a game to favorites, power down, then on, and no changes to my list.

I completely wiped the whole card and I'm back to the same nonsense:

metadata save results:
MENU shutdown - will save
MENU reboot system - will save
MENU restart ES - will save

soft shutdown button on NESpi - will NOT save

could somebody help, or explain please?

G30FF · 18 Feb 2018, 07:58

I just modded my NESPi case with the safe shutdown hack by Yahmez (https://retropie.org.uk/forum/topic/12424/retroflag-nespi-case-soft-power-reset-hack). I used the version of his script that turns the Reset button into a "quit emulator" button. I also set up this script exactly as described in the OP, same paths, same files, everything. But ES metadata is not being saved on shutdown or reboot.

The killes script is working fine. If I run killes directly, everything behaves as it should. The problem is something to do with the killes.service systemd unit. Systemctl status reports that the service is loaded, and the Active status is "active (exited)". The killes.service file is exactly as it appears in the OP, and is in the same location. I don't think it has anything to do with the NESPi script mentioned above, as it doesn't save the metadata if I call "sudo shutdown now" from SSH or the terminal either. Any help would be greatly appreciated.

meleu · 18 Feb 2018, 12:22

@semper-5 @G30FF hey guys. I'll try to take a look at it soon. Thanks for the reports.

cyperghost · 21 Feb 2018, 19:58

@meleu Is the line touch /tmp/es-shutdown && chown pi:pi /tmp/es-shutdown still relevant?

meleu · 21 Feb 2018, 20:25

@cyperghost forgive my memory, but where are we using that line?

cyperghost · 21 Feb 2018, 20:58

@meleu https://retropie.org.uk/forum/topic/11750/mausberry-shutdown-script-doesn-t-save-metadata

It was the base code developed in this thread.
But I really don't know if you need this in your script.

It is a fact that if you kill the binary the emulationstation.sh is still active and it waits for a file es-shutdown, es-sysrestart.... whatever ... to do action and to finish ES properly. But I really don't know if it's needed here and if it's the culprit.
I don't think so... I really doubt it.

Maybe the two guys are using an older version of the killes script?
And then updated the runcommand.sh?

Semper 5 · 22 Feb 2018, 14:36

@cyperghost I would like to add that, like G30FF mentioned, this is exactly where I got the script from and how I went about putting it into my system with the NESpi. Maybe that's a clue to what you were mentioning we might be running the same old script? Anyways just a thought.

stoo · 22 Feb 2018, 20:17

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

Fitz2380 · 22 Feb 2018, 23:33

I too am having the issue of saving Metadata. In particular the favorites and last game played. I have defined a 'system' called Power which appears on my ribbon menu and when I select it, I can do a restart or powerdown. They perform either a shutdown or a shutdown -r command. I implemented the service as defined above and can see that the killes service get started, but neither of my shutdown commands results in the meta data being saved.

Yahmez · 24 Feb 2018, 03:27

For what it's worth, I just made a fresh retropie install for another modded nespi. Installed the scripts for safe shutdown exactly as outlined in the PDF and then installed @meleu killes script as explained in the OP. Everything works fine, metadata is saving.

meleu · 24 Feb 2018, 09:12

@yahmez phew!

Yahmez · 24 Feb 2018, 13:03

@meleu God, I miss that guy.

dankcushions · 24 Feb 2018, 13:42

@stoo said in ensuring ES gracefully finish and save metadata in every system shutdown:

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

most stuff in retropie only saves on exit - game saves will only save to disk on exit of a game. retroarch settings only save on exit of retroarch, etc.

when you're dealing with SD card storage, you generally don't want to do multiple writes; it will kill your card quicker, especially with something as intense as an ES xml write.

stet · 24 Feb 2018, 16:18

I just wanted to put in my thanks for putting this together. I use a RemotePi board to power down my RetroPie setup and this solved the problem of ES not saving the favorites list. :)

meleu · 24 Feb 2018, 16:22

@dankcushions You just gave me an idea for a feature request: an option on the ES menu to Save Current Metadata (something like the "Save Current Configuration" on RetroArch) Then a careful user can do it to prevent loss of data.

My suggestion for such option: in the OPTIONS menu (the one that appears when the users press Select).

@Zigurana @Hex @pjft @jdrassa and other ES hackers, do you guys think it's feasible?

stoo · 24 Feb 2018, 16:37

@dankcushions Hey Dank, thanks for your input!

Before I begin, I'm not 100% sure about the behaviour of ES (and many other things!). I don't actually know if scrapers write the XML files directly at runtime or if they let ES handle it. The following assumes that ES does it:

A change like I suggested wouldn't make much difference in terms of number of writes. If it were up to me, ES would write out metadata on a - let's say 10 mins for argument - schedule unless a scraper is running (no point writing mid-scrape) and only if there are changes to write.

I don't think ES records things like duration spent in a game or number of game launches, so the only data written would be:

Scraped data (which is going to be written on shutdown anyway)
Data the user has explicitly changed e.g. added favourites (which is going to written on shutdown anyway)

In this situation, writing before shutdown and writing at shutdown are the same, the only difference is that we mitigate the case where user metadata is lost when ES is killed before it gets a chance to write it. From a quick glance at the source it appears to me (an almost total non-coder) that ES only writes if there are changes anyway.

(Incidentally, I wrote a tiny change for ScummVM to help scrapers ID ScummVM titles by hash. The PR I submitted had .svm files write only if they didn't exist or were empty. The PR was changed to write them all out every time ScummVM launches. Why? No idea.)

@meleu I had a similar thought :)

Hex · 25 Feb 2018, 02:30

@meleu Cant the MD be saved while exiting from any game? That is the only time MD needs to be updated. The other part would be when MD is manually editted.

Delaying the MD saving can help with minimal writes to the SD card but this problem happens with GPIO shutdown. Would it be more useful to have the script that handles safe shutdown signal ES that the System is going down and it should perform cleanup. Another option would be to have a service that handles the MD saving only and let it handle the clean shutdown.

My recommendation is to update the safe shutdown script to first send sig_term to ES and thus enabling safe storage of MD.