ensuring ES gracefully finish and save metadata in every system shutdown

caver01

@cyperghost That's fantastic! So, you found the definitive answer to the shutdown vs. reboot problem. Way to go!

cyperghost

@caver01 Yes, that's the solution for this problem so far. But don't thank me (directly), it were the people from PIMORONI and theirclean SHIM shutdown daemon here that give me hints how this work. The solution is nested into the setup script ;)

meleu

@cyperghost said in ensuring ES gracefully finish and save metadata in every system shutdown:

Can you link this posting in your first posting please?

Done!

Thanks for sharing. ;)

Semper 5

this script had been working great until about 2 days ago. I noticed it stopped saving my favorites.

i add a game to favorites, power down, then on, and no changes to my list.

I completely wiped the whole card and I'm back to the same nonsense:

metadata save results:
MENU shutdown - will save
MENU reboot system - will save
MENU restart ES - will save

soft shutdown button on NESpi - will NOT save

could somebody help, or explain please?

G30FF

I just modded my NESPi case with the safe shutdown hack by Yahmez (https://retropie.org.uk/forum/topic/12424/retroflag-nespi-case-soft-power-reset-hack). I used the version of his script that turns the Reset button into a "quit emulator" button. I also set up this script exactly as described in the OP, same paths, same files, everything. But ES metadata is not being saved on shutdown or reboot.

The killes script is working fine. If I run killes directly, everything behaves as it should. The problem is something to do with the killes.service systemd unit. Systemctl status reports that the service is loaded, and the Active status is "active (exited)". The killes.service file is exactly as it appears in the OP, and is in the same location. I don't think it has anything to do with the NESPi script mentioned above, as it doesn't save the metadata if I call "sudo shutdown now" from SSH or the terminal either. Any help would be greatly appreciated.

meleu

@semper-5 @G30FF hey guys. I'll try to take a look at it soon. Thanks for the reports.

cyperghost

@meleu Is the line touch /tmp/es-shutdown && chown pi:pi /tmp/es-shutdown still relevant?

meleu

@cyperghost forgive my memory, but where are we using that line?

cyperghost

@meleu https://retropie.org.uk/forum/topic/11750/mausberry-shutdown-script-doesn-t-save-metadata

It was the base code developed in this thread.
But I really don't know if you need this in your script.

It is a fact that if you kill the binary the emulationstation.sh is still active and it waits for a file es-shutdown, es-sysrestart.... whatever ... to do action and to finish ES properly. But I really don't know if it's needed here and if it's the culprit.
I don't think so... I really doubt it.

Maybe the two guys are using an older version of the killes script?
And then updated the runcommand.sh?

Semper 5

@cyperghost I would like to add that, like G30FF mentioned, this is exactly where I got the script from and how I went about putting it into my system with the NESpi. Maybe that's a clue to what you were mentioning we might be running the same old script? Anyways just a thought.

stoo

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

Fitz2380

I too am having the issue of saving Metadata. In particular the favorites and last game played. I have defined a 'system' called Power which appears on my ribbon menu and when I select it, I can do a restart or powerdown. They perform either a shutdown or a shutdown -r command. I implemented the service as defined above and can see that the killes service get started, but neither of my shutdown commands results in the meta data being saved.

Yahmez

For what it's worth, I just made a fresh retropie install for another modded nespi. Installed the scripts for safe shutdown exactly as outlined in the PDF and then installed @meleu killes script as explained in the OP. Everything works fine, metadata is saving.

meleu

@yahmez phew!
phew

Yahmez

@meleu God, I miss that guy.

dankcushions

@stoo said in ensuring ES gracefully finish and save metadata in every system shutdown:

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

most stuff in retropie only saves on exit - game saves will only save to disk on exit of a game. retroarch settings only save on exit of retroarch, etc.

when you're dealing with SD card storage, you generally don't want to do multiple writes; it will kill your card quicker, especially with something as intense as an ES xml write.

stet

I just wanted to put in my thanks for putting this together. I use a RemotePi board to power down my RetroPie setup and this solved the problem of ES not saving the favorites list. :)

meleu

@dankcushions You just gave me an idea for a feature request: an option on the ES menu to Save Current Metadata (something like the "Save Current Configuration" on RetroArch) Then a careful user can do it to prevent loss of data.

My suggestion for such option: in the OPTIONS menu (the one that appears when the users press Select).

@Zigurana @Hex @pjft @jdrassa and other ES hackers, do you guys think it's feasible?

stoo

@dankcushions Hey Dank, thanks for your input!

Before I begin, I'm not 100% sure about the behaviour of ES (and many other things!). I don't actually know if scrapers write the XML files directly at runtime or if they let ES handle it. The following assumes that ES does it:

A change like I suggested wouldn't make much difference in terms of number of writes. If it were up to me, ES would write out metadata on a - let's say 10 mins for argument - schedule unless a scraper is running (no point writing mid-scrape) and only if there are changes to write.

I don't think ES records things like duration spent in a game or number of game launches, so the only data written would be:

Scraped data (which is going to be written on shutdown anyway)
Data the user has explicitly changed e.g. added favourites (which is going to written on shutdown anyway)

In this situation, writing before shutdown and writing at shutdown are the same, the only difference is that we mitigate the case where user metadata is lost when ES is killed before it gets a chance to write it. From a quick glance at the source it appears to me (an almost total non-coder) that ES only writes if there are changes anyway.

(Incidentally, I wrote a tiny change for ScummVM to help scrapers ID ScummVM titles by hash. The PR I submitted had .svm files write only if they didn't exist or were empty. The PR was changed to write them all out every time ScummVM launches. Why? No idea.)

@meleu I had a similar thought :)

Hex

@meleu Cant the MD be saved while exiting from any game? That is the only time MD needs to be updated. The other part would be when MD is manually editted.

Delaying the MD saving can help with minimal writes to the SD card but this problem happens with GPIO shutdown. Would it be more useful to have the script that handles safe shutdown signal ES that the System is going down and it should perform cleanup. Another option would be to have a service that handles the MD saving only and let it handle the clean shutdown.

My recommendation is to update the safe shutdown script to first send sig_term to ES and thus enabling safe storage of MD.