ensuring ES gracefully finish and save metadata in every system shutdown

meleu

@semper-5 @G30FF hey guys. I'll try to take a look at it soon. Thanks for the reports.

cyperghost

@meleu Is the line touch /tmp/es-shutdown && chown pi:pi /tmp/es-shutdown still relevant?

meleu

@cyperghost forgive my memory, but where are we using that line?

cyperghost

@meleu https://retropie.org.uk/forum/topic/11750/mausberry-shutdown-script-doesn-t-save-metadata

It was the base code developed in this thread.
But I really don't know if you need this in your script.

It is a fact that if you kill the binary the emulationstation.sh is still active and it waits for a file es-shutdown, es-sysrestart.... whatever ... to do action and to finish ES properly. But I really don't know if it's needed here and if it's the culprit.
I don't think so... I really doubt it.

Maybe the two guys are using an older version of the killes script?
And then updated the runcommand.sh?

Semper 5

@cyperghost I would like to add that, like G30FF mentioned, this is exactly where I got the script from and how I went about putting it into my system with the NESpi. Maybe that's a clue to what you were mentioning we might be running the same old script? Anyways just a thought.

stoo

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

Fitz2380

I too am having the issue of saving Metadata. In particular the favorites and last game played. I have defined a 'system' called Power which appears on my ribbon menu and when I select it, I can do a restart or powerdown. They perform either a shutdown or a shutdown -r command. I implemented the service as defined above and can see that the killes service get started, but neither of my shutdown commands results in the meta data being saved.

Yahmez

For what it's worth, I just made a fresh retropie install for another modded nespi. Installed the scripts for safe shutdown exactly as outlined in the PDF and then installed @meleu killes script as explained in the OP. Everything works fine, metadata is saving.

meleu

@yahmez phew!
phew

Yahmez

@meleu God, I miss that guy.

dankcushions

@stoo said in ensuring ES gracefully finish and save metadata in every system shutdown:

Does ES only save metadata on shutdown?

Wouldn't it make sense to just flush metadata to storage every X minutes if there are changes? At least then you only lose very recent changes if ES goes down "unclean".

most stuff in retropie only saves on exit - game saves will only save to disk on exit of a game. retroarch settings only save on exit of retroarch, etc.

when you're dealing with SD card storage, you generally don't want to do multiple writes; it will kill your card quicker, especially with something as intense as an ES xml write.

stet

I just wanted to put in my thanks for putting this together. I use a RemotePi board to power down my RetroPie setup and this solved the problem of ES not saving the favorites list. :)

meleu

@dankcushions You just gave me an idea for a feature request: an option on the ES menu to Save Current Metadata (something like the "Save Current Configuration" on RetroArch) Then a careful user can do it to prevent loss of data.

My suggestion for such option: in the OPTIONS menu (the one that appears when the users press Select).

@Zigurana @Hex @pjft @jdrassa and other ES hackers, do you guys think it's feasible?

stoo

@dankcushions Hey Dank, thanks for your input!

Before I begin, I'm not 100% sure about the behaviour of ES (and many other things!). I don't actually know if scrapers write the XML files directly at runtime or if they let ES handle it. The following assumes that ES does it:

A change like I suggested wouldn't make much difference in terms of number of writes. If it were up to me, ES would write out metadata on a - let's say 10 mins for argument - schedule unless a scraper is running (no point writing mid-scrape) and only if there are changes to write.

I don't think ES records things like duration spent in a game or number of game launches, so the only data written would be:

Scraped data (which is going to be written on shutdown anyway)
Data the user has explicitly changed e.g. added favourites (which is going to written on shutdown anyway)

In this situation, writing before shutdown and writing at shutdown are the same, the only difference is that we mitigate the case where user metadata is lost when ES is killed before it gets a chance to write it. From a quick glance at the source it appears to me (an almost total non-coder) that ES only writes if there are changes anyway.

(Incidentally, I wrote a tiny change for ScummVM to help scrapers ID ScummVM titles by hash. The PR I submitted had .svm files write only if they didn't exist or were empty. The PR was changed to write them all out every time ScummVM launches. Why? No idea.)

@meleu I had a similar thought :)

Hex

@meleu Cant the MD be saved while exiting from any game? That is the only time MD needs to be updated. The other part would be when MD is manually editted.

Delaying the MD saving can help with minimal writes to the SD card but this problem happens with GPIO shutdown. Would it be more useful to have the script that handles safe shutdown signal ES that the System is going down and it should perform cleanup. Another option would be to have a service that handles the MD saving only and let it handle the clean shutdown.

My recommendation is to update the safe shutdown script to first send sig_term to ES and thus enabling safe storage of MD.

meleu

@hex said in ensuring ES gracefully finish and save metadata in every system shutdown:

Would it be more useful to have the script that handles safe shutdown signal ES that the System is going down and it should perform cleanup. Another option would be to have a service that handles the MD saving only and let it handle the clean shutdown.

eh... pretty much what this whole thread is all about. :D

G30FF

I have some new information for you.

After doing a bit of research, I tried to test out the killes service. If I SSH into a terminal session and run "sudo systemctl stop killes" while the killes service is running, EmulationStation will quit. The next time I start up EmulationStation, the metadata was saved correctly.

So the service does work, and IS invoking the shell script to kill EmulationStation when the service stops. However, it seems that the service is stopping AFTER EmulationStation is already killed. For reference, I updated my RetroPie install along with the kernel tonight to see if it would help.

dankcushions

@meleu said in ensuring ES gracefully finish and save metadata in every system shutdown:

@dankcushions You just gave me an idea for a feature request: an option on the ES menu to Save Current Metadata (something like the "Save Current Configuration" on RetroArch) Then a careful user can do it to prevent loss of data.

My suggestion for such option: in the OPTIONS menu (the one that appears when the users press Select).

@Zigurana @Hex @pjft @jdrassa and other ES hackers, do you guys think it's feasible?

doesn't restarting emulationstation accomplish the same goal? seems like a redundant option to me.

cyperghost

@dankcushions
I totally agree, most of these "errors" seems to be somehow user made. The service should work and if somehow the power is cut then it's a hardware error and not software.

@meleu
The only thing that can be improved imho is the emulator detection routine. The detection of ES PID is done 100% and I think there is no more improvement to do in ES itself. The ES restart via options menu is exactly the feature you mentioned.

mediamogul

@cyperghost said in ensuring ES gracefully finish and save metadata in every system shutdown:

The detection of ES PID is done 100%

This is as good a time as any to thank you guys for putting this together. I dual boot between RetroPie and OSMC and a while back I used part of this to create a script that launches from the ES Kodi system menu that allows ES to shut down and save all MD properly before rebooting into OSMC. This was a very welcome addition after the update to 'favorites', 'last played', etc. Very keen and thanks again.