EmulationStation mod

[ben_thatmustbeme]

@robertybob given that RC images are out, I don't think that's likely, but I don't know the release procedure. I think it's more likely to get a script to change ES versions

[meleu]

Just for curiosity:
Could changing from xml to another database avoid/minimize this problem?
(exit emulationstation take ages when "save metadata on exit" is on)

[jacobfk20]

@meleu Probably. Wouldn't have to dump metadata to a file on exit anymore. A db would keep everything up to date as it is modified.

[meleu]

@jacobfk20
Then you guys have here one more vote to go ahead with this changing... :-)

[jacobfk20]

@meleu It would be interesting to test that out.

[herb_fargus]

@jacobfk20 there are a couple things to consider on that front, in the past a bit of testing has been done and response times are somewhat comparable and it seemed that the only real benefit for a SQL backend came for the rom hoarders who have like 20,000 games. Im also not sure if gamelists/scrapes are included in that or still stay as separate xmls? Anyways there is more testing to be done on that front and would be interesting to see the results.

The other semi-related thing is the white screen of death issue that is prolific with any themes that are 1080p with unique wallpapers. If there were a way to load each system dynamically kinda like you've tried to do with the grid boxart that would open up more variety of themes but I don't know how extensive code changes for that would be

[jacobfk20]

@herb_fargus I'm glad to see that it has been attempted. I need to take a closer look at metadata sometime still. GridView's dynamic loading will get better and I will see what happens when I apply something similar to system view.

[Arcuza]

How much resources do SQLite/MariaDB consume? Memory, CPU, Power? Should the daemon be kept running in the background when firing up an emulator, or should it be started, shut down, started, shut down?

DBs are mainly for very large datasets (some of them have 100 different algorithms for sorting data fast, depending on how you want the data sorted). We don't need that?

DBs are built with transactional integrity in mind. We don't need that?

DBs are built for multiple concurrent connections. We don't need that?

DBs are built for network access, with security, access control, logins and such. We don't need that?

When I develop on Windows .Net has really great features for dealing with XML and datasets. I vote for finding such a library to work with if you want to load static data, change something and write it back.

DBs consume too much resources because they are build for very heavy and complicated situations.

Writing 1 MB of XML data from a dataset should take no more than 10 ms. Just like any other 1 MB file.

[jacobfk20]

I took a quick look at Metadata and I love how it is formatted. I see that on exit MetaData will append everything to xml regardless if it has been modified or not. It could be made to where metadata only saves to file when something has been modified, that would be a simple approach and it could make it to where you don't have to go re-enable save meta data in the menu when you want to save a change.

It could even keep an array of indexes and then only modify those nodes on the xml doc on exit but that may not actually speed it up because it will now have to search the xml nodes for the correct child to change. I don't know fast the pugixml library would do that.

I actually want to know exact time it takes for someone with long exit times. How big are their game lists?

I don't know how much SQLite uses in resources as I've never used it in anything. If it has been looked into before and it has shown to not have any real improvements then maybe time should be spent looking for a different solution than going to a sql db.

[Arcuza]

@jacobfk20 Often, you load the XML into memory as a tree of data. You do search against that tree with XPath or other methods. Algorithms for doing so are very optimized. Changing data is just like changing a value for any variable. When writing to disk, you write the full file with XML-serialized data tree. If you do not need table locking, concurrent users, security, joining multiple tables, group by, then XML is fast and small footprint (Or the developed applications and frameworks dealing with data trees). In .Net on Windows you can load XML into memory and query that dataset with Linq, more extensively than SQL.

[meleu]

@jacobfk20 said in EmulationStation mod:

@meleu Probably. Wouldn't have to dump metadata to a file on exit anymore. A db would keep everything up to date as it is modified.

Can't this updating method be done with xml?

@jacobfk20 said in EmulationStation mod:

I actually want to know exact time it takes for someone with long exit times. How big are their game lists?

I timed it here: exactly 4 minutes and 58 seconds. But we can say 5 minutes.

game list size:
Atari2600: 1443
FBA: 1
GameBoy Advance: 10
Master System: 545
Mega Drive: 1239
NeoGeo: 7
NES: 2141
PC Engine: 2
SNES: 2271

[Arcuza]

@meleu something's wrong, how can we troubleshoot? XML is just like any text file with a strict defined way of representing tree data. Databases are by no means faster... They do solve some particular problems, but that comes with a price.

[jacobfk20]

@meleu thank you for doing that test. How about we just save gamelist/metadata on systems that have been flagged with data that has been modified? Like if you only change something in Master System then it will only take 30ish seconds to shut down. Or if you change nothing it'll shut down in normal time.

@Arcuza I'm not fully sure, but it looks like MetaData creates a complete new node tree from scratch every time it saves. I could be missing something though.

[herb_fargus]

@jacobfk20 just for reference someone added a PR this morning to address this issue somewhat. Doesn't solve it completely but should make it faster, I still need to test it.

https://github.com/RetroPie/EmulationStation/pull/59

[meleu]

@Arcuza
I'm not the first to have this issue. It's in the RetroPie's FAQ: https://github.com/RetroPie/RetroPie-Setup/wiki/FAQ#why-does-shut-down-and-reboot-take-ages

If I understood right, it happens because emulationstation dumps all the metadata (no matter if some change was made or not) to a file on exit. When I got these 5 minutes, I just turned on my raspi2, waited for emulationstation start, and then "exit emulationstation". I didn't change anything and it took 5 long minutes to exit.

Well, I'm not an DB approach advocate, but I see some emulationstation hackers tending to it. As @jacobfk20 said: "A db would keep everything up to date as it is modified." And I saw @Zigurana arguing in favor of a DB approach in other posts [edit: not exactly, as he says below].

This discussion reminded me an old MAME frontend I used years ago on Windows. They had several options of filtering the game list (no clones, beat'em'up, fighting, 2 players, 4 players, and many many others). I think they use a db for it.

[meleu]

@herb_fargus

https://github.com/RetroPie/EmulationStation/pull/59

Installed and tested it here. Same 5 minutes to exit emulationstation when "save metadata on exit" is on.

If I understood it right, this PR reduces the loading time when "parse gamelist only" is on.

[BuZz]

Yes (and an optimisation in how the games are stored). It is unrelated to saving meta data.

[herb_fargus]

@BuZz thanks for clarifying

[Rookervik]

Ooh, themes need variables. So you can set a color once and have it apply to many sections of the theme. :D

[Zigurana]

Hi @Arcuza and @jacobfk20 !
It's nice to have some people who actually know their stuff joining the xml vs db discussion, I am sure I'll learn a lot from you!
While I've argued for a better (db) implementation in other places, it really is a means to an end, and maybe what I (we?) want can also be achieved using XMLs. I think the big benefit of XML files is that they are human-readable and easily edited using any text editor, and that is something that greatly facilitates tinkering (the main RPi objective if you ask me :-) ).

As I see it, there are two sides to this: functionality and performance.

Functionality
@Arcuza, you are right we do not need many of the features that a full database implementation would bring. Multiple users, security layers and network access are probably not necessary.
However, what we are sorely lacking is a way to build queries to create lists of game and perform sorting on all available metadata (across platforms if need be). Also, we need some way to add user-defined tags to allow arbitrary collections and filters. In the current implementation of ES XML trees, none of this is possible, and I am hoping/expecting that a db approach would make this possible.

Performance
The xml implementation in ES is based on pugiXML, which should support XPath for queries, but I have no idea how these compare to SQL (or other db) based queries. I've seen mention uptread of a comparison of performance of the db branch of ES with the regular one, and its performance, but I have never seen the results (or reports of this) first-hand.
I am of the opinion that the current problems in XML performance are most likely due to the implementation of pugiXML or its limitations.
The dumping of all metadata is only because of the updating of two metadata fields: "# times played" and "time since last play". These two were added some time after the first introduction of ES, and could/should be made optional, in order to reduce exit times.
None of this solves the issue of long startup times that have sometimes been reported. Maybe the PR that @herb_fargus linked to above will help iwith this issue.

Again, I am not advocating for a db approach or xml, I just want the possibility to generate lists across platforms, and apply arbitrary filters. If this can be achieved with XML, I'm all for it.