EmulationStation mod

Arcuza

How much resources do SQLite/MariaDB consume? Memory, CPU, Power? Should the daemon be kept running in the background when firing up an emulator, or should it be started, shut down, started, shut down?

DBs are mainly for very large datasets (some of them have 100 different algorithms for sorting data fast, depending on how you want the data sorted). We don't need that?

DBs are built with transactional integrity in mind. We don't need that?

DBs are built for multiple concurrent connections. We don't need that?

DBs are built for network access, with security, access control, logins and such. We don't need that?

When I develop on Windows .Net has really great features for dealing with XML and datasets. I vote for finding such a library to work with if you want to load static data, change something and write it back.

DBs consume too much resources because they are build for very heavy and complicated situations.

Writing 1 MB of XML data from a dataset should take no more than 10 ms. Just like any other 1 MB file.

jacobfk20

I took a quick look at Metadata and I love how it is formatted. I see that on exit MetaData will append everything to xml regardless if it has been modified or not. It could be made to where metadata only saves to file when something has been modified, that would be a simple approach and it could make it to where you don't have to go re-enable save meta data in the menu when you want to save a change.

It could even keep an array of indexes and then only modify those nodes on the xml doc on exit but that may not actually speed it up because it will now have to search the xml nodes for the correct child to change. I don't know fast the pugixml library would do that.

I actually want to know exact time it takes for someone with long exit times. How big are their game lists?

I don't know how much SQLite uses in resources as I've never used it in anything. If it has been looked into before and it has shown to not have any real improvements then maybe time should be spent looking for a different solution than going to a sql db.

Arcuza

@jacobfk20 Often, you load the XML into memory as a tree of data. You do search against that tree with XPath or other methods. Algorithms for doing so are very optimized. Changing data is just like changing a value for any variable. When writing to disk, you write the full file with XML-serialized data tree. If you do not need table locking, concurrent users, security, joining multiple tables, group by, then XML is fast and small footprint (Or the developed applications and frameworks dealing with data trees). In .Net on Windows you can load XML into memory and query that dataset with Linq, more extensively than SQL.

meleu

@jacobfk20 said in EmulationStation mod:

@meleu Probably. Wouldn't have to dump metadata to a file on exit anymore. A db would keep everything up to date as it is modified.

Can't this updating method be done with xml?

@jacobfk20 said in EmulationStation mod:

I actually want to know exact time it takes for someone with long exit times. How big are their game lists?

I timed it here: exactly 4 minutes and 58 seconds. But we can say 5 minutes.

game list size:
Atari2600: 1443
FBA: 1
GameBoy Advance: 10
Master System: 545
Mega Drive: 1239
NeoGeo: 7
NES: 2141
PC Engine: 2
SNES: 2271

Arcuza

@meleu something's wrong, how can we troubleshoot? XML is just like any text file with a strict defined way of representing tree data. Databases are by no means faster... They do solve some particular problems, but that comes with a price.

jacobfk20

@meleu thank you for doing that test. How about we just save gamelist/metadata on systems that have been flagged with data that has been modified? Like if you only change something in Master System then it will only take 30ish seconds to shut down. Or if you change nothing it'll shut down in normal time.

@Arcuza I'm not fully sure, but it looks like MetaData creates a complete new node tree from scratch every time it saves. I could be missing something though.

herb_fargus

@jacobfk20 just for reference someone added a PR this morning to address this issue somewhat. Doesn't solve it completely but should make it faster, I still need to test it.

https://github.com/RetroPie/EmulationStation/pull/59

meleu

@Arcuza
I'm not the first to have this issue. It's in the RetroPie's FAQ: https://github.com/RetroPie/RetroPie-Setup/wiki/FAQ#why-does-shut-down-and-reboot-take-ages

If I understood right, it happens because emulationstation dumps all the metadata (no matter if some change was made or not) to a file on exit. When I got these 5 minutes, I just turned on my raspi2, waited for emulationstation start, and then "exit emulationstation". I didn't change anything and it took 5 long minutes to exit.

Well, I'm not an DB approach advocate, but I see some emulationstation hackers tending to it. As @jacobfk20 said: "A db would keep everything up to date as it is modified." And I saw @Zigurana arguing in favor of a DB approach in other posts [edit: not exactly, as he says below].

This discussion reminded me an old MAME frontend I used years ago on Windows. They had several options of filtering the game list (no clones, beat'em'up, fighting, 2 players, 4 players, and many many others). I think they use a db for it.

meleu

@herb_fargus

https://github.com/RetroPie/EmulationStation/pull/59

Installed and tested it here. Same 5 minutes to exit emulationstation when "save metadata on exit" is on.

If I understood it right, this PR reduces the loading time when "parse gamelist only" is on.

BuZz

Yes (and an optimisation in how the games are stored). It is unrelated to saving meta data.

herb_fargus

@BuZz thanks for clarifying

Rookervik

Ooh, themes need variables. So you can set a color once and have it apply to many sections of the theme. :D

Zigurana

Hi @Arcuza and @jacobfk20 !
It's nice to have some people who actually know their stuff joining the xml vs db discussion, I am sure I'll learn a lot from you!
While I've argued for a better (db) implementation in other places, it really is a means to an end, and maybe what I (we?) want can also be achieved using XMLs. I think the big benefit of XML files is that they are human-readable and easily edited using any text editor, and that is something that greatly facilitates tinkering (the main RPi objective if you ask me :-) ).

As I see it, there are two sides to this: functionality and performance.

Functionality
@Arcuza, you are right we do not need many of the features that a full database implementation would bring. Multiple users, security layers and network access are probably not necessary.
However, what we are sorely lacking is a way to build queries to create lists of game and perform sorting on all available metadata (across platforms if need be). Also, we need some way to add user-defined tags to allow arbitrary collections and filters. In the current implementation of ES XML trees, none of this is possible, and I am hoping/expecting that a db approach would make this possible.

Performance
The xml implementation in ES is based on pugiXML, which should support XPath for queries, but I have no idea how these compare to SQL (or other db) based queries. I've seen mention uptread of a comparison of performance of the db branch of ES with the regular one, and its performance, but I have never seen the results (or reports of this) first-hand.
I am of the opinion that the current problems in XML performance are most likely due to the implementation of pugiXML or its limitations.
The dumping of all metadata is only because of the updating of two metadata fields: "# times played" and "time since last play". These two were added some time after the first introduction of ES, and could/should be made optional, in order to reduce exit times.
None of this solves the issue of long startup times that have sometimes been reported. Maybe the PR that @herb_fargus linked to above will help iwith this issue.

Again, I am not advocating for a db approach or xml, I just want the possibility to generate lists across platforms, and apply arbitrary filters. If this can be achieved with XML, I'm all for it.

gomisensei

Here is my wishlist:

Multiple Images in detailed view, possibly by pressing left/right trigger, switching between however many images there are, so, <image1><image2>, or even just multiple <image> items.

Adding Total Time played along with Last played, and times played. These (along with maybe a record of which image was showing) could be saved in an alternative xml file (Stats.xml) on exit, so the other metadata wouldn't need to all be saved on every exit if it wasn't changed.

thank you for taking the time to work on this for us.

meleu

@Zigurana sorry for that misunderstanding about your position.

Your post clarify that the problem is (can be) in the way that ES manages XML, and not in the XML itself.

I read what @Arcuza said and googled about XML vs DB, and it seems (at least in theory) that XML is good enough for this application (ES). But I would like to compare it in practice.

Do you guys know any ES fork with a DB approach instead of XML? I would like to make some tests.

Zigurana

@meleu said in EmulationStation mod:

@Zigurana sorry for that misunderstanding about your position.

Do you guys know any ES fork with a DB approach instead of XML? I would like to make some tests.

No worries :-)
The fork I meant is this one: https://github.com/Aloshi/EmulationStation/tree/gamelistdb, although I am not quite sure it's ready to use as is. It could be that it tries to convert/import your existing gamelist xmls into the database upon every startup. I never took the time to fully understand it.

jacobfk20

I put together a simple quick branch that will only save metadata if that system has had any changes to its metadata. So if you were to change something in SNES it will only save SNES's gamelist. Changing metadata in Gamelist Options or scraping for that system (fail or succeed) will trigger that system to save.

https://github.com/jacobfk20/EmulationStation/tree/Metadata

This will not save times played. Ever. Will probably make "Save MetaData on Exit" be a list where you can pick Save/Don't Save/Save On Change Only.

Hopefully this will make it slightly more bearable on ES exit for those with huge gamelists, like @meleu . I may try something more complex soon but not right now.

Please backup all gamelists if you try it. The chance of this destroying them is pretty much none-- but you never know.

@Rookervik Like create a pallet? such as <pallet><primary>00AAFF</primary><secondary>AA8FDD</secondary></pallet> Or something similar?

Rookervik

@jacobfk20 Uh, I don't understand that. But yes it would be used for a palette. So a themer could put in say "themecolor"=ff0000 and then in the theme you could go to the <color> of an item and instead of having to put <color>ff0000</color> you could do <color>themecolor</color> That way if you make a color change, any item with "themecolor" in it would change as well. Would even be more amazing if you could change color inside ES but that's TOTALLY dreaming. Don't bother with anything like that right now.

Or even let the variable act as text in general. You could do anything with it. "boxheight"=440 and then you could use boxheight instead of the number 440.

mattrixk

If anyone is considering the Tag idea, I had a thought about how we could use the the X and Y buttons as shortcut keys.

@Zigurana's Child-Friendly ES uses the X and Y buttons as shortcut keys for favourite and kid-friendly, and it got me thinking if there is a way (once the tagging system is working) to specify what the buttons do within ES.

Say you've created a few tags:

favourites
2_player_brawler
games_to_play
best_vert_shmup

You could then go into Tag Settings and choose which Tag to attach to the 2 spare shortcut buttons (X and Y). This could potentially save a bunch of time, and also make it so those 2 buttons can be used for more than just 2 set things.

simbz23

@jacobfk20 Great job on this new ES! Makes using many of the internal functions much easier!

Only issue I am having is the available wifi networks aren't showing up when I select the "connect to wifi" option in the network settings. All it shows is "Available Networks" and the back button. I can see my current info for the wifi I'm connected to, and can turn on and off the wifi.

I did see this error in the terminal when I exited out of emulationstation:

wlan0 interface doesn't support scanning: Device or resource busy

I put the wificonnect file in /home/pi/.emulationstation/app/wifi/

Right location? :-/

4.0 RC1
pi3
Logitech F310