Metadata Improvements in EmulationStation

fieldofcows

@mediamogul said in Metadata Improvements in EmulationStation:

Would the new set of filters you mention be able to replace this type of use case scenario?

Maybe not.

I spent some time last night looking at how to build folder support into my metadata model and I think I've cracked it so it looks like folders should be back on the feature list.

mediamogul

@fieldofcows

That's fantastic! I really appreciate all of the time you've put into all this. Emulation Station is really on the move and I look forward to seeing it all come together.

Zigurana

@fieldofcows said in Metadata Improvements in EmulationStation:

...
I had a look at the 'sqlite' changes made (I think) by Aloshi and decided I wanted to go a stage further so I'm currently busy implementing a new set of C++ classes that manage the game list and metadata using sqlite over multiple tables. I'm not a database software engineer so it may not be perfect but should hopefully be an improvement over what's there now.

I can see a number of benefits once this is working:

Very fast startup and shutdown of ES

Metadata can be managed separately to the installed game list meaning (for example) the metadata can be shared and distributed without a romset but when roms are available the metadata is already there

Metadata fields can be added or removed without recompiling ES

Tags will be supported meaning ES can filter based on tags (like favourites or 'kid friendly' for example)

...
I'm trying not to making it a 'breaking' change in that the new version should be able to read the current gamelist.xml files and populate the database automatically.

Ok, maybe I am just in a negative mood, but please help me reason through this here: With the step to a fully querieable SQLite database, are you/we moving away from XML files altogether?
That would mean we are also moving away from a lot of supporting scripts and tools, from the venerable Sselph scraper, to the new and flashy UXS tool. Also, the Big Benefit of XML is that the data is user readable and editable. It would be very nice if we can somehow retain that aspect of the community-effort.

#1 was a real issue but is it still? PR#78 was merged in recently, and should mostly solve the issue of long saving times (loading not so much propably). #4 is an important additional feature, I agree, but SQL is not needed for it. #2 and #3 are nice to have features, but their gain lie mostly in the future, when other aspects (i.e. a framework for sharing metadata packages, and some interface to change and populate the metadata) are put in place.

And really, do we need SQLite for that? Once the metadata is in memory, it doesn't really matter one way or the other, or does it?This is a real question of which i do not know the answer.
There have been many discussions on this before:

here at the RetroPie-repo,
here at Herdingers fork, and some more, see also their design document, for inspiration.

and of course here on the forum:

pro DB by @SimonBelmont and
anti-DB by @Arcuza

I guess the answer really comes down to what the metric is you are trying to improve. If it is performance, maybe we should start with some profiling to see if the loading can be just as easily sped up as the the saving apparently could (see the recent PR). Its clear that things can be improved, but I rather do it in a way that keeps the existing XML based tooling.

Again, I am really not trying to be a downer, it's just that we should be doing things for the right reasons, and I'm not seeing those at the moment.

sselph

These types of questions should be answered. Adding complexity to maintaining the project should have measurable benefits.

@Zigurana said in Metadata Improvements in EmulationStation:

With the step to a fully querieable SQLite database, are you/we moving away from XML files altogether?

In the original sqlite branch there was code to detect a new xml on boot and import it. It also would be fairly trivial to write an importer/exporter for the current XML format, which would be good when migrating to a new system.

That would mean we are also moving away from a lot of supporting scripts and tools, from the venerable Sselph scraper, to the new and flashy UXS tool. Also, the Big Benefit of XML is that the data is user readable and editable.

I could write to sqlite but it is a little more of a pain since using the C library makes it difficult to cross-compile. The ability for a human to edit xml is questionable. Simple things are okay but I've helped lots of people who have mangled their xml by opening it in windows. I also wrote a neat web based metadata editor a long time ago but it was pending the sql update.

#2 and #3 are nice to have features, but their gain lie mostly in the future

I actually don't know if #2 is very feasible. It might be for MAME but for console games unless you use my renamer or my hash algorithms, it would make it hard to have an entry in a DB waiting for the file to appear.

fieldofcows

@Zigurana - the question should absolutely be asked as to whether this is the right approach. Believe me I've asked myself the same thing many times over the last few days looking at how much time this is taking me.

At the end of the day, it's not my decision as to whether this should be implemented in any 'official' branch or not. I am, after all, doing this as a hobby and would not be offended if it is decided that this is the wrong approach. All I can do is provide the facts, code and test results and leave it up to others to decide. Some of the 'facts' that I've found so far though:

I have already implemented the parser from both gamelist.xml and scanning the ROMS directory exactly as non sqlite ES did. This means that we
could if we wanted to still use this as the only persistent game/metadata source, recreating the database of every start (it doesn't take long). The sqlite database is then only used as an internal query engine.
The whole system/game view architecture is based on the layout of the configuration folders and XML files. This needed large changes in order to support some of the filtering/queries I plan to implement. It is simpler for me to understand my new code than try to massage the old code when changing this. I think implementing #4 with the current architecture would be hard.
It would be very easy to write a standalone import/export tool to convert from gamelist.xml to sqlite. I can certainly put this together as part of this work
The tag/filtering changes were always going to be large regardless of whether we use sqlite or XML. I wanted to put some unit testing into the code in order to reduce the risk of introducing bugs but the existing code does not lend itself to unit testing without a major rewrite so I used the opportunity to switch to sqlite.

I'm sure there are more but it's getting late and my brain is getting fuzzy.

Zigurana

@fieldofcows Thank you for your considered response! I will mull it over while trying to sleep.
I am glad you are taking this so very professional, even when it's your hobby!

Lets get back to this when we are both fresh and less fuzzy.

mattrixk

@fieldofcows just throwing it out there as I thought of it: how likely is a "related games" filter?

Eg: on the Detailed View would it be possible to have a list of other games that share a number of tags/filters/whatever?

On further thinking, it probably wouldn't work with the way things work now, but I just wanted to mention it.

fieldofcows

@mattrixk That's a really good idea. Technically it's easily possible but I don't know how well it would fit in the UI.

mattrixk

@fieldofcows yeah, it was the UI implementation that I wasn't sure about. I was originally thinking of a second gamelist, but how do you access it? If up/down travels the main gamelist and left/right changes system, how do you move over to the "related" gamelist? Without menu intervention at least.

Interesting in theory, but I don't know how it would work in practice.

MWGemini

I just wanted to add that for those of us with large collections, folder structure is vitally important. Also, @Zigurana, yes performance is still very much an issue. I've got about 63,000 ROMS in my collection currently, and startup time on my stock-clocked Rpi3 is roughly 8 minutes using the most recent "official" (v2.1.0RP) version of EmulationStation. ES is also not saving metadata, and although I haven't fully tracked down the issue yet, I believe the collection size is playing a factor (yes, I've checked all the easy settings/configs already, but let's not derail this topic for this particular problem).

Also, @fieldofcows, you said you are not a database engineer; I've done some database design professionally and would be happy to help with this project. I sent you an email with further information a few days ago. I've been thinking about this for the last few days, and have some ideas that will allow people to use their existing folder structures (i.e. grouped by system type), but also to easily save/add new filter types and search queries for the main system select screen (i.e. "Classic Platformers" or "Mario Collection" or "Favorites" or whatever). Furthermore, I have some ideas that will allow for each of these filter types to (relatively easily) have their own custom themes.

@mattrixk and @Nismo have both been doing some great theme work and have expressed interest in recent posts with having things like video previews, fan-art backgrounds (made easier now to thanks to the UXS scraper by @screech), etc. It looks like @sselph is updating his scraper and looking towards better ES integration now as well.

I've also read posts from other users who would like to see additional artwork, and this could all be easily referenced by the new database system, and displaying it should be pretty easy too, although I'm not familiar with the guts of the current ES codebase. This could also give us the ability to do more advanced things like dynamically rotating and scaling artwork while scrolling through games, for example, which would give theme designers a much more powerful toolset to work with, without a large development burden.

From a user's perspective, I love what you've already done and am very excited to see where this goes. From a developer's perspective, I'm willing to provide some assistance with implementing these features.

fieldofcows

Hi @MWGemini. Thank you very much for your post. I was beginning to wonder about the value of the metadata update I'm working on because it is taking quite a lot of my limited spare time but you have just made it all worthwhile.

I had a bit of a problem with spam email over the last couple of day - I think some scraper has used my email address to send a load of spam and I got ~500 failed delivery reports so I just bulk deleted it all so I would appreciate it if you could resend your email. Thanks.

What I have is a little bit away from working but I think the data structure now allows three different (at least) types of grouping:

Folder structure from a relative root directory (so the path for a game is stored as "dir1/dir2/game.zip" for example)
Tags - just a text field. Any number of tags can be specified per game
Metadata - this is just a list of tag/value pairs. Like tags above, there can be any number of these per game

I have a C++ class named GameData that manages the game list and metadata. In order to query the database you just ask for the game list and specify the filter. At this stage the results are queried from the database and populated in memory. They are populated either as a flat gamelist or maintaining the folder hierarchy where equivalent relative paths in different systems are grouped together.

It all seems to make sense to me at the moment. The code is a bit scruffy and incomplete and the version of ES built on my codebase doesn't work properly at the moment but if you want to examine the code then you can take a look at https://github.com/fieldofcows/EmulationStation/tree/sqlite. Most of the code for the game data is in the es-gamedata subdirectory.

MWGemini

@fieldofcows I've just re-sent the email, and I'll try to set aside some time to look through your recent code changes and try to put my database design ideas into a diagram to send to you.

fieldofcows

@MWGemini Got your email, thanks. I'll have a good read and get back to you. Thanks!

mattrixk

@Zigurana: You asked for it, here it is.
@fieldofcows: This is for you.
@MWGemini: You might be interested in this too.

I made a design proposal doc with Google Docs, full of ideas about how I think we could make metadata and filtering work a lot better in ES.

It's still a WIP and kind of trails off towards the end, but I'll keep working on it, and people are welcome to leave comments (and if it's wanted, I can open it up to editing).

MWGemini

@mattrixk I also wrote up a design document, and it looks like we've got many of the same ideas. If you and @Zigurana want me to share it with you, send me your email (I can't seem to find a private message option on here... am I blind?). I'd love to get input from theme designers, since my document is focused more on the technical details and usability issues. I'd like to get it polished quite a bit more before sharing it with the rest of the community, but am happy to share it with other interested parties (particularly scraper developers such as @sselph and @screech).

mattrixk

@MWGemini nope, not blind. No message option (I think Buzz turned it off to discourage rom sharing). If love to see your document and give some input.

My doc is currently set to read/comment only, but I can add you, @Zigurana and @fieldofcows as editors. I think @Nismo might even want in on this.

Edit: removed email address.

MWGemini

@mattrixk Just sent you the doc. You might want to remove your email from your post to prevent spam :). I also made a few comments on your doc. I agree with basically everything you wrote, with the exception of database versus XML. Currently, gamelists are stored in XML and I (and @fieldofcows too, I think) want to replace that with a SQL database. It's a big undertaking, but it'll give us a lot more power as users and for you as a theme designer. Functionality wise, I think we're pretty much on the same page.

sselph

@MWGemini I'm happy to read it over. My email address is on my github page.

MWGemini

@sselph Just shared it with you as well :).

mattrixk

@MWGemini said in Metadata Improvements in EmulationStation:

Currently, gamelists are stored in XML and I ... want to replace that with a SQL database.

I don't really have much of a stake in the XML vs DB argument. I'm happy to go with whatever is suggested by the people who know what they are talking about. From what I've gathered, XML is lot easier to edit/add info than a database, but the DB allows faster loading/categorisation of info? Would using DB change how themes are made? Would they still be made in XML?

I was thinking if we went ahead with all of the extra filters/tags and metadata, that we would possibly need some kind of separate program to handle it all, be it web-based or desktop based. I know amadeus has his Retropie Metadata Editor for Windows. I haven't had the chance to look into it, but maybe something like that would be of use.