Metadata Improvements in EmulationStation

Zigurana

@mattrixk said in Metadata Improvements in EmulationStation:

I think I'm going to write up a document, a design proposal even...

Maybe a google document we can all add to?

herb_fargus

@Zigurana I know @jacobfk20 had done some Integrations with an on screen keyboard...

mattrixk

@Zigurana The only thing I can think to add to your list is the ability to Sort by System. If I have a list of Mario related games, I'd like to be able to view them grouped into their respective systems, or even just by release date, so you can see them evolve.

I suppose the question then becomes how do you sort the systems? By Name or by Age? I think sorting them by Age with older systems first would be the most useful/used, but would people want the option to choose Alphabetical?

Side note: I started working on a MarkDown document on stackedit.io that I have linked to my google drive. I'm not sure how that would go with sharing, but I'll look into it once I've done some more work on the doc.

fieldofcows

Thanks all for your comments so far. I wasn't expecting so much detail!

I'm just playing around with integrating the new metadata functionality into ES replacing the existing code just to see what problems I am going to be facing and I've run into the first difficulty - folders.

So, I'm just going to ask the question. Who is using folders to organise their roms? Would you miss this feature if it was removed but replaced with a more comprehensive set of filters?

I don't mind implementing folder support but it adds complication and may not be required if we have better filtering/sorting.

Nismo

@fieldofcows For me it's ok without folders, but just my opinion.

mattrixk

@fieldofcows I've been wanting to use folders to sort games into favourites, genres, 2-player games, etc, but it's a big task and I haven't had the time/motivation to do it (and I already use child-friendly ES for favourites).

I'm also sad that I can't add any metadata to folders to give them special images and descriptions and such.

So for me, I'm fine without folders, because the new filter/sort options should hopefully do everything I would want folders to do, and do them better.

Side note: I mentioned above about 'custom filter bundles' and it got me thinking; would it be possible to populate our own carousel? Instead of a carousel of just Systems, could we maybe choose what filters to show on the carousel? Like a quicklist of all the best filters? Then we could have select Systems, along with our most used other filters (like the "favourites" filter, the "3d racing - racing|psx|3do" bundle filter etc).

Just a thought anyway.

Zigurana

@fieldofcows Personally, I've never used it, but it was one of those things that came back to haunt me when I released Kid-modes for ES. This is just to say that others are using it currently, not sure what for other than creating more gategories besides systems.

ebtalk

I don't use folders but have contemplated using them for arcade. With this, I would be totally ok without folders.

One thing I would add is to be careful on how much ability of customization you enable for the end user. In my field of work with some software implementations, the more fields and abilities you give to the end user, the more problems it creates. Just my two cents though.

herb_fargus

@Zigurana I don't really use folders but there are some cases like scummvm or dos where they have multiple game files so I keep them in folders to keep them organised. That may be a use case that isn't addressed by the filters.

Zigurana

@herb_fargus said in Metadata Improvements in EmulationStation:

@Zigurana I don't really use folders but there are some cases like scummvm or dos where they have multiple game files so I keep them in folders to keep them organised. That may be a use case that isn't addressed by the filters.

True, but is there not a shell script placed in the parent directory that is used to launch the game? Maybe, the subdirectories can effectively be ignored?

mattrixk

@fieldofcows I've been thinking about filters and forgot about something I've been wanting. Is it possible to add more images to the ROM metadata? I've been hearing about marquees, and obviously video previews, so could we add other fields for fan-art, screenshots, box-art and the like? I know the Pi isn't too powerful so you don't want to go too overboard with the images, but I'd like to have the option there.

I've been using UXS to make some nice combo images involving screenshot, box-art and game logo, but I can't really do one of the things I really want to do, which is the big fan-art in the background. I can use UXS to make a small image (640x360) that will scale up to full screen on the Pi. I don't mind if the large fan-art gets blurry because it's behind a "screen" anyway, but when the box-art and screenshot scale up, the blurriness is unacceptable.

I can still make the theme I want within ES if I use UXS to put the screenshot, box-art and logo together into a single image, then use that as the scraped image, but it will be missing the large fan-art background.

Even if we could just add the option for a second image, I can do the rest myself.

mediamogul

@fieldofcows said in Metadata Improvements in EmulationStation:

Who is using folders to organise their roms?

I use folders for several different purposes and I'd genuinely hate to see them eliminated. However, if they are going to cause a lot of difficulty moving forward, I guess I could find alternative methods to adapt.

Would you miss this feature if it was removed but replaced with a more comprehensive set of filters?

My main concern is that I keep categorized folders of launch scripts in the RetroPie menu to accomplish different tasks, such as system backup, as well as dozens of controller key-mappings organized by system. Would the new set of filters you mention be able to replace this type of use case scenario?

fieldofcows

@mediamogul said in Metadata Improvements in EmulationStation:

Would the new set of filters you mention be able to replace this type of use case scenario?

Maybe not.

I spent some time last night looking at how to build folder support into my metadata model and I think I've cracked it so it looks like folders should be back on the feature list.

mediamogul

@fieldofcows

That's fantastic! I really appreciate all of the time you've put into all this. Emulation Station is really on the move and I look forward to seeing it all come together.

Zigurana

@fieldofcows said in Metadata Improvements in EmulationStation:

...
I had a look at the 'sqlite' changes made (I think) by Aloshi and decided I wanted to go a stage further so I'm currently busy implementing a new set of C++ classes that manage the game list and metadata using sqlite over multiple tables. I'm not a database software engineer so it may not be perfect but should hopefully be an improvement over what's there now.

I can see a number of benefits once this is working:

Very fast startup and shutdown of ES

Metadata can be managed separately to the installed game list meaning (for example) the metadata can be shared and distributed without a romset but when roms are available the metadata is already there

Metadata fields can be added or removed without recompiling ES

Tags will be supported meaning ES can filter based on tags (like favourites or 'kid friendly' for example)

...
I'm trying not to making it a 'breaking' change in that the new version should be able to read the current gamelist.xml files and populate the database automatically.

Ok, maybe I am just in a negative mood, but please help me reason through this here: With the step to a fully querieable SQLite database, are you/we moving away from XML files altogether?
That would mean we are also moving away from a lot of supporting scripts and tools, from the venerable Sselph scraper, to the new and flashy UXS tool. Also, the Big Benefit of XML is that the data is user readable and editable. It would be very nice if we can somehow retain that aspect of the community-effort.

#1 was a real issue but is it still? PR#78 was merged in recently, and should mostly solve the issue of long saving times (loading not so much propably). #4 is an important additional feature, I agree, but SQL is not needed for it. #2 and #3 are nice to have features, but their gain lie mostly in the future, when other aspects (i.e. a framework for sharing metadata packages, and some interface to change and populate the metadata) are put in place.

And really, do we need SQLite for that? Once the metadata is in memory, it doesn't really matter one way or the other, or does it?This is a real question of which i do not know the answer.
There have been many discussions on this before:

here at the RetroPie-repo,
here at Herdingers fork, and some more, see also their design document, for inspiration.

and of course here on the forum:

pro DB by @SimonBelmont and
anti-DB by @Arcuza

I guess the answer really comes down to what the metric is you are trying to improve. If it is performance, maybe we should start with some profiling to see if the loading can be just as easily sped up as the the saving apparently could (see the recent PR). Its clear that things can be improved, but I rather do it in a way that keeps the existing XML based tooling.

Again, I am really not trying to be a downer, it's just that we should be doing things for the right reasons, and I'm not seeing those at the moment.

sselph

These types of questions should be answered. Adding complexity to maintaining the project should have measurable benefits.

@Zigurana said in Metadata Improvements in EmulationStation:

With the step to a fully querieable SQLite database, are you/we moving away from XML files altogether?

In the original sqlite branch there was code to detect a new xml on boot and import it. It also would be fairly trivial to write an importer/exporter for the current XML format, which would be good when migrating to a new system.

That would mean we are also moving away from a lot of supporting scripts and tools, from the venerable Sselph scraper, to the new and flashy UXS tool. Also, the Big Benefit of XML is that the data is user readable and editable.

I could write to sqlite but it is a little more of a pain since using the C library makes it difficult to cross-compile. The ability for a human to edit xml is questionable. Simple things are okay but I've helped lots of people who have mangled their xml by opening it in windows. I also wrote a neat web based metadata editor a long time ago but it was pending the sql update.

#2 and #3 are nice to have features, but their gain lie mostly in the future

I actually don't know if #2 is very feasible. It might be for MAME but for console games unless you use my renamer or my hash algorithms, it would make it hard to have an entry in a DB waiting for the file to appear.

fieldofcows

@Zigurana - the question should absolutely be asked as to whether this is the right approach. Believe me I've asked myself the same thing many times over the last few days looking at how much time this is taking me.

At the end of the day, it's not my decision as to whether this should be implemented in any 'official' branch or not. I am, after all, doing this as a hobby and would not be offended if it is decided that this is the wrong approach. All I can do is provide the facts, code and test results and leave it up to others to decide. Some of the 'facts' that I've found so far though:

I have already implemented the parser from both gamelist.xml and scanning the ROMS directory exactly as non sqlite ES did. This means that we
could if we wanted to still use this as the only persistent game/metadata source, recreating the database of every start (it doesn't take long). The sqlite database is then only used as an internal query engine.
The whole system/game view architecture is based on the layout of the configuration folders and XML files. This needed large changes in order to support some of the filtering/queries I plan to implement. It is simpler for me to understand my new code than try to massage the old code when changing this. I think implementing #4 with the current architecture would be hard.
It would be very easy to write a standalone import/export tool to convert from gamelist.xml to sqlite. I can certainly put this together as part of this work
The tag/filtering changes were always going to be large regardless of whether we use sqlite or XML. I wanted to put some unit testing into the code in order to reduce the risk of introducing bugs but the existing code does not lend itself to unit testing without a major rewrite so I used the opportunity to switch to sqlite.

I'm sure there are more but it's getting late and my brain is getting fuzzy.

Zigurana

@fieldofcows Thank you for your considered response! I will mull it over while trying to sleep.
I am glad you are taking this so very professional, even when it's your hobby!

Lets get back to this when we are both fresh and less fuzzy.

mattrixk

@fieldofcows just throwing it out there as I thought of it: how likely is a "related games" filter?

Eg: on the Detailed View would it be possible to have a list of other games that share a number of tags/filters/whatever?

On further thinking, it probably wouldn't work with the way things work now, but I just wanted to mention it.

fieldofcows

@mattrixk That's a really good idea. Technically it's easily possible but I don't know how well it would fit in the UI.