EmulationStation mod

Zigurana

@ben_thatmustbeme said in EmulationStation mod:

The latest attempt to merge herdinger's code, @Zigurana and @jacobfk20's grid view code all in to mine is on the 'allmerge' branch, but a test run showed it clearly had problems on the main system list, haven't dug in to what happened yet.

Hey, I got your allmerge branch to compile and link successfully, so thats a start. Also, I started annotating your latest commit (84e93b9), to start some discussion on the merge. Do you think that is a worthwhile exercise?

And where do we go from here? Do keep working on my own repo (which you will merge whenever you see fit), or do I start serving you PRs?

ben_thatmustbeme

@Zigurana sure, anything is helpful. there was a lot in there done somewhat blindly too.

From here on out, thats up to you, I would certainly prefer PRs, makes things a lot easier. The other option is I give you direct commit access to the branch. I would not mind that.

jacobfk20

@lilbud Does it do this to you when you return from a game? If so the latest build fixes it.

Link to latest Test Build

Thanks for reporting :)

jdoolin

@Zigurana said in EmulationStation mod:

Functionality
@Arcuza, you are right we do not need many of the features that a full database implementation would bring. Multiple users, security layers and network access are probably not necessary.
However, what we are sorely lacking is a way to build queries to create lists of game and perform sorting on all available metadata (across platforms if need be). Also, we need some way to add user-defined tags to allow arbitrary collections and filters. In the current implementation of ES XML trees, none of this is possible, and I am hoping/expecting that a db approach would make this possible.

The nice thing about SQLite is that it doesn't feature multiple users, security layer, network access or daemons that run in the background. It is a VERY simple file based database used on platforms like Android. I've never run benchmarks, but I would think ES is exactly the sort of project it lends itself to.

I'm going to develop a schema tonight, write a program to convert my gamelists to the SQLite database and start running queries on my Pi3.

mattrixk

I know you guys are a having a full Dev discussion and I don't want to interrupt that, but I just had a thought I wanted to put out there:

Would it be/is it possible to choose the order of systems in the carousel? At the moment it's just alphabetical, I'm curious if we could do custom orders, like keeping all Nintendo consoles together and all Sega/Playstation/Atari etc. Also, maybe choose to start on a specific system eg: Mame.

jdoolin

@mattrixk said in EmulationStation mod:

I know you guys are a having a full Dev discussion and I don't want to interrupt that, but I just had a thought I wanted to put out there:

Would it be/is it possible to choose the order of systems in the carousel? At the moment it's just alphabetical, I'm curious if we could do custom orders, like keeping all Nintendo consoles together and all Sega/Playstation/Atari etc. Also, maybe choose to start on a specific system eg: Mame.

I'm not sure how it currently works but if the SQLite schema ends up working out it seems to me that it would be an easy option to add. It's just a different sort order in the database query. Right now in my schema I have a Systems table that has a developer field. So when ES queries the database it could just ORDER BY developer ASC then all systems would appear in order of developer, such as Atari, Coleco, Mattel, Nintendo, Sega.

mattrixk

@jdoolin That's awesome. I wonder if it could do custom orders too, like putting Mame(s), FBA and Neo Geo together, but keeping the rest in developer order.

jdoolin

Early tests with an SQLite3 game database on the RPi3 are looking good.

So first I made a very simple schema. It has four tables.

systems - a table to save information on all supported game/computer systems, such as name, developer, release date and description
games - obvious. I'm using the same fields you find in the current gamelist.xml files
tags - just a simple string for the tag
game_tags - a table connecting a game with a tag

Then I wrote a Python script and a Bash script wrapper to parse all of my gamelists and convert the entries into SQLite3 and add them to the database. This took a few minutes as my gamelists are pretty big. I've got a very good sample for testing.

Then I started writing queries and thus far, using the command line sqlite3 client I haven't seen a query take more than one tenth of a second. I take that back, when I queried the database for every single game in the db, it took 0.17 seconds. The real test would be using the SQLite3 library in the ES code and see how it performs from within ES. But right now I'm quite encouraged. Just under two tenths of a second to read my entire game database is pretty quick.

Now keep in mind that if we use SQLite3, we can commit changes to the database immediately. There won't have to be any saving MetaData when you exit ES since it's done on the fly.

Time to start looking through ES code.

jdoolin

It also occurs to me that if we save system data in the database, we may no longer need es_systems.cfg. Is it a convenience for some people to be able to edit es_systems.cfg? I've never done it myself, only referred to it.

herb_fargus

@jdoolin lots of people do manual edits, which is something that may be an issue when switching to a sqlite database but if the queries provide the functions that people use manual editing for (like sorting systems) then it may be less of an issue

Zigurana

@jdoolin :
Hey, that's great to hear! Make sure you look at Aloshi's initial attempts to an SQL db for ES (branch). Also in the (many) issues and PRs for that repo, I think I've seen people posting fixes and or enhancements for that implementation, so it might be worth it to spend some time searching the Upstream.

One thing we need to have well in place, is a robust conversion from the XMLs to the database. Not only for the initial stransition, but also for smaller updates of ones metdadata descr. I assume that many of the tools we use (eg. sselphs scraper) will only be able to work with the xmls, so there will likely never be a clean break from one approach to the other. A strategy is needed to cope with that.

herb_fargus

@Zigurana mehstation has done some work on converting xmls to his (sqlite?) db thats also something you could probably look at for a resource. its more of a secondary tool though

InsecureSpike

@jdoolin said in EmulationStation mod:

It also occurs to me that if we save system data in the database, we may no longer need es_systems.cfg. Is it a convenience for some people to be able to edit es_systems.cfg? I've never done it myself, only referred to it.

i for one would much prefer es_systems to stay the same, as i have around 10 custom entries

herb_fargus

@lilbud I've updated your theme a tad to optimise the space in the gridview list (source here: https://github.com/HerbFargus/es-theme-material/tree/gridview):

here are a few examples (set at 0 pixels):

Boxart

gridview

Screenshots

gridview-screenshots

Personally I feel that overall screenshots are a better standardised layout for gridview in terms of dimensions. (see my solution for that HERE

A few caveats:

the origin doesn't work in the theme because it treats it as a text value which doesnt have the origin option, as a consequence I had to put the game title on the bottom because if I put it at the top I couldnt have 4 rows without it covering the titles. It barely fits at the bottom, which means you need to disable your help text from the start menu in emulationstation. I suppose you could keep the top instead with the system logo if you like and just remove the first row of the grid as well.
For the most part I see this as a temporary solution as there are plans to set the game titles under each image, Once that's incorporated I may add the system logo back and go back down to 3 rows instead of 4.

Another interesting theme idea is to set a panel on the left side with the logo vertical... That may actually look pretty cool. I'll have to try that out.

jdoolin

@InsecureSpike said in EmulationStation mod:

@jdoolin said in EmulationStation mod:

It also occurs to me that if we save system data in the database, we may no longer need es_systems.cfg. Is it a convenience for some people to be able to edit es_systems.cfg? I've never done it myself, only referred to it.

i for one would much prefer es_systems to stay the same, as i have around 10 custom entries

It occurs to me that I worded that poorly. I was pretty tired at the time. :-)

It's not that you wouldn't be able to have custom entries, there would just have to be a way of editing those entries in the database.

jdoolin

@Zigurana said in EmulationStation mod:

@jdoolin :
Hey, that's great to hear! Make sure you look at Aloshi's initial attempts to an SQL db for ES (branch). Also in the (many) issues and PRs for that repo, I think I've seen people posting fixes and or enhancements for that implementation, so it might be worth it to spend some time searching the Upstream.

Oh, excellent. That will be a perfect reference. I already started looking through the code in your branch so I'm starting to see how things fit together. Seeing Aloshi's attempts at SQL will only speed things up, so thanks for that.

One thing we need to have well in place, is a robust conversion from the XMLs to the database. Not only for the initial stransition, but also for smaller updates of ones metdadata descr. I assume that many of the tools we use (eg. sselphs scraper) will only be able to work with the xmls, so there will likely never be a clean break from one approach to the other. A strategy is needed to cope with that.

I basically wrote an early version of this last night. I'll add error checking and reporting, make it as general as I can and have it use es_systems.cfg instead of a directory listing in ~/.emulationstation/gamelists.

I could also attempt a replacement scraper as well, that would dump directly to the SQLite db. I used to write scripts like that for a living too.

lilbud

@Rookervik How do you think I made the Material Theme?

Rookervik

@lilbud Hmm?

lilbud

@Rookervik Work on the theme.xml and restart ES everytime I made a Change

mattrixk

@lilbud said in EmulationStation mod:

Work on the theme.xml and restart ES everytime I made a Change

Instead of restarting ES each time to see any changes you made to your theme, you can just switch to another theme and then back to your WIP theme. It should be a lot faster than restarting ES.

... Unless I completely misunderstood what you were saying...