EmulationStation mod

jdoolin

I know you guys are a having a full Dev discussion and I don't want to interrupt that, but I just had a thought I wanted to put out there:

Would it be/is it possible to choose the order of systems in the carousel? At the moment it's just alphabetical, I'm curious if we could do custom orders, like keeping all Nintendo consoles together and all Sega/Playstation/Atari etc. Also, maybe choose to start on a specific system eg: Mame.

I'm not sure how it currently works but if the SQLite schema ends up working out it seems to me that it would be an easy option to add. It's just a different sort order in the database query. Right now in my schema I have a Systems table that has a developer field. So when ES queries the database it could just ORDER BY developer ASC then all systems would appear in order of developer, such as Atari, Coleco, Mattel, Nintendo, Sega.

mattrixk

@jdoolin That's awesome. I wonder if it could do custom orders too, like putting Mame(s), FBA and Neo Geo together, but keeping the rest in developer order.

jdoolin

Early tests with an SQLite3 game database on the RPi3 are looking good.

So first I made a very simple schema. It has four tables.

systems - a table to save information on all supported game/computer systems, such as name, developer, release date and description
games - obvious. I'm using the same fields you find in the current gamelist.xml files
tags - just a simple string for the tag
game_tags - a table connecting a game with a tag

Then I wrote a Python script and a Bash script wrapper to parse all of my gamelists and convert the entries into SQLite3 and add them to the database. This took a few minutes as my gamelists are pretty big. I've got a very good sample for testing.

Then I started writing queries and thus far, using the command line sqlite3 client I haven't seen a query take more than one tenth of a second. I take that back, when I queried the database for every single game in the db, it took 0.17 seconds. The real test would be using the SQLite3 library in the ES code and see how it performs from within ES. But right now I'm quite encouraged. Just under two tenths of a second to read my entire game database is pretty quick.

Now keep in mind that if we use SQLite3, we can commit changes to the database immediately. There won't have to be any saving MetaData when you exit ES since it's done on the fly.

Time to start looking through ES code.

jdoolin

It also occurs to me that if we save system data in the database, we may no longer need es_systems.cfg. Is it a convenience for some people to be able to edit es_systems.cfg? I've never done it myself, only referred to it.

herb_fargus

@jdoolin lots of people do manual edits, which is something that may be an issue when switching to a sqlite database but if the queries provide the functions that people use manual editing for (like sorting systems) then it may be less of an issue

Zigurana

@jdoolin :
Hey, that's great to hear! Make sure you look at Aloshi's initial attempts to an SQL db for ES (branch). Also in the (many) issues and PRs for that repo, I think I've seen people posting fixes and or enhancements for that implementation, so it might be worth it to spend some time searching the Upstream.

One thing we need to have well in place, is a robust conversion from the XMLs to the database. Not only for the initial stransition, but also for smaller updates of ones metdadata descr. I assume that many of the tools we use (eg. sselphs scraper) will only be able to work with the xmls, so there will likely never be a clean break from one approach to the other. A strategy is needed to cope with that.

herb_fargus

@Zigurana mehstation has done some work on converting xmls to his (sqlite?) db thats also something you could probably look at for a resource. its more of a secondary tool though

InsecureSpike

@jdoolin said in EmulationStation mod:

It also occurs to me that if we save system data in the database, we may no longer need es_systems.cfg. Is it a convenience for some people to be able to edit es_systems.cfg? I've never done it myself, only referred to it.

i for one would much prefer es_systems to stay the same, as i have around 10 custom entries

herb_fargus

@lilbud I've updated your theme a tad to optimise the space in the gridview list (source here: https://github.com/HerbFargus/es-theme-material/tree/gridview):

here are a few examples (set at 0 pixels):

Boxart

gridview

Screenshots

gridview-screenshots

Personally I feel that overall screenshots are a better standardised layout for gridview in terms of dimensions. (see my solution for that HERE

A few caveats:

the origin doesn't work in the theme because it treats it as a text value which doesnt have the origin option, as a consequence I had to put the game title on the bottom because if I put it at the top I couldnt have 4 rows without it covering the titles. It barely fits at the bottom, which means you need to disable your help text from the start menu in emulationstation. I suppose you could keep the top instead with the system logo if you like and just remove the first row of the grid as well.
For the most part I see this as a temporary solution as there are plans to set the game titles under each image, Once that's incorporated I may add the system logo back and go back down to 3 rows instead of 4.

Another interesting theme idea is to set a panel on the left side with the logo vertical... That may actually look pretty cool. I'll have to try that out.

jdoolin

@InsecureSpike said in EmulationStation mod:

@jdoolin said in EmulationStation mod:

It also occurs to me that if we save system data in the database, we may no longer need es_systems.cfg. Is it a convenience for some people to be able to edit es_systems.cfg? I've never done it myself, only referred to it.

i for one would much prefer es_systems to stay the same, as i have around 10 custom entries

It occurs to me that I worded that poorly. I was pretty tired at the time. :-)

It's not that you wouldn't be able to have custom entries, there would just have to be a way of editing those entries in the database.

jdoolin

@Zigurana said in EmulationStation mod:

@jdoolin :
Hey, that's great to hear! Make sure you look at Aloshi's initial attempts to an SQL db for ES (branch). Also in the (many) issues and PRs for that repo, I think I've seen people posting fixes and or enhancements for that implementation, so it might be worth it to spend some time searching the Upstream.

Oh, excellent. That will be a perfect reference. I already started looking through the code in your branch so I'm starting to see how things fit together. Seeing Aloshi's attempts at SQL will only speed things up, so thanks for that.

One thing we need to have well in place, is a robust conversion from the XMLs to the database. Not only for the initial stransition, but also for smaller updates of ones metdadata descr. I assume that many of the tools we use (eg. sselphs scraper) will only be able to work with the xmls, so there will likely never be a clean break from one approach to the other. A strategy is needed to cope with that.

I basically wrote an early version of this last night. I'll add error checking and reporting, make it as general as I can and have it use es_systems.cfg instead of a directory listing in ~/.emulationstation/gamelists.

I could also attempt a replacement scraper as well, that would dump directly to the SQLite db. I used to write scripts like that for a living too.

lilbud

@Rookervik How do you think I made the Material Theme?

Rookervik

@lilbud Hmm?

lilbud

@Rookervik Work on the theme.xml and restart ES everytime I made a Change

mattrixk

@lilbud said in EmulationStation mod:

Work on the theme.xml and restart ES everytime I made a Change

Instead of restarting ES each time to see any changes you made to your theme, you can just switch to another theme and then back to your WIP theme. It should be a lot faster than restarting ES.

... Unless I completely misunderstood what you were saying...

mattrixk

@jacobfk20

Okay, total noob question, but is there a step by step on how to get this set up on a PC?

I have a windows PC with a portable version of ES. Can I set it up on that?
Or I can set up Linux Mint on a virtual machine, but then I need instructions on how to install ES on there, as well as installing your Grid View mod.

Can anyone help me out with this? I want to update my themes to work with Grid View, but I need to be able to do it on my PC because I don't really have access to my Pi for the length of time I would need (it's mainly my media center).

jacobfk20

@mattrixk I actually haven't compiled a windows build yet. I tried yesterday but I would get a ton of compilation errors that other people are apparently having on upstream as well. I'll let you know when I get a windows build up and running.

jdoolin

I'm updating the database schema to use all of the fields in es_systems.cfg. So including the few I've added, the systems table now has the following fields, with example data:

name (snes)
fullname (Super Nintendo Entertainment System)
path (/home/retropie/RetroPie/roms/snes
extension (.bin .sfc .fig .zip...)
command (/opt/retropie/supplementary/runcommand/runcommand.sh 0 SYS snes %ROM%)
platform (snes) used for scraping
theme (snes)
developer (Nintendo)
release_date (August 23, 1991)
description (The SNES was the successor to the wildly successful SNES. Using a 16 bit blah, blah, blah)

The three fields I thought to add are in bold. If anyone has any other suggestions for the systems table or the database in general, let me know. Obviously, the leaner the database the better, but if adding features doesn't cost too much, great.

mattrixk

@jacobfk20
I'm okay with it not working for Windows, but I could do with some instructions on how to get both regular ES and your Grid View Mod working on a Linux virtual machine (I've found some instructions on how to get Linux Mint 16 running on a VM, and it looks pretty straight forward) if you have the time.

@jdoolin
Maybe a path to a system image? Either a picture of the console, or of the controller like in Carbon theme? It's not really essential as they can be added pretty easily in the theme.xml with an extra="true" tag, but I just thought I'd put it out there.

jacobfk20

@mattrixk This is from memory, so here goes nothing...
When I was taking photos for this thread initially I did it on ElementaryOS running on an old pc. I compiled on there but on linux its vary simple. On first it should create a .emulationstation directory with an es_systems.cfg in your home folder (may be hidden) it will have a example system in there (nes) and a location to put roms. put roms in that folder (will have to create those folders most likely) Once you have that it should boot like on the pi, but you'll only have NES. It won't run any roms, but you can at least test themes on there.

To compile:
Just download the zip of my Gridview fork https://github.com/jacobfk20/EmulationStation/tree/Gridview
and follow the directions for "building" in the readme below. (Linux Mint is a debian/ubuntu)

Just copy and paste the commands it says to do into terminal and it does the rest for you.

If you need more help with any of this then I will be glad to help. (except getting a emulator running)