NEW RetroPie Web Console

cyperghost

What are the reasons you think, that all these projects get stuck in development?
What frontend got the best development? (I knew only Recalbox web manager)
I think WebtroPie and YARMAN have both also good development ;)

@MileSacul
About filesize... I compared nginx and lighttpd install via aptitude. They need just round about 1MB and the server setup is usually very fast. Therefore I asked for NODE.JS and it's size. Also the database setting a SQLite* base would be a more common used one (up to now!).

So it's of course your decision what you want to do in your sparetime but if you really want to develop a frontend then fork the farthest developed frontend for now. ~~I think it's the Recalbox web manager as it was one of the first.~~ Maybe herb or other members can get annother idea here what seems the most developed RetroPie-WebUI.

Really no offence just my two cents! As there were so much frontends being in development stages but noone gets finished and it seems for me that all available programming languages were used :)
Thank you for your efforts ;) ... and I hope that one webgui gets finished

EDIT: Read more about webservices of RetroPie here

*I deeply hope that the gameslists and the metadata will be converted in one database like SQLite for portability reasons.

herb_fargus

@cyperghost idk. Everyone has their own idea of doing things and most weren't planning on long term dev or maintenance so much as just doing it cause they could. Some are bloaty, some are lightweight. We all do this in our free time so people pursue things they are interested in.

Anyways I'm interested to see how OP handles it, I welcome any new ideas, regardless of how long they last

cyperghost

@herb_fargus said in NEW RetroPie Web Console:

Anyways I'm interested to see how OP handles it, I welcome any new ideas, regardless of how long they last

Agreed ;)

@MileSacul Give us a shot of coding magic ;)

MileSacul

@cyperghost
While I like the idea of making this thing lightweight, I also have to look at application response times, developer adoption, performance footprint, the lifespan of used technologies.

I used these sites to collect initial thoughts:
HammerPrinciple SQLITE VS MONGODB
StackShare's MongoDB vs SQLLite
Web Technologies Survey
Node vs Apache vs Lighttpd vs Nginx
Apache Vs Nginx Vs Node.js And What It Means About The Performance Of WordPress Vs Ghost
Top 8 Node.js Best Practices For Your Startup

Keep in mind, the only sources I really consider reliable are StackShare and the Web Technologies Survey, but to be honest, finding valid performance reports on these technologies were a failure. I probably have to build them myself if I ever plan to (probably not).

Response times
A lot of analyses showed that Nginx pretty much beats NodeJS as a web server. While Lighthttpd is faster in smaller input/output, it gets slower as you give it more and more requests. Overal, NodeJS gives slower response times as a web server even when clustering.

As for MogoDB vs SQLLight, there were only thoughts on adoption and that both perform equally as well.

Developer Adoption
I was blown away by this statement from "Top 8 Node.js Best Practices For Your Startup:" To summarize, JavaScript is the #1 most-used language on GitHub and it looks like this trend will continue.

Although, the Web Technologies Survey showed that Nginx was only second to Apache and claimed over a third of the market while Node.js only took .3%.

As mentioned before, the fact that I can use Javascript both for my web application and my web server it means I don't have to switch my brain from going from one technology to the other. Especially when going from PHP (which Nginx uses) to JavaScript. Partnered with AngularJS, I can get help from anybody that knows JavaScript. Using one language to rule them all kind of strategy :P.

As for MongoDB vs SQLite, you are right about more people using SQLite. After all, it follows Oracle, DB2, and MySQL in terms of query syntax; I had quite a learning curve figuring out MongoDB because I am used to production databases. Also, only SQLite supports foreign keys, which is something I really like. There are a few pros for MongoDB though:

It stores JSON objects, which means no custom hooks to convert my queries
It allows JSON objects for query search, which yet again means no conversion.
It requires no real database schema. While Mongoose does use a schema, updating it is as simple as changing it in my JavaScript code. In terms of development, that means I don't have to include SQL update scripts with each database schema change.
It allows storing of arrays in a column. All of the functions that you see on the manage page is stored as a single array in a column.

** Performance Footprint **
Even though you can run very intensive operations (opt/exp lr-mame builds from source) with my app, I think keeping the performance footprint low is a good idea. Keep in mind, I designed the web console to always be running. Even when users are playing their video games.

The big thing I see from most of the nginx and NodeJS application is how many threads they run on. NodeJS is designed to run a single thread. This means that it is less likely to cause performance issues, with exception of calling all the install commands because I use NodeJS child_process to essentially run outside of NodeJS on its own.

Ningx creates a new thread for each connection to it. This is troublesome because it means it can be a resource hog when a ton of people are trying to run things. Although I don't expect many people connecting to the Web Console at once, that could change when designing the RetroPie Super Web Console.

As for resource cost of SQLite vs MongoDB, I didn't see any metrics available online, so I will probably have to prototype both of them and do comparisons. Since a lot of the lr-mame builds generate MB worth of log results (that I store in the DB), I can just execute a full install of everything and monitor performance during both operations. Although, I doubt either of them are CPU intensive since they are really based of I/O speeds.

Lifespan of used Technologies
This one is probably the hardest of all. After all, when in high school I used Yahoo as my primary search engine and developed my first web application with NotePad and Netscape browser.

A lot of the reports show that NodeJS is more of a small fry and Nginx is 2nd only to Apache. While I think NodeJS can be considered a solid investment based solely on the fact that it is built off of the Google JavaScript engine, there is a chance that Google can end up like Yahoo. PHP has stood the testament of time.

I will probably have to use GitHub as my primary source for my decision. The trends are pointing to JavaScript, so I should try to at least move in that direction even if NodeJS might be replaced.

As for the databases, SQLite seems like it will stand the testament of time more, solely because it utilizes a lot of the well known database CRUD operations. Not to mention it is used by all Andriod phones. For now, I will continue to use MongoDB and after I get my Schema strategy really down, I will probably move over to SQLite or figure out some sort of hybrid between the two.

Overal Summary
All of them look like they could work as the backbone of my application, but I've actually already spent two months (in my free time) learning NodeJS and MongoDB, then applying them in an application. Maybe once I truly know what I am doing, I will dabble with moving around, but for now, I will stay on the same path.

On a side note, I found some very interesting ideas like using a hybrid Nginx and NodeJS application, since Nginx transfers static files like CSS and images faster.

cyperghost

@milesacul Well I can't say if NODE.JS is the concept of the future. I don't have that deep economical point of view :) Of course the coding hits the mobile sector and Java is leading here.

I think there is no decision needed what PHP-server to use. RetroPie gots the PHP backend already installed. For server setup it's just a small step and you can choose between Apache/NGiNX/lighttpd.

PHP has stood the testament of time.

Yes it's a solid base and great projects are made by this. Just for an example Pi Control to control the Raspberry and to monitor system usage.

I don't know nothing about MongoDB but SQLite is quasi standard and it does not need a dedicated server structure. Therefore I think SQLite is also considerable.

As I'm not a native speaker, it's a bit difficult to explain/comment all facettes you pointed out. I just say thank you for your long and professional explaination of your project. I wish you much fun with this and I hope the result will be a

maintained
convenient
lightweight
intuitive

WEB GUI ;)

MileSacul

So I updated the application to Beta level. I added a new monitor page where you can monitor live statistics including:

CPU Utilization (Overall and for each core)
Memory (RAM)
CPU and GPU Temperatures
File System memory
Application stats including kill commands
Bluetooth devices

I also added execution results to basic and manage pages where you can view the full results.

Finally, I added queue history search page, since basic and manage only show executions over the last 24 hours.

MileSacul

I am working on the Games/Roms section and found out about gamelist.xml

When pulling games, I could utilize this XML file for looking up the game info, but found some issues regarding conflicting scrapers.

Now on page load, I check the roms directory for all of the emulator names (psx,snes,etc), so you don't have to pull the full list of all your games. I also added parsing of the gamelist.xml at page load to help with filtering the emulators, like by played game count.

When I first found gamelist.xml, it included playcount and lastplayed data like:

<game>
    <path>{path}</path>
    <name>{name}</name>
    <playcount>{playCount}</playcount>
    <lastplayed>{date: YYYYMMDDThhmmss}</lastplayed>
</game>

I tried loading a game, save stating, and exiting to see if the playcount or lastplayed had been updated, but it wasn't.

I later found that the XML file was generated by a scraper, so I tried emulationstations' scraper.

I noticed the emulation station scraper completely wiped that data out and replaced it with this:

<game>
    <path>{path}</path>
    <name>{name}</name>
    <desc>{desc}</desc>
    <image>{game photo}</image>
    <rating>{rating}</rating>
    <releasedate>{date: YYYYMMDDThhmmss}</releasedate>
    <developer>{developer}</developer>
    <publisher>{publisher}</publisher>
    <genre>{genre}</genre>
</game>

How/when is playCount and lastPlayed generated?

darknior

Hi
Today i finally try your work :)
Some test and debug already help ;)
It's really a great web interface ... i like the real time Monitor page with node.js :)

To answer your question, i know when we play a game it stock the information in memory.
And ES only update the XML when we leave it, ONLY IF we have check Other Options/Save Metadata on Exit

CPU Utilisation is cool and real time, can we also have an (session) history please ? From the time we open the web page, with the last 20 or more mesures?
When we compile source code with -j3 for exemple, or use so heavy emulator, it can be cool to have a graphic with CPU use and temperatures, maybe in two colors, to see how the pi work. Like this exemple with 4 bars for PROC and one line for TEMP.

alt text

Bluetooh devices working fine, i can see them but only the Address. Name, Company, Manufacturer are empty.
I have try with my 3 different 8bitDo controlers. I see their name in Retropie-Setup or in Retroarch.
Kill Command only show me 3 applications. I can't see Retroach for exemple, or OMXPlayer if they crash and i want to kill them. Have you hard coded the list of software we can see ?
Maybe it can be cool to make a black or white list that we can edit from your console, like that we can choose ourselves.
I think black list is a better choice, and you can give us yours. To be sure to hide systems soft, and see that we use, or new one we don't know.
Kill command is working fine.
Games/Roms section is hide for the moment.
-- What can we do in this section ?
-- Can you use the es_system file to list systems, because we use it to add custom systems like PCEngine CD, OpenBOR, CPS, etc...
-- If you show MEDIA and let us change them like update a cover, try tu use the path we already write in our gamelist.xml
Or give us an input to set : string name="uploadImageDirectory" value="~/medias/${system_name}/box art front"
For Box front, Logo, Video.

darknior

Maybe a bug, now i see the current MP4 playing in Task List ... :)
Upload/Download ROMS, saves, and states : Very interesting you think to saves and states.
Don't forgot we can also change the PATH in RETROACH config.
I hate to have my savestates mixed with roms, it's a pain to find them, i have change the path to have them in an other directory. It's really better to use. For exemple i use : ~/medias/${system_name}/savestate
I save them on my USB hard drive with ROM and MEDIA ;)
Manage Packages : Really excellent feature :)
Better if we can also view the Retropie-Extra packages too ;)
Update RetroPie-Setup Script : Can you also add the Retropie-Extra Script please ?

MileSacul

@darknior Thanks for trying it out and sorry for the slow response. While all of the features you requested are great, I would recommend you add open issues to the project so I can keep better track of them. :P

Regarding each item:
CPU Utilisation: I actually have plan to add that, just have to figure it out.
Bluetooth: Currently, I am using hcitool to get Bluetooth devices. Maybe there is a better tool.
Kill Command: I had to do a custom hook to filter and cleanup the names for everything. Typically, you have 100+ items running. Will look into a better method and options for the users.
Games/Roms: Still working on that one. Finally figured up file uploads for Roms. Working on Cheats next and then will release.
es_system: Haven't heard of that yet. Will look into it. Got a lead where I can find it?
MP4: A screenshot would be great too see what is up with that. I only display things with Retro in the name.
Retropie-Extra: That looks great. Going to add before the next update.

darknior

@milesacul said in NEW RetroPie Web Console:

Thanks for trying it out and sorry for the slow response. While all of the features you requested are great, I would recommend you add open issues to the project so I can keep better track of them. :P

Ok has you wish ;)

Games/Roms: Working on Cheats next and then will release.

I'm curious lol but what do you want to do with cheat ?
Only cheat we have are RETROARCH cheat, for some systems ...

es_system: Haven't heard of that yet. Will look into it. Got a lead where I can find it?

/opt/retropie/configs/all/emulationstation

If it can help you, my ES file for custom system list and my XML gems list to see path media.

http://www.mediafire.com/file/p9kvgpqdq92vnys/es_systems.cfg
http://www.mediafire.com/file/hzmd20bmh82lmc8/es_gamelists.tgz

MileSacul

I will be adding the ability to not only upload cheat files, but also allow you to modify and/or merge those files. I also found cheats in libretro-database that I will allow you to download cheat files from.

The main reason I wanted to add cheat functionality is for Earthbound. I got tired of fighting enemies to get the really rare drops for the nth time. Anytime I find a reason to use vi to edit something I add it as a potential enhancement to the web console.

darknior

@milesacul said in NEW RetroPie Web Console:

I will be adding the ability to not only upload cheat files, but also allow you to modify and/or merge those files. I also found cheats in libretro-database that I will allow you to download cheat files from.

The main reason I wanted to add cheat functionality is for Earthbound. I got tired of fighting enemies to get the really rare drops for the nth time. Anytime I find a reason to use vi to edit something I add it as a potential enhancement to the web console.

Ok cool, it's why i ask it to you :)
I like CHEAT code too, and there are no many cheat in RETROARCH :(

I have some more from EmuXtra wa cen maybe import to RETROARCH.
And if one day we have an easy way to manually add some more from internet it can be cool :)
Some site have ACTION REPLAY code we can copy paste on a web console.

Here are my cheat code if you want to try them : http://www.mediafire.com/file/kbev46gcegociqr/Cheats_xbox_pour_Darknior.rar

Atari 78000- NEW system
Mattel Colecovision - NEW system
NEC Turbografx - NEW system
Nintendo NES - For update
Nintendo SNES - For update
Sega MAster System - For update
Sega Megadrive - For update
Sony PSX - For update

And the cool feature, for each of them we made RUMBLE code too :)
But for the moment this feature is not implement to RETROARCH :(
So bad because is is really cool to have it on system never have it before ;)

MileSacul

Here are my cheat code if you want to try them

Interesting that you are using .inf files. Is that what you mean by RUMBLE code? The only file type I found that it supports so far are .cht files using this format:

cheat0_desc =
cheat0_code =
cheat0_enable =

I guess I will have to add another parser for reading inf files.

If it can help you, my ES file for custom system list and my XML gems list to see path media.

es_systems.cfg is awesome. I was trying to find something that specifies the file types so I am not blindly searching through folders for games. Just to double check, I found it at /etc/emulationstation, but is there an alternative directory for that?

As for your es_gamelist.tgz, it is also useful to see what options are available.

Bluetooh devices working fine, i can see them but only the Address. Name, Company, Manufacturer are empty.

I found the es_input.cfg file containing the controller UUID. Problem is finding how to tie that into the bluetooth connection details. The good news is, I might be able to support usb connected devices once I figure that out too.

Very interesting you think to saves and states. Don't forgot we can also change the PATH in RETROACH config.

Do you know where those settings are saved to? I will add it to my scripts.

darknior

Interesting that you are using .inf files.

Yes, the emulators on Xbox are using them. And it is not compaatible with RETROARCH :(
But if you can convert them, nobody never do it before, there are no more RA code over the net or i don't found them :(

Is that what you mean by RUMBLE code? The only file type I found that it supports so far are .cht files using this format:

They made them like CHEAT CODE, with a memory adress to survey.
And when it change, the emulator not lock it to give you infinite life, but launch an event.
And this event is to make the controller RUMBLE with different strong !
It is really excellent and a really cool idea :D
I have a friend that add this feature today to Zelda Solaris for future release, so strange nobody never do it on RA.

/opt/retropie/configs/all/emulationstation/es.systems.cfg is the only PATH for me.

I found the es_input.cfg file containing the controller UUID.

Excellent :D I have forgot there is the UUID there.

Do you know where those settings are saved to? I will add it to my scripts.

/opt/retropie/configs/all/retroarch.cfg : Default settings and PATH
/opt/retropie/configs/ $SYSTEM$ /retroarch.cfg : Can be modify here, i change PATH by SYSTEM here.

# Savefile / Savestate directory
savefile_directory = "/home/pi/RetroPie/medias/3do/savefile"
savestate_directory = "/home/pi/RetroPie/medias/3do/savestate"
screenshot_directory = "/home/pi/RetroPie/medias/3do/screenshots"

It can also have file there : /opt/retropie/configs/all/retroarch/config/ $CORE$ /

But it's not for change PATH ... i don't use them.
And it will be hard for you to verify them because it is not a SYSTEM name you can found on the es_system file but the CORE name.

MileSacul

I posted an update to the RetroPie Web Console.
If you already have the RetroPie Web Console, you can execute the self-update through the updates page.
Also, I have added RetroPie-Extra to the update for RetroPie-Setup.

FYI, the monitor page is now unfiltered until I can figure out a good filtering setup, so my apologies if the list is massive.

Regarding Games section, I am working on better support for file types and alternative configurations, but it is not in yet. Cheats upload is not functional yet. Also, deeply nested folders are not supported yet. Once I figure out file types configurations, that will be resolved.

darknior

I posted an update to the RetroPie Web Console.
If you already have the RetroPie Web Console, you can execute the self-update through the updates page.

I think it works but i'm not sure ...
I have also made an git pull and ./updates.sh

Also, I have added RetroPie-Extra to the update for RetroPie-Setup.

I don't see it. you have merge it with Retropie-Setup ?

FYI, the monitor page is now unfiltered until I can figure out a good filtering setup, so my apologies if the list is massive.

I can't see nothing. Before it not works fine all the time, but now i haven't any stats.

Regarding Games section, I am working on better support for file types and alternative configurations, but it is not in yet. Cheats upload is not functional yet. Also, deeply nested folders are not supported yet. Once I figure out file types configurations, that will be resolved.

I will stand you finish this page before testing it.
For the moment it's empty.

MileSacul

@darknior Any chance you can clear your browser cache. ctrl+shift+delete (Only cached files and cookies)

As for pulling updates for Retropie-Extra, I found there is a lag when dealing with the conversion from shell files to the database to the browser. I will look into that.

darknior

@milesacul said in NEW RetroPie Web Console:

Any chance you can clear your browser cache. ctrl+shift+delete (Only cached files and cookies)

The monitor works with the console opened. Because like you write it clean the cache.

MileSacul

I think the games section is almost ready, but I need some opinions on if I should include a feature or not.

Currently, no games are displayed on the page until you select your emulators and then press the search button. For me, a full search takes about 22 seconds to load all 298 games. Should I auto-load from the start or enforce that they select their emulators?