NEW RetroPie Web Console

MileSacul

0_1508881196803_Script Update.JPG

MileSacul

I also added these images to the GitHub Wiki

Ganja

This looks Awesome!

grabbing this tonight to test.

herb_fargus

This is probably the 5th retropie web console of sorts? Always good to have variety I guess.

I think it's a tad funny where the readme says " The purpose of this utility is to simplify the setup of RetroPie, which requires command line knowledge." Yet there is more command line work to install the tool than there is to install retropie. If this is meant to be simple, it would probably be beneficial to code an installation module that can be run through the setup script.

Just some constructive criticism, I'm looking forward to seeing the development.

MileSacul

Regarding, 'more command line work' I totally agree with you. While writing the README, I was telling myself that I need to simplify that part too (Hard part is adding to crontab). Once at Beta stage, I will have that figured out.

As for being another RetroPie Web Console, I only was aware of RetroPie-Manager, which it didn't support installing and configuring RetroPie.

I will look into the other web consoles to see if there are more offerings I could potentially add.

cyperghost

@milesacul Just for interest. Why Node.JS? How big is the install of the whole package? Can we compare with a php plattfrom like lighttpd or nginx?

MileSacul

@cyperghost There are several reasons why I went with NodeJS.

I was interested in learning NodeJS in general. The thought of building a web application with javascript sounded amazing.
NodeJS allows npm, which really helps me manage what external tools I am using.
Configuring REST APIs with NodeJS is easier than PHP (Disclaimer, I haven't used PHP in years, so it might have changed)
NodeJS works well with MonogDB, which is a database program that handles JSON objects.

I ran a check on all the apts my tool uses. On individual installs I see this:
nodejs
After this operation, 50.9 MB of additional disk space will be used.
mongodb
After this operation, 208 MB of additional disk space will be used.
libssl-dev
After this operation, 6,480 kB of additional disk space will be used.
libcurl4-openssl-dev
After this operation, 961 kB of additional disk space will be used.
vim
After this operation, 28.2 MB of additional disk space will be used.

As to the full size of my application and all of its dependencies, it totals ~1.3 GB. I verified this by running du-ch before and after running install.sh.

darknior

WOAW i love your project <3
It's always ccol to have new easy interface to manage our PI :)

I have think a moment to take Retropie-Manager and update it with JQuery, because i love it and it is powerful.
But Node.JS is better ... you can do many more things with it :)
Now like write @herb_fargus there are some manager already existing ... we can't use them all ...
Have you plan to include Retropie-Manager functions to your manager ?
To let us with only one manager, more powerful, and with the install script to add it from Retropie-Setup :D
And maybe the last Recalbox manager can also give you some good idées ;)

Thanks a lot

herb_fargus

Recalbox web manager:

https://retropie.org.uk/forum/topic/2303/retropie-manager-web-app-recalbox-manager-fork-mod/

https://github.com/botolo78/RetroPie-Manager

Retropie web gui 1

https://retropie.org.uk/forum/topic/4627/retropie-web-gui/

https://github.com/fechy/retropie-web-gui

Retropie web gui 2

https://github.com/gwhitcher/RetroPie-Web-GUI

Rom visualiser

https://retropie.org.uk/forum/topic/8020/rom-visualizer/

Webtropie

https://retropie.org.uk/forum/topic/10164/webtropie-wip-was-web-app-wip-please-give-it-a-name/

https://github.com/gazpan/WebtroPie

YARman

https://retropie.org.uk/forum/topic/8266/yarman-web-beta

https://github.com/daeks/yarman

Retropie metadata editor

https://retropie.org.uk/forum/topic/4152/retropie-metadata-editor

https://rme.codeplex.com/releases

And now yours :) though there are probably a few others floating around too

caver01

@herb_fargus But you gotta love "Webtropie" for that name alone!

darknior

WOAW idon't know this exist all these Web interface, so fantastic <3
I will try some of them ...
Webtropie and YARman are so excellent :p

Wish will merge all these fantastic WEB UI lol

MileSacul

@darknior As mentioned, "I will look into the other web consoles to see if there are more offerings I could potentially add."

I agree that having 8 different web consoles for managing one application is excessive. The hard part is making this work on Mobile devices.

My biggest goal is the RetroPie Super Web Console. This will essentially be a manager where you can push configurations/ROMS/etc from a single computer to all of your Pis. I already kind of have special hooks for this by utilizing REST APIs. For example, if you go to http://{IPADDRESS}:3000/apps you will see JSON objects of all available packages on your Pi.

cyperghost

@herb_fargus

What are the reasons you think, that all these projects get stuck in development?
What frontend got the best development? (I knew only Recalbox web manager)
I think WebtroPie and YARMAN have both also good development ;)

@MileSacul
About filesize... I compared nginx and lighttpd install via aptitude. They need just round about 1MB and the server setup is usually very fast. Therefore I asked for NODE.JS and it's size. Also the database setting a SQLite* base would be a more common used one (up to now!).

So it's of course your decision what you want to do in your sparetime but if you really want to develop a frontend then fork the farthest developed frontend for now. ~~I think it's the Recalbox web manager as it was one of the first.~~ Maybe herb or other members can get annother idea here what seems the most developed RetroPie-WebUI.

Really no offence just my two cents! As there were so much frontends being in development stages but noone gets finished and it seems for me that all available programming languages were used :)
Thank you for your efforts ;) ... and I hope that one webgui gets finished

EDIT: Read more about webservices of RetroPie here

*I deeply hope that the gameslists and the metadata will be converted in one database like SQLite for portability reasons.

herb_fargus

@cyperghost idk. Everyone has their own idea of doing things and most weren't planning on long term dev or maintenance so much as just doing it cause they could. Some are bloaty, some are lightweight. We all do this in our free time so people pursue things they are interested in.

Anyways I'm interested to see how OP handles it, I welcome any new ideas, regardless of how long they last

cyperghost

@herb_fargus said in NEW RetroPie Web Console:

Anyways I'm interested to see how OP handles it, I welcome any new ideas, regardless of how long they last

Agreed ;)

@MileSacul Give us a shot of coding magic ;)

MileSacul

@cyperghost
While I like the idea of making this thing lightweight, I also have to look at application response times, developer adoption, performance footprint, the lifespan of used technologies.

I used these sites to collect initial thoughts:
HammerPrinciple SQLITE VS MONGODB
StackShare's MongoDB vs SQLLite
Web Technologies Survey
Node vs Apache vs Lighttpd vs Nginx
Apache Vs Nginx Vs Node.js And What It Means About The Performance Of WordPress Vs Ghost
Top 8 Node.js Best Practices For Your Startup

Keep in mind, the only sources I really consider reliable are StackShare and the Web Technologies Survey, but to be honest, finding valid performance reports on these technologies were a failure. I probably have to build them myself if I ever plan to (probably not).

Response times
A lot of analyses showed that Nginx pretty much beats NodeJS as a web server. While Lighthttpd is faster in smaller input/output, it gets slower as you give it more and more requests. Overal, NodeJS gives slower response times as a web server even when clustering.

As for MogoDB vs SQLLight, there were only thoughts on adoption and that both perform equally as well.

Developer Adoption
I was blown away by this statement from "Top 8 Node.js Best Practices For Your Startup:" To summarize, JavaScript is the #1 most-used language on GitHub and it looks like this trend will continue.

Although, the Web Technologies Survey showed that Nginx was only second to Apache and claimed over a third of the market while Node.js only took .3%.

As mentioned before, the fact that I can use Javascript both for my web application and my web server it means I don't have to switch my brain from going from one technology to the other. Especially when going from PHP (which Nginx uses) to JavaScript. Partnered with AngularJS, I can get help from anybody that knows JavaScript. Using one language to rule them all kind of strategy :P.

As for MongoDB vs SQLite, you are right about more people using SQLite. After all, it follows Oracle, DB2, and MySQL in terms of query syntax; I had quite a learning curve figuring out MongoDB because I am used to production databases. Also, only SQLite supports foreign keys, which is something I really like. There are a few pros for MongoDB though:

It stores JSON objects, which means no custom hooks to convert my queries
It allows JSON objects for query search, which yet again means no conversion.
It requires no real database schema. While Mongoose does use a schema, updating it is as simple as changing it in my JavaScript code. In terms of development, that means I don't have to include SQL update scripts with each database schema change.
It allows storing of arrays in a column. All of the functions that you see on the manage page is stored as a single array in a column.

** Performance Footprint **
Even though you can run very intensive operations (opt/exp lr-mame builds from source) with my app, I think keeping the performance footprint low is a good idea. Keep in mind, I designed the web console to always be running. Even when users are playing their video games.

The big thing I see from most of the nginx and NodeJS application is how many threads they run on. NodeJS is designed to run a single thread. This means that it is less likely to cause performance issues, with exception of calling all the install commands because I use NodeJS child_process to essentially run outside of NodeJS on its own.

Ningx creates a new thread for each connection to it. This is troublesome because it means it can be a resource hog when a ton of people are trying to run things. Although I don't expect many people connecting to the Web Console at once, that could change when designing the RetroPie Super Web Console.

As for resource cost of SQLite vs MongoDB, I didn't see any metrics available online, so I will probably have to prototype both of them and do comparisons. Since a lot of the lr-mame builds generate MB worth of log results (that I store in the DB), I can just execute a full install of everything and monitor performance during both operations. Although, I doubt either of them are CPU intensive since they are really based of I/O speeds.

Lifespan of used Technologies
This one is probably the hardest of all. After all, when in high school I used Yahoo as my primary search engine and developed my first web application with NotePad and Netscape browser.

A lot of the reports show that NodeJS is more of a small fry and Nginx is 2nd only to Apache. While I think NodeJS can be considered a solid investment based solely on the fact that it is built off of the Google JavaScript engine, there is a chance that Google can end up like Yahoo. PHP has stood the testament of time.

I will probably have to use GitHub as my primary source for my decision. The trends are pointing to JavaScript, so I should try to at least move in that direction even if NodeJS might be replaced.

As for the databases, SQLite seems like it will stand the testament of time more, solely because it utilizes a lot of the well known database CRUD operations. Not to mention it is used by all Andriod phones. For now, I will continue to use MongoDB and after I get my Schema strategy really down, I will probably move over to SQLite or figure out some sort of hybrid between the two.

Overal Summary
All of them look like they could work as the backbone of my application, but I've actually already spent two months (in my free time) learning NodeJS and MongoDB, then applying them in an application. Maybe once I truly know what I am doing, I will dabble with moving around, but for now, I will stay on the same path.

On a side note, I found some very interesting ideas like using a hybrid Nginx and NodeJS application, since Nginx transfers static files like CSS and images faster.

cyperghost

@milesacul Well I can't say if NODE.JS is the concept of the future. I don't have that deep economical point of view :) Of course the coding hits the mobile sector and Java is leading here.

I think there is no decision needed what PHP-server to use. RetroPie gots the PHP backend already installed. For server setup it's just a small step and you can choose between Apache/NGiNX/lighttpd.

PHP has stood the testament of time.

Yes it's a solid base and great projects are made by this. Just for an example Pi Control to control the Raspberry and to monitor system usage.

I don't know nothing about MongoDB but SQLite is quasi standard and it does not need a dedicated server structure. Therefore I think SQLite is also considerable.

As I'm not a native speaker, it's a bit difficult to explain/comment all facettes you pointed out. I just say thank you for your long and professional explaination of your project. I wish you much fun with this and I hope the result will be a

maintained
convenient
lightweight
intuitive

WEB GUI ;)

MileSacul

So I updated the application to Beta level. I added a new monitor page where you can monitor live statistics including:

CPU Utilization (Overall and for each core)
Memory (RAM)
CPU and GPU Temperatures
File System memory
Application stats including kill commands
Bluetooth devices

I also added execution results to basic and manage pages where you can view the full results.

Finally, I added queue history search page, since basic and manage only show executions over the last 24 hours.

MileSacul

I am working on the Games/Roms section and found out about gamelist.xml

When pulling games, I could utilize this XML file for looking up the game info, but found some issues regarding conflicting scrapers.

Now on page load, I check the roms directory for all of the emulator names (psx,snes,etc), so you don't have to pull the full list of all your games. I also added parsing of the gamelist.xml at page load to help with filtering the emulators, like by played game count.

When I first found gamelist.xml, it included playcount and lastplayed data like:

<game>
    <path>{path}</path>
    <name>{name}</name>
    <playcount>{playCount}</playcount>
    <lastplayed>{date: YYYYMMDDThhmmss}</lastplayed>
</game>

I tried loading a game, save stating, and exiting to see if the playcount or lastplayed had been updated, but it wasn't.

I later found that the XML file was generated by a scraper, so I tried emulationstations' scraper.

I noticed the emulation station scraper completely wiped that data out and replaced it with this:

<game>
    <path>{path}</path>
    <name>{name}</name>
    <desc>{desc}</desc>
    <image>{game photo}</image>
    <rating>{rating}</rating>
    <releasedate>{date: YYYYMMDDThhmmss}</releasedate>
    <developer>{developer}</developer>
    <publisher>{publisher}</publisher>
    <genre>{genre}</genre>
</game>

How/when is playCount and lastPlayed generated?

darknior

Hi
Today i finally try your work :)
Some test and debug already help ;)
It's really a great web interface ... i like the real time Monitor page with node.js :)

To answer your question, i know when we play a game it stock the information in memory.
And ES only update the XML when we leave it, ONLY IF we have check Other Options/Save Metadata on Exit

CPU Utilisation is cool and real time, can we also have an (session) history please ? From the time we open the web page, with the last 20 or more mesures?
When we compile source code with -j3 for exemple, or use so heavy emulator, it can be cool to have a graphic with CPU use and temperatures, maybe in two colors, to see how the pi work. Like this exemple with 4 bars for PROC and one line for TEMP.

alt text

Bluetooh devices working fine, i can see them but only the Address. Name, Company, Manufacturer are empty.
I have try with my 3 different 8bitDo controlers. I see their name in Retropie-Setup or in Retroarch.
Kill Command only show me 3 applications. I can't see Retroach for exemple, or OMXPlayer if they crash and i want to kill them. Have you hard coded the list of software we can see ?
Maybe it can be cool to make a black or white list that we can edit from your console, like that we can choose ourselves.
I think black list is a better choice, and you can give us yours. To be sure to hide systems soft, and see that we use, or new one we don't know.
Kill command is working fine.
Games/Roms section is hide for the moment.
-- What can we do in this section ?
-- Can you use the es_system file to list systems, because we use it to add custom systems like PCEngine CD, OpenBOR, CPS, etc...
-- If you show MEDIA and let us change them like update a cover, try tu use the path we already write in our gamelist.xml
Or give us an input to set : string name="uploadImageDirectory" value="~/medias/${system_name}/box art front"
For Box front, Logo, Video.