I need your help: Web based ROM visualization

[Arcuza]

Hi!

I've created a HTML5 web page for visualizing your ROM set.

On one single page you can see which games you play most/least, which reserves most space on your SD card. Which console you like most, which games you've never played etc.

The thought is to incorporate this into the web based management.

All I need to continue finalizing the project is to try this out on huge amount of data.

If someone please could lend me a statistics file in its raw format, including:

• Game Name (Mario 64)
• Platform (Nintendo 64)
• Platform Manufacturer (Nintendo)
• Day last played (2016-12-05)
• ROM size (6 564 324 bytes)
• Times played (32 times)
• Rating (4.5/5)
• Year released (1996)
• Genre (Platformer)

The more the merrier.

I hope someone has a file containing something like 5000+ games/roms, so I can really load test the code.

Thank you very much! I hope this will be very nice and appreciated by you.

[herb_fargus]

@Arcuza you mean a gamelist.xml for a fully scraped set?

I have one here for mame:

https://github.com/HerbFargus/Mame-Extras/tree/master/mame-2003/snaps

It doesn't have all the tags, mostly just for images and game names but it's a start.

Looking forward to seeing what you come up with

[Arcuza]

@herb_fargus Thank you very much! I'll start with this data and give you a demo tomorrow. Is it possible to gather more meta data? Especially times played, date last played, rom file size and perhaps rating? Doesn't Emulation Station produce some kind of metadata as well?

[daeks]

could you share your webcode with us? I can try with huge gamelists as well.

[MWGemini]

@Arcuza I have about 63,000 ROMS across something like 18 systems, but I have not scraped all of them. In fact, most are not scraped. I haven't yet had the chance to perfect all of my gamelists, because I'm planning/hoping to replace gamelists with a database in the not too distant future.

It should be pretty easy for you to create data to load-test your system, though. Writing XML is pretty easy, so you could just create a program to create a random name and some random data for it, and loop as many times as you want.

[Arcuza]

@MWGemini Thank you for your support. I ran into a problem today when testing Herb Fargus data, and that's dealing with many-to-many relationships. I didn't solve that during the day, but I'm on my way.

The code is already finished when receiving expected data, now I want to test what happens when running into unexpected data.

Thank you

[joyrider3774]

Will it eventually be running from the pi itselve somehow so you can access a webserver on the pi and view all this data ? Do you have some screenshots i wonder how it looks like

[Arcuza]

@joyrider3774 It's dependant on that the web server is able to push the gamelist.xml through a HTTPRequest. All is client side, so all data is processed in the browser. Of course you can run a web browser on the PI, but I think it would be too slow. In that case I would rather make it native on the PI in C++. But I need to learn Linux graphics, io and input since I've never done that before.

I just have to solve null data and finite/rational data filtering first. Then you can have a full blown demo.

[Arcuza]

@herb_fargus Thank you for your help. You don't have a well scraped set with several consoles? Is there one gamelist.xml for each console?

[herb_fargus]

@Arcuza yes as far as I know there is one gamelist.xml per system.

I don't really have massive romsets so I don't really have any lists.

I'm sure there are plenty of others the forum that do though

[herb_fargus]

@Arcuza I found another one for NES

https://drive.google.com/open?id=0B2TMeZ6iEFvHVTEtLWNubk00ZEE

[Arcuza]

@herb_fargus Hi, do these gamelists.xml have different schemas per console, or do they follow a standard? Do anyone know? Is there a document describing this?

[herb_fargus]

@Arcuza they should be the same for every system.

https://github.com/Aloshi/EmulationStation/blob/master/GAMELISTS.md

[Jeck11]

Found this set online. Maybe it'll help ya.

https://dl.dropboxusercontent.com/u/23090041/gamelists.rar

[Arcuza]

Seems like I'm into some nasty mathematics issues, regarding the filters and decimal values.

But, at least you can have a nonworking look here:

http://elitlogik.se/visualizer.html (Data in this demo is based on herb_fargus NES gamelist)

Things that needs to be done:

• Fix handling of null values
• Be able to set custom color schemes and directions (now its green for low and red for high)
• Fix the darn Filters to work better
• Sort the selectables descending, maybe group "others" if less than 1%
• XMLHttpRequest to recieve the gamelist.xml of your own Retropie

In a far future maybe having some nice options when clicking on a game title in the rectangles, like editing, commenting or socializing with your friends on facebook ;)

Do you have any suggestions, please post.

[herb_fargus]

@Arcuza to add a cog into your system emulationstation was just updated recently to include video and marquee tags as well just fyi