Game statistics
-
Very cool, this would make an awesome addition to a theme, if it could be incorporated.
-
I have put in more or less the features I want in the tool now. I will probably add some more documentation on how to use it.
If you want to try you can use it like this:$ python3 retro-stats/game_stats.py -f game_stats.log -s nes -c times -m 120 1 Smurfs, The.7z for nes, played 2 times, time played: 0:08:33, avg: 0:08:33, median: 0:08:33 2 Balloon Fight (USA).zip for nes, played 1 times, time played: 0:11:05, avg: 0:11:05, median: 0:11:05 3 Adventures of Lolo (U) [!].nes for nes, played 0 times, time played: 0:00:00, avg: 0:00:00, median: 0:00:00$ python3 retro-stats/game_stats.py -f game_stats.log 1 for kodi, played 2 times, time played: 1:13:47, avg: 0:36:53.500000, median: 0:36:53.500000 2 Wario Land 3 (World) (En,Ja).gbc for gbc, played 3 times, time played: 0:43:21, avg: 0:14:27, median: 0:06:03 3 Crash Team Racing [SCUS-94426].cue for psx, played 1 times, time played: 0:16:20, avg: 0:16:20, median: 0:16:20 4 Balloon Fight (USA).zip for nes, played 1 times, time played: 0:11:05, avg: 0:11:05, median: 0:11:05 5 Wild Arms.cue for psx, played 1 times, time played: 0:10:49, avg: 0:10:49, median: 0:10:49 6 Super Mario 64 (U) [!].z64 for n64, played 1 times, time played: 0:10:31, avg: 0:10:31, median: 0:10:31 7 Smurfs, The.7z for nes, played 1 times, time played: 0:08:33, avg: 0:08:33, median: 0:08:33 8 Crash Bandicoot 3.cue for psx, played 1 times, time played: 0:05:30, avg: 0:05:30, median: 0:05:30 9 Super Smash Bros. (U) [!].z64 for n64, played 1 times, time played: 0:03:27, avg: 0:03:27, median: 0:03:27 10 Super Mario Bros. Deluxe (U) (V1.1) [C][!].gbc for gbc, played 0 times, time played: 0:00:00, avg: 0:00:00, median: 0:00:00 11 Adventures of Lolo (U) [!].nes for nes, played 0 times, time played: 0:00:00, avg: 0:00:00, median: 0:00:00 12 Bubble Ghost (U) [!].gb for gb, played 0 times, time played: 0:00:00, avg: 0:00:00, median: 0:00:00For more information use
-h, ask in this thread or raise an github issue.$ python3 retro-stats/game_stats.py -h usage: game_stats.py [-h] -f FILE [-c CRITERIA] [-s SYSTEM] [-m MINIMUM_SESSION_LENGTH] Calculate some play statistics for your retro gaming optional arguments: -h, --help show this help message and exit -f FILE, --file FILE path to the stats file -c CRITERIA, --criteria CRITERIA which criteria to order by, available options are: total (time), times (played), average (session length), median (session length), defaults to total -s SYSTEM, --system SYSTEM the system you want statistics for, if omitted, will use all systems -m MINIMUM_SESSION_LENGTH, --minimum-session-length MINIMUM_SESSION_LENGTH skip sessions shorter than this number of seconds, defaults to 120 -
@WeirdH said in Game statistics:
this would make an awesome addition to a theme, if it could be incorporated.
I totally agree. I don't know what support themes have to add custom data fields. I guess that is up to the theme, emulation station or pegasus devs et.c. to figure out.
-
@mitu Do you think this would fit somehow into retropie? And maybe you know who else might have a say if it could be part of the system.
Maybe it could fit into a menu entry in the emulation station start menu that lists your statistics? -
@langest If you'd like to add something to RetroPie, just create a Pull Request on the Github's project.
Personally, I think it's nice, but it's not something that I see included in the base image. Don't let this discourage you from developing it further, though :). -
I added support for using skyscraper cache for getting rom names to get a prettier print. However, it is much slower. I haven't profiled the implementation, but I might optimize this at some point. Also, it doesn't seem to work for all the systems. Probably has to do with which files skyscraper uses to calculate the sha1 sum.
If you want to try it:retro-stats/game_stats.py -f ../../RetroPie/game_stats.log -t ~/.skyscraper/cache/Pass the path to skyscraper cache to the
-tparameter. -
@langest I see this as usefull information for the MAME RoWs ;)
Hmmm.... I think this could be made better by using bash dialogs.
So you parse your output to info box and you can scroll and select with joypad.
Then I'm sure you will hit a wider audience. If you can do some coding in python then you can also set some bash scripts ;) -
@cyperghost
Yeah, that could be the next step. You don't really need to rewrite the script in bash though.
You can call the python script from bash and just format the output nicely into an info box. Would be a nice project to learn some bash.Seems like the sha1 sum is different for the arcade games as well. Maybe has something to do with them being ziped.
I think I will try to fix this problem before I create a bash module.Maybe @muldjord can point me in the right direction to figure out how and for which files skyscraper calculates the sha1? (compressed files and .cue are the ones that doesn't work)
-
@langest said in Game statistics:
You don't really need to rewrite the script in bash though.
This wasn't my intention at all ;)
You explainded later the correct situationYou can call the python script from bash and just format the output nicely into an info box. Would be a nice project to learn some bash.
👍
-
@langest Depends on the module. For ScreenScraper it does a sha1, md5, crc32 and exact filename search. If either of these return a unique result, ScreenScraper will hand back the game data. For the other modules it is filename search based. So if you have a good filename, you will have pretty good results aswell. So, for instance, a filename of "abrq10.zip" is a poor name, as it makes no sense to search for. If instead it is "Game_name_1.zip" is will work a lot better. But there are some nifty tricks on top of that which I do. For both mame and scumm names I have a lookup which converts the names such as "Golden Axe" instead of "goldnaxe" (or whatever it's called).
All of the modules are described here, which should provide at least some insight as to how each work when looking up roms.
Lastly, the CD based platforms are notoriously hard to scrape, as they often have multifiles per game and / or are based on .cue files which can vary since they are often user generated. So for these platforms even ScreenScraper is highly reliant on the exact filenames instead. So it will need to exactly match, for instance, "game_name.cue" inside of ScreenScrapers database to work.
-
@muldjord
Thanks for the informative answer.What I am doing is parsing the db.xml files. And in there the entries are stored based on sha1sums. When I use sha1sum on a rom (and when I use it in my python program I get the same result):
$ sha1sum roms/arcade/joust.zip 48c27d4b623b83e3e724ed348b40ff6193b92b1e roms/arcade/joust.zipIt is different from the sum in cache/arcade/db.xml
<resource sha1="ed1455f34143e6781dc443c064ca7d69e4093cff" type="title" source="openretro" timestamp="1562952656836">Joust</resource>Same with .cue (understandably)
$ sha1sum roms/psx/Bishi\ Bashi\ Special\ 2\ \(Japan\).cue 3d1fb64cd75afd208f7eedb3059a0b0a09fbb78a roms/psx/Bishi Bashi Special 2 (Japan).cue<resource sha1="cdd966e49743a5571eeb81d249d6a3abefbfb408" type="title" source="thegamesdb" timestamp="1563723954281">Bishi Bashi Special 2</resource>It works fine for files that are "plain" roms:
$ sha1sum roms/gbc/Wario\ Land\ 3\ \(World\)\ \(En\,Ja\).gbc bb7877309834441fd03adb7fa65738e5d5b2d7ba roms/gbc/Wario Land 3 (World) (En,Ja).gbc<resource sha1="bb7877309834441fd03adb7fa65738e5d5b2d7ba" type="title" source="openretro" timestamp="1563722070330">Wario Land 3</resource>Do you know why this happens?
-
@langest imho the FBN/MAME games or fixed by their names.
pacman.zipis always PACMAN independent from ROM version.About PSX games .... if you check the bin/img file. Did you then get a result?
-
I also figured out why using skyscraper data is "slow". It is because you might be reading and hashing a big file, dreamcast rom et.c.
And the IO is what takes the most time. I have added a function cache to reduce this problem. But it would probably be best to cache all the results on disk so you don't have to recalculate it every time you run restart the program. -
@langest said in Game statistics:
But it would probably be best to cache all the results on disk so you don't have to recalculate it every time you run restart the program.
Isn't Skyscraper already doing this ? You could compare the
timestampwith the file date and rely on Skyscraper's already built cache - of course, for entries which are present there (i.e. without roms added since the last scraping session). -
@langest Ah, you mean the cache rom id's. Yes, they are a mix of sha1 sums of the actual rom data and sha1 sums of filenames, depending on whether the input is a script (or zipped, read on for explanation) or an actual rom. So in the case of .cue files for instance, it is a sha1 of the filename. If the files are more than 50 megs, they are also id'd by the sha1 of their filenames for speed optimization. But please know that the cache rom id's are completely separatate from the checksums I do when using the ScreenScraper module. In fact I was thinking of renaming the "sha1" attribute in db.xml entirely to just me "id" as to avoid confusion. But then that would break backwards compatibility and the db.xml files aren't means to be edited by hand anyway. It should always be edited with
--cache edit.And there's reason behind the madness. I could just id the files by filenames. But some people like to have games in subfolders and potentially have two games with the same filename under the same platform even if the games are different. To avoid this I use the sha1 checksum instead. But of course this doesn't make sense for scripts, which change often. So for those I have to use the sha1 of the filename instead. I also always use the sha1 of the filename for zipped files, since people might unzip them and rezip them. This would also break the id for that game. So yeah, there's some considerations behind the madness, just to let you guys know. I'm not saying it's the best way, but it's kind of locked in now, as to avoid breaking people's caches, and it works well (although slow, as you point out, for the bigger roms) so.
-
@langest said in Game statistics:
I also figured out why using skyscraper data is "slow". It is because you might be reading and hashing a big file, dreamcast rom et.c.
And the IO is what takes the most time. I have added a function cache to reduce this problem. But it would probably be best to cache all the results on disk so you don't have to recalculate it every time you run restart the program.I actually thought about doing this. Do a lookup if the file hasn't changed since last scrape. Something for 4.0 or 3. 5 I guess.
-
@muldjord
That answers my question.
It seems this leaves two options for fixing the issue.- Implement the same hashing and calculation of all the id's as they are implemented in skyscraper so I can use the db.xml as a dictionary.
or - Extend skyscraper to cache rompath -> id/sha1sum in a file given an argument.
I can't see why 1 would be a useful option, that would be double work and I don't want to extend the stats tool to have scraper functionality.
2 makes a lot of sense to me. It opens up the db.xml data to be used with external tools and it would save time if you were to run skyscraper again. I imagine this wouldn't be too complicated to implement.
Is this something you would be able to implement in the near future or should I consider taking a look? - Implement the same hashing and calculation of all the id's as they are implemented in skyscraper so I can use the db.xml as a dictionary.
-
@langest said in Game statistics:
Is this something you would be able to implement in the near future or should I consider taking a look?
I'd go with option 1 in this case. I'm not currently looking for contributions and I'm eager to work on this feature myself as it's an interesting problem (the checksum lookups) with high optimization potential. But I am not able to give you an ETA.
EDIT: Just a thought: Is it not a bit much to make this rely so much on Skyscraper's db.xml (unless it's just one of several lookup methods you plan to implement)? Just my 5 cents.
-
@muldjord
Alright,
Maybe I could suggest adding the rom path to the Resource struct
https://github.com/muldjord/skyscraper/blob/225136b245c1fca4936062f1fb5430691beed283/src/cache.h#L40
You already seem to have it in the GameEntrys path member. And then together with timestamp it would be easy to see if you need to update the resource.
You probably can't save the path to sha1 mapping in the same resource nodes that the rest of the resources use, but it shouldn't be a problem to introduce a new node either in db.xml or in a separate file. -
@langest I promise it will be well thought through and backwards compatible in the sense that it will convert old entries if needed on-the-fly. If anything this has made me realize that now is probably the time to look into this.
EDIT: A file location for the rom is not very useful. If you move your files to a different directory (or a different system entirely, or someone sends you their cache), they need to still be identifiable. This has to be contained to the file itself, so modification time makes sense for this purpose.
I might also change it to always only do the checksum on the first 512 k of data instead of the entire rom. This will make it quite a lot faster to begin with. And combined with the lookup table this will prove a significat optimization. And for anyone following this, I will of course make sure this will convert "old" cache id's on the fly, so it won't break anything.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.