Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

[Metadata / game description] feedbacks wanted + idea => Completed!!



  • After the recent changes done to the amazing themes made by @Rookervik I was a bit frustrated (just a little bit as the themes especially Luminous are gorgeous) to see less room was given to the game description and metadata.

    Over the last months I've spent a considerable amount of time to hand-craft definitive gamelist.xml's for the most popular systems (NES/Famicom, Megadrive/Genesis, PCEngine, SMS, Neo Geo, N64 + popular Amiga, Arcade and PSX games). I got now complete xml's and complete images sets for those systems that I'm ready to share. All games for those systems are present as I wanted to be as exhaustive as possible.

    By sharing those the content and description could be updated regularly for instance, typos corrected, etc. At some point (I'm dreaming right now!!) we could perhaps include those gamelist's into RP as default (not sure about the images) so that people will automatically get info and images whenever they add games. No need to scrape things out anymore (at least for the popular systems listed previously).

    Yet the main obvious question: who cares about games description+metadata? If you have complete desciption (+history+tips) for your games will you enjoy that? Or really am I the only one to care about? :p

    I love gaming history and to have some clues about the games and some details about the game system/mechanic is a must for me. I'm ready to share my files anytime.

    2 examples...
    Megadrive gamelist => http://expirebox.com/download/d0a14fe6afc8ca8edadce7c2bd9c9fe3.html
    NES gamelist => http://expirebox.com/download/1128a5dc428a1af7e57a193238f0f136.html

    UPDATE: I have completed gamelists for the systems above + cover arts for every single game using the best resolution possible.



  • I think this is a wonderfull idea!
    Re including it in the RetroPie packages, I do not see a problem there, (just the text, artwork is probably still copyrighted).
    Maybe team up with Sselph to see if this could be part of the scraper?
    How much data are we talking about here?



  • I agree re the legal side of things. Perhaps the fair-use may apply? Most of the texts come from sources like MobyGames and Wikipedia (some I do by myself but a tiny part). Images come from various sources mostly Moby and Gamefaqs.

    in term of data... this is what I have as we speak for the systems I care about :)

    • Amiga: gamelist - 110kB, images - 6.40MB, 73 games
    • FBA: gamelist - 120kB, images - 7.30MB, 66 games
    • Mame (wip): gamelist - 300kB+, images - 15.3MB+, 230 games
    • MasterSystem: gamelist - 450kB, images - 33MB, 333 games
    • Megadrive/Genesis: gamelist - 1.21MB, images - 100MB, 961 games
    • N64 (need to redo images): gamelist - 400kB, images - 19MB, 395 games
    • Neo Geo: gamelist - 295kB, images - 15.6MB, 159 games
    • NES/Famicom: gamelist - 1.90MB, images - 133MB, 1506 games
    • PC Engine (+SuperGrafx + few CD games): gamelist - 476kB, images - 29.2MB, 328 games
    • PSX: gamelist - 117kB, images - 4.1MB, 60 games
    • Sega32X: gamelist - 38kB, images - 2.5MB, 37 games

    TOTAL: gamelists - 5.4MB, images - 365.4MB, 4148 games

    Next in queue is SNES which is massive as I have 800+ games to review...



  • @nemo93 said in [Metadata / game description] feedbacks wanted + idea:

    I agree re the legal side of things. Perhaps the fair-use may apply? Most of the texts come from sources like MobyGames and Wikipedia (some I do by myself but a tiny part). Images come from various sources mostly Moby and Gamefaqs.

    in term of data... this is what I have as we speak for the systems I care about :)

    • Amiga: gamelist - 110kB, images - 6.40MB, 73 games
    • FBA: gamelist - 120kB, images - 7.30MB, 66 games
    • Mame (wip): gamelist - 300kB+, images - 15.3MB+, 230 games
    • MasterSystem: gamelist - 450kB, images - 33MB, 333 games
    • Megadrive/Genesis: gamelist - 1.21MB, images - 100MB, 961 games
    • N64 (need to redo images): gamelist - 400kB, images - 19MB, 395 games
    • Neo Geo: gamelist - 295kB, images - 15.6MB, 159 games
    • NES/Famicom: gamelist - 1.90MB, images - 133MB, 1506 games
    • PC Engine (+SuperGrafx + few CD games): gamelist - 476kB, images - 29.2MB, 328 games
    • PSX: gamelist - 117kB, images - 4.1MB, 60 games
    • Sega32X: gamelist - 38kB, images - 2.5MB, 37 games

    TOTAL: gamelists - 5.4MB, images - 365.4MB, 4148 games

    Next in queue is SNES which is massive as I have 800+ games to review...

    Wow, that's one heck of a job! Sign me up for a copy!



  • I was just thinking about this, and wondered if it would be nice to have a system-info function that you could call directly from the system-select screen (press select maybe?).
    It could be populated with info about the hw itself, so production dates, total nr of games, notable games, picture of system + controller, total nr sales etc.
    Basically the wiki entry for the system :-) (hmm, or maybe shorter version and add a link to the wiki, to be accessed by a rpi native browser...)

    This info could be retrieved from the system's gamelist, reading from a new entry called system.

    In any case, it seemed something right up your alley.



  • hey @Zigurana that's a brilliant idea and definitely something I want to see as well!! All I need to know if that's something other people want to :) Anyway I'm still ok to upload to any location my "work" if that helps / makes you and people happy! I don't have any contact with Sselph and don't know how to! If you know feel free to intro me.

    Any other suggestions more than welcome. Meanwhile I keep working on SNES :)



  • Paging dr. @sselph, dr. @sselph to the courtesy phone please



  • Ok, so I was tinking how to make this project of yours maintainable over time, more of an group effort, and less dependant on a single expert (splendid as your work no-doubt will be).
    So here goes another wild idea:

    According to this , github wiki pages are just another type of git repository. That means that:

    • one can batch transfer a lot of files in a single command, and thereby populate a wiki.
    • the wiki can be maintained by the community, containing info on a (possibly, eventually) complete set of supported games.
    • we can describe its merits on a wider scale within the context of other systems of that age, an also emulation on the Rpi ecosystem (specific tips and tricks) .

    Now, in principle, it should be fairly straightforward to convert an xml file to a collection of marked-down wiki files and vice versa, especially if we stick to a pre-defined set of headers. That way we could use your work, and be off to a running start!

    A newer generation scraper would then be able to simply download this data from the wiki (by cloning the repository) , convert it backwards and populate the game-lists. This would reduce our dependance on other websites than github.



  • Hi,

    It is good to see more people trying to improve the metadata we have available. This reminds me of another effort from recalbox at https://github.com/recalbox/recalbox-gamelists

    One thing to keep in mind is that not everyone names things the same. If you stick to using No-Intro names where possible, I have code that can rename things in a standard way. https://github.com/sselph/scraper/tree/master/rename



  • Hello All,

    people at Recalbox have made a fine job too!!! that's impressive. Tbh I don't have a preference on how to publish these metadata, I'm fine with either @Zigurana's proposal or Recalbox' way. I'm new to Github and stuff so the easiest for everyone the better.

    @sselph: 100% agree. That's a concern of mine I got for some time. I did my best to stick to no-intro names for the rom names wherever possible (exception: MAME, FBA / Neo Geo). The rom names do use no-intro yet the names showing in ES are not the no-intro ones (usually I've stripped the Japan or Europe from the no-intro leaving just the real name of the games). I do confirm that naming all roms for all systems in a consistent way is a must here (stating the obvious!).

    let's try to keep the ball rolling on that topic as it seems people are interested :) Meanwhile I keep working on the SNES gamelist \o/



  • hi there, I've finished Mame and FBA (not all games only my favorites for now). And I've redone the PC Engine entirely before starting SNES. Are there still people interested in this "project" ? If so I really need some guidance here on the next steps please. If you want me to post the gamelists somewhere let me know. same for box arts. I could create a PR on Recalbox too if that helps? just let me know :)



  • Hey! I sure am interested!
    It's just that I am travelling Indonesia at the moment, and my access to the forum is limited. I'll be fully online again in a week and a half.
    Lets discuss then what the most efficient way forward is. Maybe we can use / reuse the recallbox data in some way.

    In any case, I think you should continue only if it is something you like to do. Then, the quality of the work will speak for itself, and will inspire others to pick up where-ever you leave it.



  • @nemo93 Great work you've done!
    I'm interested too.



  • @sselph Any chance for mobygames to be included in your scraper as a source and Vic20 as a platform? That would be really appreciated.



  • I for one am incredibly interested in this idea. I was coming on here to ask how people got such nice box arts, so consistently, across their games, other than just going through them manually.

    @nemo93 I would love to get access to those files, if you have any way of sharing them.



  • Ok it's good to see some traction and interest around that topic :) For now I focus on finishing SNES... and it's massive. Oh boy! Yet it's really something I enjoy so not an issue for me to carefully craft those metadata.

    Once SNES will be completed I'll do another round to check and review all previously done systems. By then I'm sure @Zigurana will be back from vacations, fully relaxed and fresh with ideas on next steps!!! I have no idea as we speak on best way to share and store the files with you. Also the copyright issue is a serious consideration even if I'm almost certain the fair use could apply here (for both texts and images).

    Vic20 is not part of my system list but I guess that I could have a look at it if that makes sense.

    My objective is simple: to have as accurate and interesting description for each game + best looking boxarts. Given the limitation of ES and hardware we can't have 4k images (useless anyway right?) so I stick to 600px width which turns out to be satisfactory in most cases. It's good compromise between quality, file size and resource usage.

    Keep posting if you're interested and if you'd like some systems to be "manually scraped"!



  • Great to hear - very interested :)
    I've been toying around with exactly the same idea recently, but it is a lot of effort. I started to go wobbly after 32x ;) Appreciate the work.

    My inferior approach was to use libretro thumbnails and sselph's super scraper to provide meta data and pick up the odd missing image. Been using png format though - not sure about that, but it does offer easy interoperability with lakka as well. Good also to see Recalbox doing this too, will check that out.

    Would definitely be a great addition to the project.

    I came across the rather wonderful arcade previews on this forum, now that would be awesome for the console stuff.



  • @nemo93 This would be amazing! I don't consider a game completely added until it's been scraped. You need that metadata and screenshots. Many times a simple screenshot makes you go oh I remember that game or it can jog a whole lot of memories. As for other info in MAME I always load history.dat and mameinfo.dat.



  • Good to see people's interest! I'm having a hard time with Super Nintendo as the number of games is just insane for that system!!! Yet I'm making progress and I won't let it go before it's 100% done.

    By the way if you know some good places to get box arts in good definition for that system that would be great. Especially the Japanese box arts as they're extremely hard to find in decent quality. Thanks!



Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.