[Metadata / game description] feedbacks wanted + idea => Completed!!

Zigurana

@nemo93 said in [Metadata / game description] feedbacks wanted + idea:

I agree re the legal side of things. Perhaps the fair-use may apply? Most of the texts come from sources like MobyGames and Wikipedia (some I do by myself but a tiny part). Images come from various sources mostly Moby and Gamefaqs.

in term of data... this is what I have as we speak for the systems I care about :)

Amiga: gamelist - 110kB, images - 6.40MB, 73 games

FBA: gamelist - 120kB, images - 7.30MB, 66 games

Mame (wip): gamelist - 300kB+, images - 15.3MB+, 230 games

MasterSystem: gamelist - 450kB, images - 33MB, 333 games

Megadrive/Genesis: gamelist - 1.21MB, images - 100MB, 961 games

N64 (need to redo images): gamelist - 400kB, images - 19MB, 395 games

Neo Geo: gamelist - 295kB, images - 15.6MB, 159 games

NES/Famicom: gamelist - 1.90MB, images - 133MB, 1506 games

PC Engine (+SuperGrafx + few CD games): gamelist - 476kB, images - 29.2MB, 328 games

PSX: gamelist - 117kB, images - 4.1MB, 60 games

Sega32X: gamelist - 38kB, images - 2.5MB, 37 games

TOTAL: gamelists - 5.4MB, images - 365.4MB, 4148 games

Next in queue is SNES which is massive as I have 800+ games to review...

Wow, that's one heck of a job! Sign me up for a copy!

Zigurana

I was just thinking about this, and wondered if it would be nice to have a system-info function that you could call directly from the system-select screen (press select maybe?).
It could be populated with info about the hw itself, so production dates, total nr of games, notable games, picture of system + controller, total nr sales etc.
Basically the wiki entry for the system :-) (hmm, or maybe shorter version and add a link to the wiki, to be accessed by a rpi native browser...)

This info could be retrieved from the system's gamelist, reading from a new entry called system.

In any case, it seemed something right up your alley.

nemo93

hey @Zigurana that's a brilliant idea and definitely something I want to see as well!! All I need to know if that's something other people want to :) Anyway I'm still ok to upload to any location my "work" if that helps / makes you and people happy! I don't have any contact with Sselph and don't know how to! If you know feel free to intro me.

Any other suggestions more than welcome. Meanwhile I keep working on SNES :)

Zigurana

Paging dr. @sselph, dr. @sselph to the courtesy phone please

Zigurana

Ok, so I was tinking how to make this project of yours maintainable over time, more of an group effort, and less dependant on a single expert (splendid as your work no-doubt will be).
So here goes another wild idea:

According to this , github wiki pages are just another type of git repository. That means that:

one can batch transfer a lot of files in a single command, and thereby populate a wiki.
the wiki can be maintained by the community, containing info on a (possibly, eventually) complete set of supported games.
we can describe its merits on a wider scale within the context of other systems of that age, an also emulation on the Rpi ecosystem (specific tips and tricks) .

Now, in principle, it should be fairly straightforward to convert an xml file to a collection of marked-down wiki files and vice versa, especially if we stick to a pre-defined set of headers. That way we could use your work, and be off to a running start!

A newer generation scraper would then be able to simply download this data from the wiki (by cloning the repository) , convert it backwards and populate the game-lists. This would reduce our dependance on other websites than github.

sselph

Hi,

It is good to see more people trying to improve the metadata we have available. This reminds me of another effort from recalbox at https://github.com/recalbox/recalbox-gamelists

One thing to keep in mind is that not everyone names things the same. If you stick to using No-Intro names where possible, I have code that can rename things in a standard way. https://github.com/sselph/scraper/tree/master/rename

nemo93

Hello All,

people at Recalbox have made a fine job too!!! that's impressive. Tbh I don't have a preference on how to publish these metadata, I'm fine with either @Zigurana's proposal or Recalbox' way. I'm new to Github and stuff so the easiest for everyone the better.

@sselph: 100% agree. That's a concern of mine I got for some time. I did my best to stick to no-intro names for the rom names wherever possible (exception: MAME, FBA / Neo Geo). The rom names do use no-intro yet the names showing in ES are not the no-intro ones (usually I've stripped the Japan or Europe from the no-intro leaving just the real name of the games). I do confirm that naming all roms for all systems in a consistent way is a must here (stating the obvious!).

let's try to keep the ball rolling on that topic as it seems people are interested :) Meanwhile I keep working on the SNES gamelist \o/

nemo93

hi there, I've finished Mame and FBA (not all games only my favorites for now). And I've redone the PC Engine entirely before starting SNES. Are there still people interested in this "project" ? If so I really need some guidance here on the next steps please. If you want me to post the gamelists somewhere let me know. same for box arts. I could create a PR on Recalbox too if that helps? just let me know :)

Zigurana

Hey! I sure am interested!
It's just that I am travelling Indonesia at the moment, and my access to the forum is limited. I'll be fully online again in a week and a half.
Lets discuss then what the most efficient way forward is. Maybe we can use / reuse the recallbox data in some way.

In any case, I think you should continue only if it is something you like to do. Then, the quality of the work will speak for itself, and will inspire others to pick up where-ever you leave it.

UDb23

@nemo93 Great work you've done!
I'm interested too.

UDb23

@sselph Any chance for mobygames to be included in your scraper as a source and Vic20 as a platform? That would be really appreciated.

SnipedintheHead

I for one am incredibly interested in this idea. I was coming on here to ask how people got such nice box arts, so consistently, across their games, other than just going through them manually.

@nemo93 I would love to get access to those files, if you have any way of sharing them.

nemo93

Ok it's good to see some traction and interest around that topic :) For now I focus on finishing SNES... and it's massive. Oh boy! Yet it's really something I enjoy so not an issue for me to carefully craft those metadata.

Once SNES will be completed I'll do another round to check and review all previously done systems. By then I'm sure @Zigurana will be back from vacations, fully relaxed and fresh with ideas on next steps!!! I have no idea as we speak on best way to share and store the files with you. Also the copyright issue is a serious consideration even if I'm almost certain the fair use could apply here (for both texts and images).

Vic20 is not part of my system list but I guess that I could have a look at it if that makes sense.

My objective is simple: to have as accurate and interesting description for each game + best looking boxarts. Given the limitation of ES and hardware we can't have 4k images (useless anyway right?) so I stick to 600px width which turns out to be satisfactory in most cases. It's good compromise between quality, file size and resource usage.

Keep posting if you're interested and if you'd like some systems to be "manually scraped"!

chavatar

Great to hear - very interested :)
I've been toying around with exactly the same idea recently, but it is a lot of effort. I started to go wobbly after 32x ;) Appreciate the work.

My inferior approach was to use libretro thumbnails and sselph's super scraper to provide meta data and pick up the odd missing image. Been using png format though - not sure about that, but it does offer easy interoperability with lakka as well. Good also to see Recalbox doing this too, will check that out.

Would definitely be a great addition to the project.

I came across the rather wonderful arcade previews on this forum, now that would be awesome for the console stuff.

Riverstorm

@nemo93 This would be amazing! I don't consider a game completely added until it's been scraped. You need that metadata and screenshots. Many times a simple screenshot makes you go oh I remember that game or it can jog a whole lot of memories. As for other info in MAME I always load history.dat and mameinfo.dat.

nemo93

Good to see people's interest! I'm having a hard time with Super Nintendo as the number of games is just insane for that system!!! Yet I'm making progress and I won't let it go before it's 100% done.

By the way if you know some good places to get box arts in good definition for that system that would be great. Especially the Japanese box arts as they're extremely hard to find in decent quality. Thanks!