Versatile C++ game scraper: Skyscraper

muldjord

@analoghero Everything is appreciated mate :)

AnalogHero

@muldjord For some strange reason patreon cant process my pledge. I tried several times adding and removing my mastercard my paypal account. Is there any other way i can send you money? Paypal maybe?

muldjord

@analoghero I appreciate the sentiment but please don't bother too much with it. I appreciate the help you've done with bug hunting and helping other users a lot more than monetary support. :)

bighoffa

@muldjord I am currently using advmame for a few roms (mostly goldentee). I have the roms in the mame-advmame directory with the rest in the arcade dir. Is there a way to scrape the advmame games from the mame-advmame directory without messing up the db or already scraped info from the arcade directory?

bighoffa

@bighoffa said in Versatile C++ game scraper: Skyscraper:

arcade

Please ignore my question. I discovered how to use runcommands and specify the emulator to run for each rom so I can put everything in the arcade directory now.

AnalogHero

@muldjord A few days ago i did a complete new retropie setup as a birthday present. Just a few classic systems without video previews. Did everything in simplemode and cant report any bugs or flaws.

muldjord

@analoghero Glad to hear it :)

muldjord

Working on improving the attractmode export module. Currently testing game descriptions implementation with user qqplayer. Requests for further attractmode improvements should come now, as this is what I am focused on. So let me know if you have anything that should be supported.

muldjord

Skyscraper version 2.4.6 released: https://github.com/muldjord/skyscraper

Added 'overview' support for AttractMode. It will now create the necessary cfg files to show the game descriptions
Added '<kidgame>bool</kidgame>' output to Emulationstation gamelist generation
Added 'ages' support in 'screenscraper' module. Will convert PEGI and ESRB to numeric
Added 'ages' support in 'thegamesdb' module. Will convert PEGI and ESRB to numeric

The 'kid friendly release'. As per user request, I've implemented the <kidgame> Emulationstation tag and added classification scraping support for both the 'thegamesdb' and 'screenscraper' modules. This means that you can now make use of the "kids" mode in Emulationstation, which will then hide all non-kid-friendly games from the lists.

IMPORTANT! If you want to scrape the 'ages' on games you already have cached, you have to scrape using the '--updatedb' flag. Otherwise it will just scrape with the pre-2.4.6 cached data which doesn't contain the ages resources.

Lastly, I've also added 'description' support for the 'attractmode' frontend module. I'm not an Attract-Mode user myself, but I've done basic testing on it, and user 'qqplayer' has verified that it works as expected.

:)

bleuge

@muldjord I've seen this very nice and complete MSX db : https://www.msxgamesworld.com/index.php
maybe you want to check it for scraping.
Thanks as usual!

muldjord

This post is deleted!

muldjord

[deleted, makes no sense]

AnalogHero

@muldjord Thats bad news. Good that there is still some other good sources left.

Used2BeRX

@muldjord

Hey bud,

I know it's been a long time since we talked. I'm still plugging away at my NES project.

I can't remember if we discussed it at all back in the day, but I was wondering if you'd be interested in scraping off of my synopsis files whenever I finally release them? I'd be putting them in several places, but I do think that github would probably be the best for something like this. I still haven't attempted uploading anything there myself, but I've seen archives other people have made with artwork and such that would seem to be a perfect fit with my work and the RetroPie experience.

I eventually plan on doing all console and handheld systems between Odyssey II and the SNES/Genesis era, but who knows whenever I'll actually complete them all.

So far, 2,118 unique NES and FDS games are accounted for, including all complete translations, pirates, unlicensed, English Japanese games, Europe games that weren't released in the US, and hundreds of game hacks. I probably won't be adding many more at this point except for all new translations that come out before my release as well as any interesting looking hacks.

I've still got a lot of work to do and I think I want to release everything at once, so I can't even give a timeframe of when I'd make this available to you, but you're welcome to scrape it when it's out there. :)

muldjord

Thank you, it's appreciated. The problem here being finding a place where I can scrape from. I am almost certain I am not allowed to bash away on the github servers for this purpose (which I will respect of course), so people would have to clone your files themselves and scrape from it using the import module somehow.

We can look into the options when you have it online somewhere.

muldjord

[deleted, makes no sense]

mitu

@muldjord said in Versatile C++ game scraper: Skyscraper:

As I mentioned before, it won't work without an api key

So each user will need an API key, they don't have per-app app keys ?

muldjord

[Deleted, too much negativity.]

Bottom line: 'thegamesdb' is rendered useless for automated scrapers with the new api, since they decided to go with a developer api limit instead of a user request limit.

I don't even know why I am implementing the new api. It won't work for anyone except myself...

cyperghost

@muldjord I read your comment (even the deleted one, with too much negativity)

Well you're right with your expressions. But there would be annother way. Maybe with your API you can ask for unlimited access, just write the guys from thegamedb and tell them what is your project for. To be true, I'm the wrong person here to talk to. Because I never scraped any game ;) I just use the bare filestructure.

If this won't work then consider with every closed door a new one opens... Why not building your own database? You have a mass of users here that will likely help to build this. But on the other hand there are also a bunch of other databases ;)

muldjord

@cyperghost I've just created a post on their forums, asking them about their stance on automated scrapers. Their answers will basically decide the fate of the 'thegamesdb' module in Skyscraper. We'll see...

EDIT: I keep asking myself why I keep working on this project. There's so much negativity related to using these databases. It's always a feeling of being unwanted. The only site that implemented a great solution for this is screenscraper. They basically let their users earn requests, either by paying or by updating the database. It's a true community effort.