Versatile C++ game scraper: Skyscraper

AnalogHero

Maybe im wrong about the traffic. It could be a copyright thing aswell.

I found this the best scraper around cause it works on pi and it can combine data from various sources. And the thing that it saves data local saves traffic if you need to rescrape.

It seems @muldjord has enough and strong reasons to pull the plug. Anyway thanks for this great tool. (Would have been a nice addition for retropie-setup with a small gui like sselphs scraper). :(

muldjord

Stay tuned for news. Skyscraper might (MIGHT!) be online again soon'ish. But in a bit of a cut-down state I am afraid... More info when I get through all of the paperwork.

jdrassa

Great news. Regardless of if it comes back, hopefully you can share some details as to what exactly happened that cause you to pull it. At a minimum it would be useful information for the developers of other scrapers like @sselph so that they don't run into the same issues.

muldjord

@jdrassa Over the course of the past few weeks, I've felt like I was walking around a minefield. Sources started contacting me with not so friendly mails to take out support for their sites and I just didn't want to deal with that sort of negativity in a project that's supposed to be fun and helpful. Hence the take-down. I'd like to point out that I completely understand why sites won't allow scraping! It can hit a database hard if overused.

Skyscraper won't be available again until I have some official permission to use each module. And unfortunately that also means that Skyscraper will be back in a very cut-down version... I completely understand this! But it also is pretty demotivating when all I wanted to do was to help people out.

I even implemented the local cache to try and make people reuse the data. But I still can't control how people use it! And I created the local importer so you could get data from your own source text files and image files and so on...

Bottom line: Skyscraper might be back with ONLY the sources I have official permission from. And even then, I need to just trust the users not to overdo the scrapings.

Used2BeRX

That sucks man. I can understand why people want it blocked though, considering the effort put into making the websites and also the bandwidth. I'd imagine "scraping" emumovies would be immediately shut down if all of the movies were being pulled from a source outside of their own site where they can try to entice the user into a paid membership for the higher quality videos.

Like I said, the NES collection I'm working on is looking great, and I'd like to start making videos of my own that conform to a standard length with a title sequence at the end and all have the same volume one day. I'd also like to do this for all of the other major console systems out there.

If there was a way to do this when I've got everything put together that would be easy for users to get access to, I'm open to any suggestions.

I don't know how long it would ever take me to do this though. I've been working odd jobs to make ends meet the last few weeks to buy myself some more time, but this is an insane amount of work and I'm going to need a real job soon if I can't figure out some way of crowd funding the job.

Hopefully I can put out a full NES release that can be distributed to the public by the end of the year though.

jdrassa

@muldjord Thanks for sharing. Hopefully you will be able to relaunch it in some form. I can understand concerns about load, but I feel like scraping is the whole purpose of many of these sites.

screech

As a https://www.screenscraper.fr administrator, and after discussion with the big boss ;) we grant you a second time an official authorisation to use our DB ^^

As we already say, till it's free and open source, you completely are in our philosophy, and you can use all you need from the API ^^

(And more, if you need help, don't hesitate to ask ^^)

muldjord

@screech said in Versatile C++ game scraper: Skyscraper:

As a https://www.screenscraper.fr administrator, and after discussion with the big boss ;) we grant you a second time an official authorisation to use our DB ^^

As we already say, till it's free and open source, you completely are in our philosophy, and you can use all you need from the API ^^

(And more, if you need help, don't hesitate to ask ^^)

Dude, this is awesome! Thank you!!!

paradadf

@muldjord ScreenScraper is the best database anyway!

muldjord

Currently awaiting reply from some of the other sources. I will keep you updated on the progress.

Used2BeRX

@screech said in Versatile C++ game scraper: Skyscraper:

As a https://www.screenscraper.fr administrator, and after discussion with the big boss ;) we grant you a second time an official authorisation to use our DB ^^

As we already say, till it's free and open source, you completely are in our philosophy, and you can use all you need from the API ^^

(And more, if you need help, don't hesitate to ask ^^)

That's pretty awesome of you guys to do that. Don't know why you'd do it, and I can't imagine that many other places are going to give it their blessing. :)

screech

@Used2BeRX : The reason why screenscraper exist is simple ^^ we want to share our work on collecting media and data in every langage and country we can ;) and with the help of the community we have now a great DB open to everyone who want to scrape ;) (again till it's free and open sources).

That's the power of open community project ;) great work by and for everyone thanks to all of you ;)
And muldjord work is a part of this ;) so no reason to refuse his software, even more, it's a good reason to help him ;)

If that's simples words can help other "DB owner" to share their works for this project it's great. I haven't connection with other Scraping DB (I know it's a shame) But if some of you have some, don't hesitate to send them a small message (with kindness ;) no aggressivity).

(for your info, I just check some Stats, we got about 4 millions request a day (with an average loads of 20/30% of the server ressources). And Skyscraper generate only 1 or 2K... so don't worry about the overuse ;) )

muldjord

Things are going rather well. I just got permission to use arcadedb aswell, and even got a full description of their API. Only downside is that I am forced to only using 1 thread when using it, but that is of course completely ok! It's better than not having it at all.

So the list of permissions are growing and it feels good to do it the right way now. Looking forward to getting Skyscraper back up online once all permissions are settled and I have coded the new API connections.

BladeHunter

From reading this thread, I can't believe how far you got by scraping and not using API's (Hat off to you for perseverance though :)).

I use the Screenscraper API a lot and it's amazing (I use it to read against MD5 and CRC and SHA-1 then write back to the DB to help update missing info), you will have a much simpler time building your app when you are using all the API's from the sites. Most of them are really cool about giving out dev access too :).

Most of the Screenscraper API doco is in French so drop me a line if you have any questions, I might be able to help answer them for you, it took me a while to work it out ;).

If you parse in SS login details (The end user, not yours), the SS API will allow you to have as many threads as the user is entitled to. If the user makes a one off donation to SS they get something like 5 threads.

2Play

This post is deleted!

muldjord

@bladehunter Can you give me names of other sites that provide API's that could be useful to integrate into Skyscraper? API's are, of course, always preferred. I currently use the official API's for ScreenScraper and TheGamesDB and I am awaiting details for MobyGames.

BladeHunter

@muldjord said in Versatile C++ game scraper: Skyscraper:

@bladehunter Can you give me names of other sites that provide API's that could be useful to integrate into Skyscraper? API's are, of course, always preferred. I currently use the official API's for ScreenScraper and TheGamesDB and I am awaiting details for MobyGames.

Apologies, I think I gave the wrong impression, I only have experience with SS (Screenscraper). I didn't bother integrating with any others as my goal was always to find one and help build on it (SS being the obvious choice with Retropie and UXS / SSelphs scraper). I built tools that allow me to enhance the SS database from what I can get from other sources. If you are already up and running with the SS API then that's the extent of my experience :).

Keen to chat anyway, we may have some ideas that can help each other. Feel free to drop me a line if you like :).

muldjord

Skyscraper 2.0.0 released: https://github.com/muldjord/skyscraper

Back to basics: Removed several web sources. Now only allows the ones I have explicit permission to use.
Properly implemented official API for 'arcadedb' module
Added scraping module info to output per result but only when using '--verbose'
Added check for unreasonably bad scraping runs, making Skyscraper exit if 30 of 30 files miss from the get-go

We're back in business! As promised Skyscraper has been brought back online. I have removed most of the scraping modules. They will be put back in if I get permission to use them.

Please let me know if you run into problems with this release. I've only done some minor testing myself and everything seems to work well. :)

AnalogHero

@muldjord Nice to see you back in business. Will try it out asap!

muldjord

Currently implementing region support for the 'screenscraper' module. User can define region with '-r' in next release. For instance '-r jp' or '-r de'. ScreenScraper will then make use of this and use the correct rom name, date, description and so on. It will default back to 'wor' and then 'us' if the specified region doesn't exist in the data. Still trying to figure out if I should also let user define language, since region and language aren't the same thing.