Versatile C++ game scraper: Skyscraper

cyperghost

@muldjord You're right! Maybe they overcome their descission - time will show. But I think 1000 entries per unique IP are enough for a user.

What is this "queue" thing? I understand this as connection request to their server and with one request you can do 20 actions. So in theory you can retrieve 10.000-20.000 entries per IP - or am I wrong?

muldjord

@cyperghost If I understand it correctly, which I might not, their API can contain up to 20 game results per request. So optimally, if I knew the ID's of the games in their database beforehand, I could requests a comma-seperated list of 20 specific ID's per request. And all of those 20 games would be returned to be in a JSON answer. Problem here being that I do not know the ID's, that's what Skyscraper is trying to figure out. So I'd have to search for the filename one at a time instead, find the best result and its ID, and then fetch the data. So I'd use 2-3 requests per game.

But I might have misunderstood this completely which it seems I have a habbit of doing when it comes to the new API. Pretty embarrasing to be honest.

cyperghost

@muldjord Well ... I think there is a possibilty to get these IDs. I think it's just a checksum of the ROM files (surely rearranged and changed with aretmetics)

Or am I completely wrong this time?

mitu

@cyperghost said in Versatile C++ game scraper: Skyscraper:

Or am I completely wrong this time?

I'd say yes, the ID refers to the (internal) identifier of the game in thegamesdb database, not the hash of the file. You search by a game name (don't know if their API has a 'search by hash' option) and you get a list of games with their IDs.

cyperghost

@mitu Yes you're right ;)
The id for Sonic the Hedgehog is just 5544
So the call to this is only ...
httpx://thegamesdb.net/game.php?id=5544

muldjord

@mitu Correct, they don't currently support hash searches as screenscraper does. The id is just a numeric identifier starting from 0 and going upwards for any new game added it seems.

cyperghost

@muldjord Well I think for a single IP 1000 calls is okay ;)
You can check if data is received and if there is an failure then report to the user ... and I think you're fine with this. Not the best solution to satisfy all needs but good enough to go.

muldjord

@cyperghost Yes, I agree that it should be usable for some minor installations. And obviously it is not a good thing if people are scraping 50000 games at a time no matter what source they are hammering. So I have never been against limits, they are necessary.

I would love it if TheGamesDb would support md5 and sha1 hashes aswell though. Then I could fetch 1 game per request instead of using two request for 1 game. But I think I've worn out my welcome for now, so I'll leave them be and spare myself any further embarrasment.

circo

@muldjord said in Versatile C++ game scraper: Skyscraper:

So I'd use 2-3 requests per game.

You could try stringing them together? As in, first you send the requests for the individual games to get the IDs, then send just a single request for the metadata for every game being scrapped at once. Then you string those together, and you send a single request with the comma-separated IDs.
This could reduce the number of requests to n+1, where n is the number of games that are being scrapped.

muldjord

@circo Thank you for your suggestion, but it's more a question of development time. I would have to rewrite almost the entirety of the scraper system in Skyscraper for this to work as it is not designed for multigame-requests. It's not an overnight change I'm afraid. It's a substantial rewrite that would probably take me months to complete. I do not have that kind of time to work on this project.

circo

@muldjord Ah, I see.
I know from experience that it's difficult to work with APIs that don't fit with the rest :/ But Skyscraper is working really nicely, so at least the rest are cooperating!

chipsnblip

i finally got around to scraping my roms with Skyscraper. a few minor hiccups, but overall it's been really great so far. over the last few days, i noticed quite a few screenshots it was pulling in were not properly cropped, so i would like to share an example of how to autocrop them, for those who are interested.

here's a before & after example:

before

after

it's easy to batch-process a database's artwork after it's been scraped. this requires installing ImageMagick first (apt-get install imagemagick). for example, mogrify -trim /home/pi/.skyscraper/dbs/n64/screenshots/screenscraper/*.*, then it's just a matter of rescraping from the localdb.

i hope this is useful to someone.

muldjord

@chipsnblip Fantastic, thank you for the tip. And that gave me the idea to implement this automatically into my compositor. I can simply look for black borders, and remove them. I'll add that to my todo list. :)

Used2BeRX

@muldjord I didn't even know it was possible to script that. Any chance you could post a standalone script to remove black borders from images?

A few months back I re-shot all 4,236 Title and Action screenshots for NES and FDS to make sure they were all the same size and were taken while using the SONY CSX palette, but just in case I don't want to re-invent that particular wheel every time I tackle a new system down the road, that code of yours might just come in handy. :)

chipsnblip

i wasn't too impressed with the gameboy screenshots that were coming in, so i gave in and downloaded a pack of them from emumovies and imported to my skyscraper database. but something was still a bit off, the grayscale images didn't really fit my theme or look very appealing on my modest 13" CRT screen.

so i set out to make them appear more or less like the puke-green original gameboy using Skyscraper's built in compositor, a simple marquee overlay made in gimp, a snippet of xml, and a dash of bash. the result looks pretty ok, so thought i'd go ahead and share my findings with you all.

Before:
before01
before02

After:
after01
after02

depending on your priorities.xml settings for <screenshot>, you may need to fiddle with the brightness/contrast/etc in the artwork_gb.xml file. in this example i'm using imported screenshots that i downloaded from emumovies, which for the most part are the same ones from screenscraper.fr (i think).

instructions for use:

drop this "marquee" file in /home/pi/.skyscraper/import/marquees/

create a file /home/pi/.skyscraper/artwork_gb.xml and paste the following code into it:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
    <output type="screenshot" width="730" height="480">
        <layer resource="screenshot" x="20" width="433" height="390" align="center" valign="middle">
            <balance red="1" green="-10" blue="0"/>
            <brightness value="20" />
            <contrast value="-45" />
            <rounded radius="10" />
            <stroke width="5" />
        </layer>
        <layer resource="marquee" x="20" width="433" height="390" align="center" valign="middle" mode="hardlight">
            <opacity value="80"/>
            <rounded radius="10" />
            <stroke width="5" />
        </layer>
        <layer resource="cover" height="212" x="0" y="-10" valign="bottom">
            <shadow distance="5" softness="5" opacity="69" />
        </layer>
        <layer resource="wheel" width="250" x="-10" align="right">
            <shadow distance="5" softness="5" opacity="69" />
        </layer>
    </output>
</artwork>

next you can go one of two routes: download this zip package (851.0 kB) containing 1,870 pre-made .png copies of the marquee, which should cover most no-intro rom file names out there, or, open a terminal and run a command that will make copies of the marquee for just your set of roms. here's a basic command that assumes your roms are zip files. if not just exchange both instances of .zip with .7z/.gb:

ls ~/RetroPie/roms/gb | egrep *.zip | sed -e 's/^/cp "gb_marquee.png" "/' | sed 's/.zip/.png"/' > ~/.skyscraper/import/marquees/gb_roms.txt && cd ~/.skyscraper/import/marquees && bash gb_roms.txt && rm gb_roms.txt

the final step of course is to import them into your Skyscraper database, and that's pretty much it.
Skyscraper -p gb -s import --pretend && Skyscraper -p gb -s localdb --nobrackets -a ~/.skyscraper/artwork_gb.xml --region us --updatedb

so feel free to post your results and/or improvements (adding a dot matrix grid to the overlay might look nice..).

i hope this will be useful to someone, even though it barely scratches the surface of what is possible with Skyscraper =)

happy scraping!

muldjord

@chipsnblip Dude, that is friggin' awesome! Glad someone is actually starting to use the compositor :) I haven't seen many do this so far and it really is quite powerful.

muldjord

Skyscraper version 2.5.3 released: https://github.com/muldjord/skyscraper

Added '--allowext' option which will force allow a file extension for the given platform. Thank you to 'herbymachine' for suggesting this
Added 'allowExtension' to the '[homedir]/.skyscraper/config.ini' variables for both 'main' and 'platform' specific sections. This is useful if you wish to permanently add a file extension to all or one platform when scraping
Implemented 'developers' change in 'thegamesdb' API
Implemented 'publishers' change in 'thegamesdb' API
Fixed 'Tags' bug in 'screenscraper' module

So I just wanted this update out there. The 'thegamesdb' module has been updated to include the latest API changes they've made. The API has been quite unstable while they have been working out the kinks, but they say this is the final version.
Also some other stuff in there. You can now add an extension yourself, simply by using '--allowext *.[ext]'. And finally I fixed a bug that made 'Tags' not work with the 'screenscraper' module.

muldjord

Skyscraper version 2.5.5 released: https://github.com/muldjord/skyscraper

Fixes an issue with the 'screenscraper' module which would cause it to not recognize a lot of games due to a region not being recognized by Skyscraper
Changed '--updatedb' to '--refresh'. '--updatedb' still works, but is considered deprecated

Updating is highly recommended if you've had trouble with some games not being recognized by screenscraper even though the filename is in their database. I would like to thank all of the people who reported this issue. It took me a while to realize why it didn't work, so it's a fix that's been a long time coming.

muldjord

2.6.0 on the way. Now with 'mobygames' support. It is extremely slow though. Only scrapes about 100 roms an hour to accomodate their limits. But it is very useful if no other sources provide info.

muldjord

So the MobyGames scraping source implementation is basically done. I did some testing of it last night, and it works well. One thing to note though. As the 360 requests per hour is a KEY limit and not an IP limit, this means that if simultaneous Skyscraper users scrape with 'mobygames' it will probably give back errors.

What are your thoughts on this? I should still implement it, right? I could just give back an error saying something like "Some other Skyscraper user is currently using the 'mobygames' scraping source. Please try again later.". And I will also apply a limit to only scraping 30 games or so per scraping run for the source. Does this seem sensible to you guys?

MobyGames would therefore be a source you could use for a few games that you are having trouble with with the other sources. Not that MobyGames would guarantee a result, but at least then you would have one extra tool in the toolbox to try. :)