Skyscraper times out on very large videos

sleve_mcdichael

I've been having some trouble getting certain videos to scrape. Others will work fine at the same time but certain titles will timeout on the video over and over again until finally they "just work."

I recently had a particularly troublesome title that took all day of trying. I checked screenscraper.fr on the web interface and the video was there but it took about a minute to load (and I feel like Skyscraper only waits about 30 seconds between "request timed out" messages?) (I don't know if it's relevant or just coincidence but the video finally did download when I got impatient and hit ctrl-C after the first of, usually four timeout messages. Instead of cancelling immediately, it told me to wait a few seconds, then gave me one or two more timeout messages, and finally "video conversion...success!")

When it finally did download the video, I checked it out in my cache and it's easily the largest one in the folder at 6.6mb (most of the rest are like 1-3 mb.) It's like the video is trying to download, but Skyscraper just keeps giving up early, until I get lucky and hit a patch of low traffic where it gets high enough speed to deliver the whole video before it gives up (all the while contributing to the high traffic problem myself by continually trying and failing to scrape the video.)

Is there (or could there be, since I don't think I saw one?) a command-line flag that's like, "try hard" or something and tells it to wait 2 or 3x as long before giving up, for these occasional large uncooperative scrapes?

muldjord

@sleve_mcdichael Skyscraper has a 30 sec request timout. But only for the initial wait for any data to come through. If data has started coming through, the timeout stops counting. So this sounds more like a screenscraper (maybe they close the connection) than a skyscraper issue to me, but I am not 100% sure. If you get the request timed out message from Skyscraper, it means that Screenscraper is taking longer than 30 seconds to even get back to Skyscraper with ANY data. That is problematic. I could up the timeout to 60 seconds. But to be honest, 30 seconds is WAAAAAY too long for any request to return data as it is. And I'm not even sure that fixes the problem, as a 30+ sec request from Screenscraper seems to indicate a different problem entirely. At that point I feel like maybe their internal process has failed and it won't ever bring back data.

In other words: It's very unusual for a network request to take 30+ seconds to respond. And if it did, the chance of the request having failed serverside seems more plausible than waiting 60 seconds would make a difference.

sleve_mcdichael

@muldjord said in Skyscraper times out on very large videos:

But only for the initial wait for any data to come through. If data has started coming through, the timeout stops counting.

That makes sense but the behavior I'm seeing seems slightly different. I wonder if it has to do with this:

I was poking around some more on the web interface (on my phone/tablet, I'm away from my PC currently.) Some videos will start playing immediately, or almost immediately. They buffer for a few seconds and then just start right up. If the blue "playing" progress bar outstrips the gray "buffered" progress bar, the video will pause while more of it buffers before it continues playing again.

On some other videos, like the one that was giving me trouble yesterday, they don't start to play until the buffering is 100% complete. The gray progress bar slowly creeps all the way over to the right, and only after it's 100%, does the video start playing.

Is it possible Skyscraper doesn't "see" whatever is happening until this moment when the playback actually starts? And if that took longer than 30 seconds, is when it gives the "request timed out" error? Because that's sure what it looks like is happening, from my end...

Edit - examples:

Sly Spy (web interface: video playback starts before buffering is complete. Skyscraper: downloads without issue): https://www.screenscraper.fr/gameinfos.php?plateforme=162&gameid=61361

Darius II (web interface: buffering starts almost immediately, but playback does not start until after buffering completes which takes 45 seconds-1 minute. Skyscraper: times out waiting for video. All other medias download except video. One time out of many: the stars align and Skyscraper completes video download without -- or in spite of -- timeout error): https://www.screenscraper.fr/gameinfos.php?plateforme=157&gameid=145321

muldjord

@sleve_mcdichael said in Skyscraper times out on very large videos:

On some other videos, like the one that was giving me trouble yesterday, they don't start to play until the buffering is 100% complete. The gray progress bar slowly creeps all the way over to the right, and only after it's 100%, does the video start playing.

It probably just means it has been encoded in a format that can't be streamed. Some formats require the entire file, before it is able to play it (which is what you are seeing). But the data is still downloaded and the request has been replied many times at that point. The fact that you see the progress bar moving, means it is downloading the file to your browser cache, getting ready to play it as soon as the file has a beginning and end.

I would like to look more into it, but I'm not focused on Skyscraper at the moment. So I'm just laying down my thoughts on the matter.

sleve_mcdichael

@muldjord Yeah, I was about to say that I'm now getting some more videos that do start before fully downloaded in the web interface, but are still timing out on Skyscraper so I guess it's not that.

Most of the time it says no, but once in a while it will say Video: YES (screenscraper) after timing out four times but then still not have downloaded it. All the other medias are still downloaded.

Also I did see a message on the webpage (briefly, it's gone now) about high server activity so maybe that's all this is, but it seems weird (to me, who doesn't know how any of this works) that it just starts right up (downloading at least, if not playing immediately) on my phone but then fails over and over again with the API.

Also, my ctrl-c trick worked one more time, but I haven't been able to replicate it since.

I would like to look more into it, but I'm not focused on Skyscraper at the moment. So I'm just laying down my thoughts on the matter.

Understood, I appreciate even that much.