No videos on skyscraping

hawkes_84

Hi there

Pi Model or other hardware: pi 4 model B
Power Supply used: official
RetroPie Version Used : 4.7.1
Built From: Pre made SD Image on RetroPie website
USB Devices connected: none
Error messages received: none
Attachment of config files: N/A

I tried to scrape (Skyscraper) for some Megadrive games forgetting to toggle "download videos" option to "enable" and it only downloaded the screenshots and info as expected . I tried again after having enabling video downloads and it wouldn't download the video at all just stating "NO" . Videos scrape fine for other systems but I feel like because I already tried for Megadrive and got it wrong it doesn't give you the chance to do it again? Does anyone know what the issue could be?

muldjord

@hawkes_84 You need to enable refresh. Then do it again.

sleve_mcdichael

@muldjord said in No videos on skyscraping:

@hawkes_84 You need to enable refresh. Then do it again.

Or disable “only missing”? Doesn’t refresh just force a re-download of already-cached media? Shouldn’t prevent a video run for already-scraped roms with only images, I would think, but “only missing” would do that, if I understand.

hawkes_84

@sleve_mcdichael @muldjord thanks both. Where am I finding either of these commands? do I have to edit the Skyscaper config file or something similar?

hawkes_84

@hawkes_84 no worries. Found it

muldjord

@sleve_mcdichael No, onlymissing would skip the files he wants to scrape.

He should enable --cache refresh or simply --refresh (they are the same). And for optimization he could also set the skipexistingcovers and similar flags as documented here. That would stop it from re-downloading the image artwork that he already have cached.

And, of course, he should also enable the --flags videos flag.

sleve_mcdichael

@muldjord said in No videos on skyscraping:

@sleve_mcdichael No, onlymissing would skip the files he wants to scrape.

Well, yeah. That’s why I suggested disabling it might be the solution. It’s not downloading the videos, even though the “download videos” option is enabled. The videos aren’t already cached, but the other media are. Is this not “only missing” behavior? (1)The videos are not cached but the images are; (2)since some media is already cached, the roms are skipped (only missing); (3)the roms are skipped, so no videos are downloaded to the cache.

He should enable --cache refresh or simply --refresh (they are the same).

I realize I am arguing with the creator, so it’s entirely likely that I am the one who has misunderstood. What does “refresh” do? I was of the impression that it will force a re-download of already-cached media, which would only be necessary if the specific media had already been cached. OP has not yet cached the megadrive videos, therefore should not need to “refresh” them, yeah?

Does it not, by default, pick and choose any missing bits and pieces (among the media types chosen for each run) to fill out the cache on subsequent runs when neither “refresh” nor “only missing” are enabled? If I only cache covers, and then I come back later and want to cache marquees and screenshots, do I need “refresh”?

muldjord

@sleve_mcdichael

Well, yeah. That’s why I suggested disabling it might be the solution.

Sorry, I misread that. Yes, removing that flag is necessary under all circumstances in this case.

What does “refresh” do?

Refresh is explained here. If a rom already has data, it will always be loaded from the cache if refresh is not enabled.

Does it not, by default, pick and choose any missing bits and pieces (among the media types chosen for each run) to fill out the cache on subsequent runs when neither “refresh” nor “only missing” are enabled?

I would love for it to do that, but due to design of both the scraping sources and Skyscraper that is not currently how it works.

hawkes_84

@muldjord @sleve_mcdichael believe or not I had to action both your advice for it to eventually work. Not sure why. I'm guessing if it's already tried to scrape something and failed it won't succeed unless you enable refresh?

Does the "purge" option just delete all scrape content from the chosen console then?

muldjord

@hawkes_84

Does the "purge" option just delete all scrape content from the chosen console then?

Yes. It will wipe the cache for the platform. You shouldn't ever need to use it as refresh enabled (and onlymissing disabled) will do the same thing.

sleve_mcdichael

@muldjord said in No videos on skyscraping:

Refresh is explained here. If a rom already has data, it will always be loaded from the cache if refresh is not enabled.

So, on a per-game basis, it’s either all (refresh, or no data exists) or nothing (some data exists in cache)? I guess I’m beginning to understand. A suggestion, then: unless I’m still confused, --refreshisn’t necessarily “only” for when data has been changed at the source, as suggested in the docs, but also when you need to collect previously-uncached data from an already-scraped source (like if you cache images at first and then only later decide you want videos too), yes?

So then, what about onlymissing? The default behavior is sounding a lot like what I thought onlymissing was supposed to do. Does it have any effect, then, when re-scraping from the same source, or really only when using a secondary source?

muldjord

@sleve_mcdichael --refresh and --flags onlymissing are very much related. onlymissing was introduced due to users complaining about having to see the data from all of the already cache entries in the terminal output.

EDIT: Also they should never be used together, as it will end in a regular boxing match between them.

sleve_mcdichael

@muldjord I think I get it. Default will skip roms that have any piece of data from this source, and “only missing” will skip roms that have any piece of data from any source?

I also want to thank you for making this project available and for helping me understand, and I’m sorry for hijacking the thread, especially after you had already posted the correct answer in the first reply :-S

muldjord

@sleve_mcdichael

Default will skip roms that have any piece of data from this source, and “only missing” will skip roms that have any piece of data from any source?

Yes, I had actually forgotten that point, it's quite an important distinction.

And you're welcome. Have fun with it!