Skyscraper now officially part of RetroPie, please test

muldjord

@mitu, @2fst4u If cacheFolder is set in config.ini it will cache data there and purge and vacuum data from there aswell unless a separate -d is set on commandline by the script. There is no difference in the cache location between those operations, they all adhere to cacheFolder in config.ini.

EDIT: Just moved my own cache to a non-default location, changed the cacheFolder to point there and ran both vacuum and purge:all. It works as expected.

2fst4u

I'm not able to run the purge command from the script because it only shows one system - GBC, when I have many systems on the USB drive that have been scraped.

Edit: here is my list when I go to "gather resources" or "generate gamelist":

But here is my list when I try to "vacuum" or "purge":

(Excuse the keyboard, sorry)

mitu

@2fst4u Man, those are some big pictures. I realised later that the purge/vacuum actions are only shown in the context of a platform (system) and if you moved the cache before starting scraping, the newly cached platforms will not be available in the selection screen.
You'll have to use the CLI and run the cleanup actions in this case.

I might be able to improve the detection of cache folder, but my reasoning is that if a user is knowledgeable enough to edit the .ini file and move the cache, then they are already familiar with the command line and don't need a GUI.

@muldjord is the cache folder a global options, or can it be also set in a platform section in the configuration file ? I might be able to extract the cache location, but if the .ini can contain more than one location (i.e. default location is X, but for platform amiga it's Y, etc.) , then this would not work.

2fst4u

@mitu unfortunately when I try a manual purge now I get that artwork.xml error I was having earlier. Would that be related to moving the cache too?

mitu

@2fst4u What is the command line you use for purge/vacuum ?

2fst4u

@mitu

sudo /opt/retropie/supplementary/skyscraper/Skyscraper -p mame-mame4all --cache purge:all

mitu

@2fst4u Why are you using sudo ? That's necessary only for running the setup script. Remove it and then re-run the commands normally.
If you use Skyscraper from the command line, don't use sudo.

2fst4u

@mitu well that was easy. I'm so used to needing to use Sudo any time I need to run a command that it just became second nature. I've never come across something like this where it actively hampers it.

Thanks for that. I'm trying to purge my mame cache since I'm getting those buggy video clips too.

muldjord

@mitu The cacheFolderoption can be set globally and per-platform. If it's set both globally and per-platform, the platform one overrides the global one.

When set globally in [main] it's a base folder where the platform subfolders will reside inside, just as it is in ~/.skyscraper/cache. When it's set on the platform level, it's the subfolder you are defining such as ~/.skyscraper/cache/amiga. It would be rare for people to define it on the platform level and you can't know what they might have called it (they could decide to call it amigacache instead of just amiga ). If it's set globally it's a bit more manageable as the subfolders would be equal to the platform shortname definitions (such as amiga).

muldjord

@2fst4u said in Skyscraper now officially part of RetroPie, please test:

@mitu well that was easy. I'm so used to needing to use Sudo any time I need to run a command that it just became second nature. I've never come across something like this where it actively hampers it.

Thanks for that. I'm trying to purge my mame cache since I'm getting those buggy video clips too.

So just as a heads up, I will elaborate a bit on this. :) Never, never, never use sudo unless you absolutely need it. It can have great implications for your permissions on your drive. It will mess with your homedir (which becomes /root temporarily every time you run anything with sudo) and any file that's created when using sudo will be owned by root which can cause them to be locked down for your ordinary user and might not even be able to be read. It will give you problems at some point that are really tiresome to fix and figure out and can, potentially, force you to reinstall your entire system because it can get so bad it will take more time to fix than to just start over (but that's an extreme case of course, but I've seen it happen in non-RetroPie context). :)

mitu

@muldjord So this is probably the reason I didn't implement a cache location detection - the ability to have multiple cache folders. I might be able to add a few modifications though to warn the user or to account for the situations where the global setting is present.

2fst4u

@muldjord thanks for the heads-up, guys. I never knew this, I'll keep it in mind.

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

@muldjord So this is probably the reason I didn't implement a cache location detection - the ability to have multiple cache folders. I might be able to add a few modifications though to warn the user or to account for the situations where the global setting is present.

The global cacheFolder setting is widely used, mostly by CLI users I would guess though. I don't think the platform setting is used all that often, if at all (it doesn't make much sense to use if you ask me).

2fst4u

Another quick question that I can't find the answer to anywhere:

How do I get rid of data that has been scraped? Is purging the cache sufficient? I'm wanting to delete my downloaded video clips completely for a particular platform and then re-do my gamelist. After I purge, does regenerating the gamelist get rid of the videos in my ROMs folder? I've done a purge but the videos are still in my ROMs folder (where I've set the data to be located), which makes sense in a way, I know that data isn't the cache. I'm just curious if the regeneration of the gamelist will delete the old data when it finds it isn't in the cache anymore.

mitu

@2fst4u Skyscraper will not remove the generated artwork (video/images). You can delete it yourself or just re-scrape and the files would be overwritten with the new data from cache.

2fst4u

@mitu oh cool so it isn't as touchy about that stuff as it is with the cache data?

mitu

@2fst4u said in Skyscraper now officially part of RetroPie, please test:

@mitu oh cool so it isn't as touchy about that stuff as it is with the cache data?

Why would it delete the generated data ? That's up to the user to do it. You don't see a browser deleting the downloaded files from the disk when you instruct it to clear the cache.

2fst4u

@mitu true. I'm a big believer in "no stupid questions", hopefully someone else finds my questions and the answers you guys give helpful! Thanks.

Riverstorm

@mitu - I couldn't quite get it to crash like it was the other day but ran out of time. I think I did an update between those times somewhere too though I did get an error in the middle of scraping.

Basically I have 271 games in the Arcade folder. Nothing excessive. I booted fresh and went to the scraper (no games ran or any other activity). I scraped them with thegamesdb with Arcade selected and no other systems. After scraping I viewed each one with the arrow down key.

At the end of the game selection list I pulled up the scraper again and changed the source to the other one (I can't remember the name--but there's only one other). I started a new scrape and around 90 or so I got this error message Error saving resized image. Out of memory? Disk full? I did click Retry and it continued but I had to go before it finished.

I have a 32GB card with about 17GB used and 12GB free. On the Pi 3 b I believe the default is 256MB. It's at the defaults.

I am running a Pi 3 b, RetroPie 4.4.11, RA 1.7.6, ES 2.8.2RP. The ES log had a lot of lvl1 messages similar to below but nothing really useful it seemed. I did see several memory parsing errors when shutting down where you briefly see the command line screen.

lvl1: 	System "vectrex" has no games! Ignoring it.
lvl1: 	  Warning from theme "/etc/emulationstation/themes/carbon/zxspectrum/theme.xml"
  (from included file "/etc/emulationstation/themes/carbon/zxspectrum/../carbon.xml")
    could not find file "" 
lvl1: 	System "zxspectrum" has no games! Ignoring it.
lvl1: 	  Warning from theme "/etc/emulationstation/themes/carbon/auto-allgames/theme.xml"
  (from included file "/etc/emulationstation/themes/carbon/auto-allgames/../carbon.xml")
    could not find file "" 
lvl1: 	  Warning from theme "/etc/emulationstation/themes/carbon/auto-favorites/theme.xml"
  (from included file "/etc/emulationstation/themes/carbon/auto-favorites/../carbon.xml")
    could not find file "" 
lvl1: 	  Warning from theme "/etc/emulationstation/themes/carbon/auto-lastplayed/theme.xml"
  (from included file "/etc/emulationstation/themes/carbon/auto-lastplayed/../carbon.xml")
    could not find file "" 
lvl1: 	  Warning from theme "/etc/emulationstation/themes/carbon/custom-collections/theme.xml"
  (from included file "/etc/emulationstation/themes/carbon/custom-collections/../carbon.xml")
    could not find file ""

muldjord

@Riverstorm This doesn't seem to be related to the Skyscraper scraper script (which this thread is about). Perhaps this is about SSelph's scraper?