Idea: option to use ccache and/or persistent source

psyke83

I'm thinking about writing/submitting some optional scriptmodules for review, aimed at speeding up building.

Idea 1:

Add a module to enable ccache in order to speed up (re)building, at the cost of extra storage needed to maintain the cache. This is an extremely simple tweak that can be achieved with the following:
Install & setup environment variable:

apt-get install ccache
export PATH="/usr/lib/ccache:$PATH"

Add ccache configuration options to easily manage ccache -c/C (cleanup/clear), etc.

Idea 2:

Add a scriptmodule to enable an optional behaviour of gitPullOrClone. Instead of the current behaviour where it will fetch a shallow copy of the source into tmp/build/name, the function will clone its shallow copy into tmp/source/name and set the working directory to unpack the git repository to the regular location, tmp/build/name. When the "clean" command is issued for a module after build completes, it will remove the working directory (tmp/build/name), but leave the git repository (tmp/source/build/name) folder intact, so future builds can fetch updates against the existing repository.
A symlink to the .git folder can be added to the working directory if needed.
Perhaps add another option to the module menu to allow deleting either the build or source folders, so users can manually delete source if desired.

The issue with both of these ideas is that it will require more storage space dedicated for building - but the idea is to make them optional features, disabled by default.

Thoughts?

psyke83

I wrote the ccache scriptmodule in case anybody's interested: https://github.com/psyke83/RetroPie-Setup/tree/ccache_scriptmodule

BuZz

@psyke83 I don't want idea 2 as we already have some code for persistent repos - it just needs a little more work - but for RPI users, they mostly don't want to keep the source hanging around due to limited storage space etc. You can try the __persistent_repos=1 variable to test but it's not a high priority at the moment.

for most users ccache will just use more space and may not be of benefit if using binaries etc - developers can install manually - I don't want to include it in RetroPie-Setup at least at this time (We could include this later in a module I had in mind which would simplify some other stuff such as setting up my dev distcc environment with cross compiler).

psyke83

@buzz,

Thanks for the reply. Yes, I suppose it would only be useful for a small subset of users. I actually have a regular HD permanently connected (for Kodi), and symlink the tmp/ccache folders to it in order to minimize SD wear, as I do a lot of building.

I didn't even start looking at gitPullorClone closely enough to notice the feature you mentioned. I'll check it out - thanks!

psyke83

Looking at it now, I think that you can definitely improve persistent_repos. AFAIK, it's possible to update an existing local shallow clone via git pull (it may not have been possible in the past). This stuff is interesting to me, so I'll do some testing for my own curiosity... ;)

BuZz

@psyke83 There is some information in ticket I think - this was intentional. Supposedly doing a git pull on shallow clone can strain the github servers (a popular project was asked not to do this). Also a full history may be important in these cases anyway, for those that want to keep the source around.