RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Idea: option to use ccache and/or persistent source

    Scheduled Pinned Locked Moved Ideas and Development
    ccachebuildinggitnew feature
    6 Posts 2 Posters 994 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      psyke83 Global Moderator
      last edited by psyke83

      I'm thinking about writing/submitting some optional scriptmodules for review, aimed at speeding up building.

      Idea 1:

      • Add a module to enable ccache in order to speed up (re)building, at the cost of extra storage needed to maintain the cache. This is an extremely simple tweak that can be achieved with the following:
      • Install & setup environment variable:
      apt-get install ccache
      export PATH="/usr/lib/ccache:$PATH"
      
      • Add ccache configuration options to easily manage ccache -c/C (cleanup/clear), etc.

      Idea 2:

      • Add a scriptmodule to enable an optional behaviour of gitPullOrClone. Instead of the current behaviour where it will fetch a shallow copy of the source into tmp/build/name, the function will clone its shallow copy into tmp/source/name and set the working directory to unpack the git repository to the regular location, tmp/build/name. When the "clean" command is issued for a module after build completes, it will remove the working directory (tmp/build/name), but leave the git repository (tmp/source/build/name) folder intact, so future builds can fetch updates against the existing repository.
      • A symlink to the .git folder can be added to the working directory if needed.
      • Perhaps add another option to the module menu to allow deleting either the build or source folders, so users can manually delete source if desired.

      The issue with both of these ideas is that it will require more storage space dedicated for building - but the idea is to make them optional features, disabled by default.

      Thoughts?

      BuZzB 1 Reply Last reply Reply Quote 0
      • P
        psyke83 Global Moderator
        last edited by

        I wrote the ccache scriptmodule in case anybody's interested: https://github.com/psyke83/RetroPie-Setup/tree/ccache_scriptmodule

        1 Reply Last reply Reply Quote 0
        • BuZzB
          BuZz administrators @psyke83
          last edited by BuZz

          @psyke83 I don't want idea 2 as we already have some code for persistent repos - it just needs a little more work - but for RPI users, they mostly don't want to keep the source hanging around due to limited storage space etc. You can try the __persistent_repos=1 variable to test but it's not a high priority at the moment.

          for most users ccache will just use more space and may not be of benefit if using binaries etc - developers can install manually - I don't want to include it in RetroPie-Setup at least at this time (We could include this later in a module I had in mind which would simplify some other stuff such as setting up my dev distcc environment with cross compiler).

          To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

          1 Reply Last reply Reply Quote 0
          • P
            psyke83 Global Moderator
            last edited by psyke83

            @buzz,

            Thanks for the reply. Yes, I suppose it would only be useful for a small subset of users. I actually have a regular HD permanently connected (for Kodi), and symlink the tmp/ccache folders to it in order to minimize SD wear, as I do a lot of building.

            I didn't even start looking at gitPullorClone closely enough to notice the feature you mentioned. I'll check it out - thanks!

            1 Reply Last reply Reply Quote 0
            • P
              psyke83 Global Moderator
              last edited by psyke83

              Looking at it now, I think that you can definitely improve persistent_repos. AFAIK, it's possible to update an existing local shallow clone via git pull (it may not have been possible in the past). This stuff is interesting to me, so I'll do some testing for my own curiosity... ;)

              BuZzB 1 Reply Last reply Reply Quote 0
              • BuZzB
                BuZz administrators @psyke83
                last edited by BuZz

                @psyke83 There is some information in ticket I think - this was intentional. Supposedly doing a git pull on shallow clone can strain the github servers (a popular project was asked not to do this). Also a full history may be important in these cases anyway, for those that want to keep the source around.

                To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post

                Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.