fool build process into thinking sources are mounted with write permissions in order to work around the copy patch

[Atemu]

We have a patch for many generic copy functions to not copy the permissions as all permissions of the sources are the default nix store permissions. This patch has to be rebased for every major android version though and isn't the best of solutions.

An idea I had on this was to somehow fool the build system into thinking the source dirs have write permissions via some mount magic.

[jaen]

Ok, so I actually ended up playing around with it a bit and it's more complicated than it seems.

So the initial solution I mentioned in matrix (to wit, using overlayfs on top of bindfs to allow changing the files; which I guess doesn't really matter for working builds, but would probably be useful for iterating on new versions/flavours) doesn't really work, because it seems and kernel's overlayfs seems to ignore idmapping done by bindfs (which I guess make sense, as FUSE works in userspace), so trying to modify the files end up with a permission error anyway.

Your suggestion to chmod the files fails for a similar reason – to chmod the file you have to either be its owner or a superuser and even though you quack like a root after unshare -m -r, you're not really the same user and /nix/store files end up being owned by nobody/nogroup, so no dice.

I then remembered there's a FUSE-based fuse-overlayfs which probably should work with bindfs idmapping, so I tried that. And it worked — files had proper permissions and could be modified — but was terribly slow. So terribly slow in fact that I waited long enough for any output after the build variant was chosen, that I gave up waiting.

At this point I decided to do away with overlayfs for now (it's only a development nice–to–have after all) and just focus on permissions. Using bindfs only, I managed to get correct permissions in the build. There are two downsides:

• FUSE doesn't really work OOtB in builds, you have to add --option extra-sandbox-paths /dev/fuse=/dev/fuse,
• it's somewhat slower — I didn't want to wait for a full build, but soong's blueprint processing took roughly twice as long (205.08 vs 335.29 seconds); I'd imagine actual build would do similarly (or worse).

At this point I noticed that I have actually rediscovered something robotnix was already doing between 2019 and 2020: https://github.com/search?q=repo%3Anix-community%2Frobotnix+bind&type=commits. It's unclear from the commit messages why it was removed, but I can assume that it could've been the perf impact.

At some later point I remembered hearing about kernel-level idmapping support and decide to see if that could work, but so far no dice. By that I mean that sure, sudo mount --bind -o "X-mount.idmap=u:0:1000:1 g:0:100:1" /nix/store/deadbeef-pname path works perfectly, but trying the same in unshare -r -m nets an Operation not permitted. So far nothing I tried worked, but I'm trying to see if setting the mappings externally to the namespace and only then entering it could work (but if it does, it'll mean somewhat invasive changes to the builder).

But if it doesn't, then I guess the best we can do is what we currently do — that is, patching with chmods after a cp.

EDIT: idmapped kernel mounts also seem out, as they are checked against /etc/subuid/ and /etc/subgid so it seems there's no easy way for them to pretend to be a differentu uid/gid even if nothing tries to write to the file.

[jaen]

Hm, maybe this would work?
https://github.com/nix-community/robotnix/pull/203/files#diff-f4eb7f67d1474c710f28e1fb28333066e0b8f2352ae4da6c05d151d34761fcb0R89-R97

[danielfullmer]

It's been a long time since I was experimenting with this, but I remember trying to do this with both bindfs and using overlayfs + chmod. In both cases the impact on build performance too much, which is why I ultimately abandoned those approaches.

[Atemu]

A super simple hack around that'd obviate patches could be to just inject a cp wrapper into the build that does chmod afterwards.

Ideally check whether the src isn't in the output directory to avoid any potential correctness issues but, honestly, I don't see how it could cause any actual issue.

[Atemu]

@alyaeanyx tried adding a wrapper to the FHSEnv yesterday but it didn't work. Likely the AOSP build process provides its own coreutils that we'd need to patch.

[jaen]

I'm not sure what flavour you're talking about — so it might be different in your case — but at least in the case of GrapheneOS it seems it's exactly the thing I linked above does:

    # hack to make sure the out directory remains writeable after copying files/directories from /nix/store mounted sources
    source.dirs."prebuilts/build-tools".postPatch = mkIf (config.androidVersion >= 13) ''
      pushd path/linux-x86
      mv cp .cp-wrapped
      cp ${pkgs.substituteAll { src = ./fix-perms.sh; inherit (pkgs) bash; }} cp
      chmod +x cp
      popd
    '';

with fix-perms.sh having this content:

#!@bash@/bin/bash

make_writeable() {
    for a in "$@"; do
        if [[ "$a" == ${OUT_DIR}*  ]] && [[ -e "$(realpath "$a")" ]]; then
            chmod u+w -R "$a"
        fi
    done
}

bash -c "exec -a cp $(dirname ''${BASH_SOURCE[0]})/.cp-wrapped $*" && make_writeable "$@"

Not sure if that's the best way to do it, but seemed to work properly last time I touched it.

[Atemu]

Indeed, something like that would work. I had totally missed that.