README.md: Document cosa via unprivileged podman as primary #722

cgwalters · 2019-08-22T17:27:37Z

The advantages of this are compelling - among others, we don't
need the ACL dance, and it emphasizes how one can do builds as
non-root.

The doc has a lot of sudo podman which I've excised.

A downside of this is it's less documented how to get things
to work with Docker; I didn't try that, but it should be really
easy.

Quiets markdown linter.

Quiet linter.

cgwalters · 2019-08-22T17:33:42Z

The main reason I was working on this is so we can have alt-arch teams just spin up plain bare metal machines with jenkins workers running as non-root and connect to the FCOS Jenkins master in the normal Jenkins way.

Then if we change the pipeline to run podman in this way via plain shell instead of podTemplate, that gives us a fairly decent architecture that supports scheduling builds across architectures while ensuring consistency - (although I think we should still use the Jenkins Kubernetes plugin for other things).

cgwalters · 2019-08-22T17:42:05Z

I mean, this is tricky because on x86_64 we kind of do want to continue using podTemplate rather than manually managing Jenkins slaves probably, but eh, it shouldn't be too hard to abstract. And anyways we want x86_64 builds to go first and only then do altarches if x86_64 is sane, per this plan.

dustymabe

A few comments. Strategy mostly LGTM

README.md

dustymabe · 2019-08-26T00:26:54Z

The main reason I was working on this is so we can have alt-arch teams just spin up plain bare metal machines with jenkins workers running as non-root and connect to the FCOS Jenkins master in the normal Jenkins way.

Then if we change the pipeline to run podman in this way via plain shell instead of podTemplate, that gives us a fairly decent architecture that supports scheduling builds across architectures while ensuring consistency - (although I think we should still use the Jenkins Kubernetes plugin for other things).

Thanks for working on this. Running fully rootless is nice even just in my normal workflow. If it also has side benefits of helping us get to multi-arch support then that's another plus.

dustymabe · 2019-08-26T00:27:18Z

I'll use the documented cosa bash function for a day or two and see if I hit any problems!

Thanks @cgwalters

dustymabe · 2019-08-27T14:14:14Z

README.md

 The environment variables are special purpose:

-
 - `COREOS_ASSEMBLER_PRIVILEGED`


looks like this variable no longer exists

dustymabe · 2019-08-27T17:36:57Z

README.md

+   id=$(id -u)
+   set -x # so we can see what command gets run
+   podman run --rm -ti --security-opt label=disable --privileged \
+              --uidmap=${id}:0:1 --uidmap=0:1:${id} \


since we are using a static user in the container (the builder user with uid=1000) we need this mapping to be --uidmap=1000:0:1 --uidmap=0:1:1000. The other way will work for users if their uid on their laptop happens to be 1000, but won't work otherwise. In my case I am user with uid 1001.

dustymabe · 2019-08-27T17:44:17Z

README.md

+   podman run --rm -ti --security-opt label=disable --privileged \
+              --uidmap=1000:0:1 --uidmap=0:1:1000 \
+              -v ${PWD}:/srv/ --device /dev/kvm --device /dev/fuse \
+              --tmpfs /tmp --tmpfs /var/tmp --name cosa \


i'm getting an error from cosa buildextend-installer because I think it is running out of space in /var/tmp:

supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-1000/appliance.d.4x7eb0jg/root' supermin: ext2: populating from base image supermin: ext2: copying files from host filesystem supermin: error: statvfs: No space left on device: /var/tmp/.guestfs-1000/appliance.d.4x7eb0jg/root libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above libguestfs: trace: launch = -1 (error) libguestfs: trace: close libguestfs: closing guestfs handle 0x558c2687a700 (state 0) libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfsHTcQf9 fatal: Error running command virt-make-fs

[coreos-assembler]$ df -kh /var/tmp/ Filesystem Size Used Avail Use% Mounted on tmpfs 64M 0 64M 0% /var/tmp

seems like this also affects some other commands.. one of them being buildextend-openstack

Hmm. Right, as of recently coretoolbox forwards /var/tmp to the host. That's probably the best bet here.

It seems like podman defaults --tmpfs to 64M for unprivileged users? The desktop toolbox reverted a change to make it larger and seems to have /tmp actually persist in the container root filesystem (bad idea).

remove the --tmpfs lines fixes it for me.

That said...I think it'd be cleaner to move all of our large temporary data into the tmp/ in the builddir so we're using space in just one place.

remove the --tmpfs lines fixes it for me.

Yeah but that will break rpm-ostree unless you have a very new fuse-overlayfs.

Also we want /tmp to actually be a tmpfs since that's part of its "ABI"; updated to bind mount /var/tmp for now like coretoolbox does, but I'll also do a separate patch for TMPDIR.

The advantages of this are *compelling* - among others, we don't need the ACL dance, and it emphasizes how one can do builds as non-root. The doc has a lot of `sudo podman` which I've excised. A downside of this is it's less documented how to get things to work with Docker; I didn't try that, but it should be really easy.

dustymabe · 2019-08-27T21:45:26Z

one more error I'm seeing now with latest testing-devel branch:

Input state hash: 4e85dc27239a9b29bf903debc0b8cbe5da42c17aa290a291e6f58312ea6cbb76
Will download: 8 packages (13.5?MB)
Downloading from 'fedora-coreos-pool'... done
Importing packages... done
error: Importing package console-login-helper-messages-motdgen: llistxattr: Is a directory
+ rc=1
+ set +x

This is without the --tmpfs lines

dustymabe · 2019-08-27T21:52:27Z

This is without the --tmpfs lines

and if I update it to --tmpfs /tmp -v /var/tmp:/var/tmp like is done in the current version of this PR then the issue seems to go away, so ignore that previous message.

dustymabe

LGTM

dustymabe · 2019-08-27T21:55:34Z

I think we can merge this now

cgwalters added 3 commits August 22, 2019 17:16

README.md: Add a header

800ce45

Quiets markdown linter.

README.md: Trim trailing whitespace

3a08629

Quiet linter.

README.md: Lists aren't indented

4492467

cgwalters force-pushed the docs-fixes branch from 4940e34 to d130ace Compare August 23, 2019 18:44

dustymabe reviewed Aug 26, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Show resolved Hide resolved

cgwalters force-pushed the docs-fixes branch from d130ace to 4149869 Compare August 26, 2019 14:19

cgwalters mentioned this pull request Aug 26, 2019

Enable multi-arch pipeline coreos/fedora-coreos-tracker#262

Closed

dustymabe reviewed Aug 27, 2019

View reviewed changes

cgwalters force-pushed the docs-fixes branch from 4149869 to 3e5489a Compare August 27, 2019 17:38

dustymabe reviewed Aug 27, 2019

View reviewed changes

cgwalters force-pushed the docs-fixes branch from 3e5489a to 73df89a Compare August 27, 2019 18:06

dustymabe approved these changes Aug 27, 2019

View reviewed changes

dustymabe merged commit 19ba574 into coreos:master Aug 27, 2019

dustymabe mentioned this pull request Sep 27, 2019

README.md: Add instructions for rootless podman #263

Closed

		The environment variables are special purpose:


		- `COREOS_ASSEMBLER_PRIVILEGED`

README.md: Document cosa via unprivileged podman as primary #722

README.md: Document cosa via unprivileged podman as primary #722

Uh oh!

Conversation

cgwalters commented Aug 22, 2019

Uh oh!

cgwalters commented Aug 22, 2019

Uh oh!

cgwalters commented Aug 22, 2019

Uh oh!

dustymabe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dustymabe commented Aug 26, 2019

Uh oh!

dustymabe commented Aug 26, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dustymabe commented Aug 27, 2019

Uh oh!

dustymabe commented Aug 27, 2019

Uh oh!

dustymabe left a comment

Choose a reason for hiding this comment

Uh oh!

dustymabe commented Aug 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants