runtimetest: add validation of cgroups #93

Mashimiao · 2016-06-03T02:39:52Z

Signed-off-by: Ma Shimiao [email protected]

wking · 2016-06-03T05:04:06Z

cmd/runtimetest/main.go

+func validateCgroupsPath(spec *rspec.Spec) error {
+	fmt.Println("validating cgroupsPath")
+	expectedPath := spec.Linux.CgroupsPath
+	actualPath, err := getThisCgroupPath("devices")


Why are we blessing devices? What if the runtime is using unified cgroups?

Well, We need many default devices to create container. So cgroup device should be hard requirement.
The spec says "The Spec does not support split hierarchy[https://www.kernel.org/doc/Documentation/cgroup-v2.txt]. The cgroups will be created if they don't exist."
unified cgroups may not to be supported?

I think this needs some spec work before we can implement something reliable. I expect we'll want to make this comparison for unified cgroups if the container process belongs to the unified hierarchy, and with (some?) v1 cgroups if the container does not belong to the unified hierarchy.

On Thu, Jun 02, 2016 at 10:31:32PM -0700, Ma Shimiao wrote:

actualPath, err := getThisCgroupPath("devices")

Well, We need many default devices to create container. So cgroup
device should be hard requirement.

But just because you create devices doesn't mean the user is setting
anything in linux.resources.devices 1.

The spec says "The Spec does not support split
hierarchy[https://www.kernel.org/doc/Documentation/cgroup-v2.txt]. The
cgroups will be created if they don't exist." unified cgroups may
not to be supported?

No, that wording is “we don't support per-controller paths” for the
reasons discussed in cgroup-v2.txt 2. It's not about limiting the
cgroup version.

Thanks for your information. I misunderstood the referred link.
I will make more investigation and update the PR later.

wking · 2016-06-03T05:04:15Z

On Thu, Jun 02, 2016 at 07:39:53PM -0700, Ma Shimiao wrote:

runtimetest: add validation of cgrup path

226e9a3 has the “cgrup” → “cgroup” typo in the commit-message still.

cyphar · 2016-06-08T07:28:22Z

I agree with @wking that the runtime-spec still needs some more work around cgroupPath as well as general cgroup handling. I'll work on a PR for this soon. But for the moment we should probably hold off on this PR.

Mashimiao · 2016-06-08T07:35:16Z

@cyphar OK, I will not update the PR until we have a clear status of cgroupPath

Mashimiao · 2016-08-01T03:58:29Z

ping @wking @cyphar @mrunalp

wking · 2016-08-01T04:26:56Z

cmd/runtimetest/main.go

 		validateHostname,
 		validateRlimits,
 		validateMountsExist,
 	}

 	linuxValidations := []validation{
 		validateDefaultFS,
-		validateDefaultDevices,
+//		validateDefaultDevices,


Why comment this out?

Sorry, this is a mistake

cyphar · 2016-08-01T20:52:15Z

I don't understand any of the uses of strings.Replace with paths (nor strings.Contains). If you're trying to convert a path so that it no longer contains leading ../s then you can use libcontainer/utils.CleanPath for that purpose. It cleans both absolute and relative paths in a sane way.

As for replacing : with /, I don't get what you're trying to do (are you trying to generate a systemd slice path from a name?).

wking · 2016-08-01T21:02:51Z

On Mon, Aug 01, 2016 at 01:52:15PM -0700, Aleksa Sarai wrote:

If you're trying to convert a path so that it no longer contains
leading ../s then you can use libcontainer/utils.CleanPath for
that purpose.

I'd rather not introduce a libcontainer dependency, because ocitools
is intended to test runC/libcontainer (among other runtimes). But in
this case, I don't see a need to clean ../ at all, since they are not
mentioned in the current runtime-spec.

I'd be ok landing language in runtime-spec about ‘..’ being an illegal
path component in cgroupsPath, but we don't have that language now, so
ocitools shouldn't do anything special for such paths.

cyphar · 2016-08-02T11:19:54Z

cmd/runtimetest/main.go

+			return fmt.Errorf("Cgroup path expected: %v, actual: %v", *expectedPath, actualPath)
+		}
+	} else {
+		if _, err := filepath.Rel(*expectedPath, actualPath); err != nil {


Basically, this branch should either be a noop or should create a new container and compare the value. Those are the only two things we can safely do here.

It currently warns about the MAY violation, which I think is fine.

Although I think Rel erroring isn't the right condition. We want to warn if the path returned by Rel starts with ...

And “starts with ..” should be “the first path segment (split on the path separator) is ..” not “the path string starts with the two characters ..”.

Mashimiao · 2017-12-05T12:01:52Z

Test all cgroups except devices subsystem for croupv1

Mashimiao · 2017-12-05T12:02:39Z

ping @opencontainers/runtime-tools-maintainers

Signed-off-by: Ma Shimiao <[email protected]>

liangchenye · 2017-12-05T16:54:10Z

validation/util/test.go

+	}
+
+	r.SetID(uuid.NewV4().String())
+	stderr, err := r.Create()


no 'r.start'?

no 'r.start'?

What would you start? The bulk of the container is setup by the time create returns, and the runtime's dummy process is perfect for holding the container open while it is inspected from outside. So I don't think we want a start call in RuntimeOutsideValidate.

wking · 2017-12-05T17:23:51Z

validation/util/test.go

+	}
+
+	if f != nil {
+		if err := f("test"); err != nil {


I don't understand this hard-coded test. Don't we want to pass the bundle path, like we do for RuntimeInsideValidate?

No, here just to pass absolute cgroup path.
I will update PRs to support absolute and relative cgroup path validation

wking · 2017-12-05T17:31:41Z

validation/linux_cgroups_blkio.go

+		if err != nil {
+			return err
+		}
+		lbd, err := cg.GetBlockIOData(path)


I think path should point to the bundle directory, which means we'll need something else for finding the cgroup path. I think we want two tests for absolute cgroupsPath:

Get the container process PID using state, and read in /proc/{pid}/cgroup to get the associated paths. Compare those paths with cgroupsPath to test “the runtime MUST take the path to be relative to the cgroups mount point”, and raise CgroupsAbsPathRelToMount on mismatch.

Then read the cgroup limits from whichever group you read from /proc/{pid}/cgroup (which may or may not be at the correct path), and compare the value with the configured value. Failures here should probably be non-spec warnings, because the spec has no RFC 2119 requirements for cgroup values.

Signed-off-by: Ma Shimiao <[email protected]>

Mashimiao · 2017-12-06T03:17:16Z

Add support for relative cgrouppath test and add cgroup tests related with it

liangchenye · 2017-12-13T06:49:58Z

cgroups/cgroups.go

+
+var (
+	// AbsCgroupPath is absolute path for container's cgroup mount
+	AbsCgroupPath = "/cgrouptest"


Do we need to put these two variables into the cgroups pkg?
Beside these, the PR looks good to me.

We can change it later if there is a better place to leave these.

liangchenye · 2017-12-13T06:59:59Z

LGTM

zhouhao3 · 2017-12-13T07:05:55Z

LGTM

The previous values were giving me: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/.../hugetlb.1GB.limit_in_bytes: permission denied\\\"\"" The previous values are originally from 432615a (add cgroup hugetlb test for runtime, 2017-12-05, opencontainers#93), which doesn't motivate their choice. The new values are copy/pasted from the spec [1] (which doesn't motivate its choice either ;). I've kept something like Alban's comment from 984dbc8 (Fix error messages in validation cgroup tests, 2018-03-14, opencontainers#605) to at least explain where the number comes from. In testing with my local system, the issue seems to be pageSize and not the limit value. That seems to be supported by the kernel docs, which have [2]: hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot. hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages. On x86-64 and powerpc, this option can be specified multiple times interleaved with hugepages= to reserve huge pages of different sizes. Valid pages sizes on x86-64 are 2M (when the CPU supports "pse") and 1G (when the CPU supports the "pdpe1gb" cpuinfo flag). My CPU supports both: $ cat /proc/cpuinfo | grep '^flags' | head -n1 | grep -o ' $pse\|pdpe1gb$ ' pse pdpe1gb but I don't set hugepagesz, and I seem to only get 2M by default. I can get 1GB entries by booting with hugepagesz=1GB. Longer-term, we may want to auto-detect the value(s) currently enabled by the host system, but for this commit I'm hard-coding 2MB. [1]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config-linux.md#example-8 [2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/kernel-parameters.txt?h=v4.16#n1336 Signed-off-by: W. Trevor King <[email protected]>

The previous values were giving me: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/.../hugetlb.1GB.limit_in_bytes: permission denied\\\"\"" The previous values are originally from 432615a (add cgroup hugetlb test for runtime, 2017-12-05, opencontainers#93), which doesn't motivate their choice. The new values are copy/pasted from the spec [1] (which doesn't motivate its choice either ;). I've kept something like Alban's comment from 984dbc8 (Fix error messages in validation cgroup tests, 2018-03-14, opencontainers#605) to at least explain how the limit breaks down. In testing with my local system, the issue seems to be pageSize and not the limit value. That seems to be supported by the kernel docs, which have [2]: hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot. hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages. On x86-64 and powerpc, this option can be specified multiple times interleaved with hugepages= to reserve huge pages of different sizes. Valid pages sizes on x86-64 are 2M (when the CPU supports "pse") and 1G (when the CPU supports the "pdpe1gb" cpuinfo flag). My CPU supports both: $ cat /proc/cpuinfo | grep '^flags' | head -n1 | grep -o ' $pse\|pdpe1gb$ ' pse pdpe1gb but I don't set hugepagesz, and I seem to only get 2M by default. I can get 1GB entries by booting with hugepagesz=1GB. Longer-term, we may want to auto-detect the value(s) currently enabled by the host system, but for this commit I'm hard-coding 2MB. [1]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config-linux.md#example-8 [2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/kernel-parameters.txt?h=v4.16#n1336 Signed-off-by: W. Trevor King <[email protected]>

Mashimiao changed the title ~~runtimetest: add validation of cgrup path~~ runtimetest: add validation of cgrups path Jun 3, 2016

Mashimiao changed the title ~~runtimetest: add validation of cgrups path~~ runtimetest: add validation of cgroups path Jun 3, 2016

wking reviewed Jun 3, 2016
View reviewed changes

Mashimiao force-pushed the runtime-test-cgroup-path-validation branch from 226e9a3 to 946b75e Compare June 3, 2016 05:36

wking mentioned this pull request Jun 10, 2016

config-linux: clarify cgroup requirements opencontainers/runtime-spec#493

Merged

wking mentioned this pull request Jul 22, 2016

Update runtime-spec package #140

Merged

Mashimiao force-pushed the runtime-test-cgroup-path-validation branch from 946b75e to 8d6fcb6 Compare August 1, 2016 03:57

wking reviewed Aug 1, 2016
View reviewed changes

Mashimiao force-pushed the runtime-test-cgroup-path-validation branch 2 times, most recently from e26d31b to 7a670e8 Compare August 2, 2016 09:43

cyphar reviewed Aug 2, 2016
View reviewed changes

Mashimiao added this to the v0.4.0 milestone Nov 1, 2017

wking mentioned this pull request Nov 24, 2017

runtimetest: Make TAP output more granular #308

Merged

Mashimiao force-pushed the runtime-test-cgroup-path-validation branch from 7a670e8 to 5d65c3d Compare December 5, 2017 12:00

Mashimiao changed the title ~~runtimetest: add validation of cgroups path~~ runtimetest: add validation of cgroups Dec 5, 2017

Mashimiao force-pushed the runtime-test-cgroup-path-validation branch 2 times, most recently from 1d98c84 to 3d31feb Compare December 5, 2017 12:29

Ma Shimiao added 8 commits December 5, 2017 21:44

add cgroups package

9a56096

Signed-off-by: Ma Shimiao <[email protected]>

add runtimeOutsideValidate

07118a8

Signed-off-by: Ma Shimiao <[email protected]>

add cgroup memory test for runtime

4a57b0f

Signed-off-by: Ma Shimiao <[email protected]>

add cgroup hugetlb test for runtime

432615a

Signed-off-by: Ma Shimiao <[email protected]>

add cgroup blkio test for runtime

f1e02ff

Signed-off-by: Ma Shimiao <[email protected]>

add cgroup cpus test for runtime

b712995

Signed-off-by: Ma Shimiao <[email protected]>

add cgroup pids test for runtime

72e67e5

Signed-off-by: Ma Shimiao <[email protected]>

add cgroup network test for runtime

6351044

Signed-off-by: Ma Shimiao <[email protected]>

Mashimiao force-pushed the runtime-test-cgroup-path-validation branch from 3d31feb to 6351044 Compare December 5, 2017 13:44

liangchenye reviewed Dec 5, 2017

View reviewed changes

wking reviewed Dec 5, 2017

View reviewed changes

Ma Shimiao added 2 commits December 6, 2017 11:15

update to support relative cgrouppath test

1aebc09

Signed-off-by: Ma Shimiao <[email protected]>

add relative cgroupath test

7c09b4c

Signed-off-by: Ma Shimiao <[email protected]>

liangchenye reviewed Dec 13, 2017

View reviewed changes

zhouhao3 merged commit c3755c1 into opencontainers:master Dec 13, 2017

wking mentioned this pull request Dec 13, 2017

validation/linux_cgroups_*: Generate TAP output (and outside-validation cleanup) #542

Merged

wking mentioned this pull request Apr 6, 2018

validation/linux_cgroups_*hugetlb: Use smaller limits #619

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtimetest: add validation of cgroups #93

runtimetest: add validation of cgroups #93

Mashimiao commented Jun 3, 2016

wking Jun 3, 2016

Mashimiao Jun 3, 2016

wking Jun 3, 2016

wking Jun 3, 2016

Mashimiao Jun 3, 2016

wking commented Jun 3, 2016

cyphar commented Jun 8, 2016 •

edited

Loading

Mashimiao commented Jun 8, 2016 •

edited

Loading

Mashimiao commented Aug 1, 2016

wking Aug 1, 2016

Mashimiao Aug 1, 2016

cyphar commented Aug 1, 2016

wking commented Aug 1, 2016

cyphar Aug 2, 2016

wking Aug 2, 2016

wking Aug 2, 2016

wking Aug 2, 2016

Mashimiao commented Dec 5, 2017

Mashimiao commented Dec 5, 2017

liangchenye Dec 5, 2017

wking Dec 5, 2017

wking Dec 5, 2017

Mashimiao Dec 6, 2017

wking Dec 5, 2017

Mashimiao commented Dec 6, 2017

liangchenye Dec 13, 2017

liangchenye Dec 13, 2017

liangchenye commented Dec 13, 2017 •

edited by caniszczyk

Loading

zhouhao3 commented Dec 13, 2017 •

edited by caniszczyk

Loading

runtimetest: add validation of cgroups #93

runtimetest: add validation of cgroups #93

Conversation

Mashimiao commented Jun 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking commented Jun 3, 2016

cyphar commented Jun 8, 2016 • edited Loading

Mashimiao commented Jun 8, 2016 • edited Loading

Mashimiao commented Aug 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyphar commented Aug 1, 2016

wking commented Aug 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mashimiao commented Dec 5, 2017

Mashimiao commented Dec 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mashimiao commented Dec 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangchenye commented Dec 13, 2017 • edited by caniszczyk Loading

zhouhao3 commented Dec 13, 2017 • edited by caniszczyk Loading

cyphar commented Jun 8, 2016 •

edited

Loading

Mashimiao commented Jun 8, 2016 •

edited

Loading

liangchenye commented Dec 13, 2017 •

edited by caniszczyk

Loading

zhouhao3 commented Dec 13, 2017 •

edited by caniszczyk

Loading