Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running validation tests from a release #610

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Running validation tests from a release #610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7aaee9c
docs: update link to spec
alban Mar 20, 2018
294ca2a
docs: move compliance testing to separate doc
alban Mar 20, 2018
32857b5
docs: update validation test output
alban Mar 20, 2018
25522e1
docs: how to do a release
alban Mar 20, 2018
ea0a3b2
docs: how to run tests from a release
alban Mar 20, 2018
418cabb
docs: Address review comments
Aug 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 1 addition & 88 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,99 +30,12 @@ INFO[0000] Bundle validation succeeded.

## Testing OCI runtimes

The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:

```console
$ npm install tap
```

Build the validation executables:

```console
$ make runtimetest validation-executables
```

Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](docs/command-line-interface.md).
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API.
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](docs/command-line-interface.md)).

```
$ sudo make RUNTIME=runc localvalidation
RUNTIME=runc tap validation/pidfile.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/linux_readonly_paths.t validation/prestart_fail.t validation/hooks_stdin.t validation/default.t validation/linux_masked_paths.t validation/poststop.t validation/misc_props.t validation/prestart.t validation/poststart_fail.t validation/mounts.t validation/linux_cgroups_relative_pids.t validation/process_user.t validation/process.t validation/hooks.t validation/process_capabilities_fail.t validation/process_rlimits_fail.t validation/linux_cgroups_relative_cpus.t validation/process_rlimits.t validation/linux_cgroups_relative_blkio.t validation/linux_sysctl.t validation/linux_seccomp.t validation/linux_devices.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/process_oom_score_adj.t validation/linux_cgroups_relative_hugetlb.t validation/linux_cgroups_cpus.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/linux_rootfs_propagation_unbindable.t validation/delete.t validation/linux_cgroups_relative_network.t validation/hostname.t validation/killsig.t validation/linux_uid_mappings.t
validation/pidfile.t .failed to create the container
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/cgrouptest/hugetlb.1GB.limit_in_bytes: permission denied\\\"\""
exit status 1
validation/pidfile.t .................................. 1/1 315ms
validation/linux_cgroups_hugetlb.t .................... 0/1
not ok validation/linux_cgroups_hugetlb.t
timeout: 30000
file: validation/linux_cgroups_hugetlb.t
command: validation/linux_cgroups_hugetlb.t
args: []
stdio:
- 0
- pipe
- 2
cwd: /…/go/src/github.com/opencontainers/runtime-tools
exitCode: 1

validation/linux_cgroups_memory.t ..................... 9/9
validation/linux_rootfs_propagation_shared.t ...... 252/282
not ok shared root propagation exposes "/target348456609/mount892511628/example376408222"

Skipped: 29
/dev/null (default device) has unconfigured permissions
…
total ........................................... 4381/4962


4381 passing (1m)
567 pending
14 failing

make: *** [Makefile:44: localvalidation] Error 1
```

You can also run an individual test executable directly:

```console
$ RUNTIME=runc validation/default.t
TAP version 13
ok 1 - has expected hostname
---
{
"actual": "mrsdalloway",
"expected": "mrsdalloway"
}
...
…
ok 287 # SKIP linux.gidMappings not set
1..287
```

If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
For example, with [`prove`][prove]:

```console
$ sudo make TAP='prove -Q -j9' RUNTIME=runc VALIDATION_TESTS=validation/pidfile.t localvalidation
RUNTIME=runc prove -Q -j9 validation/pidfile.t
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.03 cusr 0.03 csys = 0.08 CPU)
Result: PASS
```
Use the [runtime validation suite](docs/runtime-compliance-testing.md).

[bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
[config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
[debian-nodejs]: https://packages.debian.org/stretch/nodejs
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
[node-tap]: http://www.node-tap.org/
[npm]: https://www.npmjs.com/
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
[runC]: https://github.com/opencontainers/runc
[runtime-spec]: https://github.com/opencontainers/runtime-spec
[tap-consumers]: https://testanything.org/consumers.html

[generate.1]: man/oci-runtime-tool-generate.1.md
[validate.1]: man/oci-runtime-tool-validate.1.md
63 changes: 63 additions & 0 deletions docs/devel/release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# runtime-tools release guide

## Release process

This section shows how to perform a release of runtime-tools.
The following example assumes we're going from version 0.5.0 (`v0.5.0`) to 0.6.0 (`v0.6.0`).

Let's get started:

- Start at the relevant milestone on GitHub (e.g. https://github.com/opencontainers/runtime-tools/milestones/v0.6.0): ensure all referenced issues are closed (or moved elsewhere, if they're not done). Close the milestone.
- runtime-tools does not use a [roadmap file](https://github.com/opencontainers/runtime-tools/issues/465) but GitHub milestones. Update the [other milestones](https://github.com/opencontainers/runtime-tools/milestones), if necessary.
- Branch from the latest master, make sure your `git status` is clean.
- Update the [VERSION](https://github.com/opencontainers/runtime-tools/blob/master/VERSION).
- Update the [release notes][changelog].
Try to capture most of the salient changes since the last release, but don't go into unnecessary detail (better to link/reference the documentation wherever possible).

Ensure the branch is correct:

- Ensure the build is clean!
- `git clean -ffdx && make && make test` should work.
- Integration tests on CI should be green.
- Check the version of the binaries:
- Check `./oci-runtime-tool --version`.
- Check `./runtimetest --version`.

Once everything is fine:

- File a pull request.
- Ensure the CI on the release PR is green.
- Send an email to the [mailing list][mailinglist] ([example for v0.5.0](https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/iuWpWUai4_I)) and get reviews from other [maintainers][maintainers].
- Once the maintainers agree, merge the PR.

Sign a tagged release and push it to GitHub:

- Add a signed tag: `git tag -s v0.6.0 -m "release v0.6.0"`.
- Push the tag to GitHub: `git push origin v0.6.0`.

Now we switch to the GitHub web UI to conduct the release:

- Start a [new release][gh-new-release] on Github.
- Tag "v0.6.0", release title "v0.6.0".
- Copy-paste the release notes you added earlier in [CHANGELOG.md][changelog].
- Attach the release.
This is a simple tarball:

```console
$ export VER="1.2.0"
$ export NAME="runtime-tools-v$VER"
$ mkdir -p $NAME/validation
$ cp oci-runtime-tool $NAME/
$ cp validation/*.t $NAME/validation/
$ sudo chown -R root:root $NAME/
$ tar czvf $NAME.tar.gz --numeric-owner $NAME/
```

- Publish the release!

- Clean your git tree: `sudo git clean -ffdx`.

[changelog]: ../../CHANGELOG.md
[maintainers]: ../../MAINTAINERS
[mailinglist]: https://groups.google.com/a/opencontainers.org/forum/#!forum/dev
[gh-new-release]: https://github.com/opencontainers/runtime-tools/releases/new
Loading