Skip to content

Comments

kola: Add run-ext-bin command#1215

Merged
openshift-merge-robot merged 2 commits intocoreos:masterfrom
cgwalters:kola-run-ext
Mar 14, 2020
Merged

kola: Add run-ext-bin command#1215
openshift-merge-robot merged 2 commits intocoreos:masterfrom
cgwalters:kola-run-ext

Conversation

@cgwalters
Copy link
Member

@cgwalters cgwalters commented Mar 5, 2020

Related: #1159

I want to start converting some of the ostree/rpm-ostree
tests to using kola more "natively". This is part of that; the
run-ext-bin command injects a single binary into the target
system, driving it via a systemd unit to completion.

There are further elaborations of this, including injecting
a full container, etc. (That is cheap in the local qemu
case particularly once we have virtio-fs)

@cgwalters
Copy link
Member Author

cgwalters commented Mar 5, 2020

Example:

walters@toolbox /s/w/fcos> rm tmp/kola -rf; kola run-ext-bin --output-dir tmp/kola --cosa-build builds/latest/x86_64/meta.json  /usr/bin/true
walters@toolbox /s/w/fcos> rm tmp/kola -rf; kola run-ext-bin --output-dir tmp/kola --cosa-build builds/latest/x86_64/meta.json  /usr/bin/false
Error: kolet failed: Error: Unit kola-runext.service exited with code 1
2020-03-05T00:39:27Z cli: Unit kola-runext.service exited with code 1: Process exited with status 1
walters@toolbox /s/w/fcos>

arithx
arithx reviewed Mar 10, 2020
View reviewed changes
arithx
arithx reviewed Mar 10, 2020
View reviewed changes
cgwalters added a commit to cgwalters/ostree that referenced this pull request Mar 11, 2020
@cgwalters
WIP: use kola run-ext-bin
17b3a82 
I want to port our tests to use CoreOS (and coreos-assembler)
for obvious reasons.

This depends on
coreos/coreos-assembler#1215

There's a lot of things to clean up after this but at least this
works for me:

`./tests/installed/kola.sh --cosa-build /srv/walters/rhcos-4.4/builds/latest/x86_64/meta.json --output-dir /tmp/kola/`

Note this only works on RHCOS since we depend on Python.
@cgwalters cgwalters mentioned this pull request Mar 11, 2020
Merged
@cgwalters
Copy link
Member Author

cgwalters commented Mar 11, 2020

OK rebased, fixed various things, and now has a demonstration:

ostreedev/ostree#2025

I'm really really looking forward to using this more since ostree's CI in particular is in a badly embarrassing state, and this will allow us to fix that while still keeping the ostree tests inside its project.

@cgwalters
Copy link
Member Author

cgwalters commented Mar 11, 2020

In this most recent iteration I added support for --extdata which allows uploading a binary and a directory of dependencies, which is pretty general. In fact I think we could rewrite the RT kernel test using this pretty easily.

@cgwalters cgwalters force-pushed the kola-run-ext branch 2 times, most recently from d498855 to 8137225 Compare March 11, 2020 21:29
jlebon
jlebon reviewed Mar 12, 2020
View reviewed changes
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely a cool idea worth exploring! Just a few comment.

@cgwalters
kola: Add run-ext-bin command
f2c9e04 
Related: coreos#1159

I want to start converting some of the ostree/rpm-ostree
tests to using kola more "natively".  This is part of that; the
`run-ext-bin` command injects a single binary into the target
system, driving it via a systemd unit to completion.

There are further elaborations of this, including injecting
a full container, etc.  (That is cheap in the local qemu
case particularly once we have virtio-fs)
@cgwalters
Bump vendor
699ec4d
@jlebon
Copy link
Member

jlebon commented Mar 14, 2020

/lgtm

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 14, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit ce5a64f into coreos:master Mar 14, 2020
cgwalters added a commit to cgwalters/ostree that referenced this pull request Mar 18, 2020
@cgwalters
tests: Rework tests/installed → tests/kola
40d8587 
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
coreos/coreos-assembler#1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
coreos/coreos-assembler#1252 (comment)

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.
cgwalters added a commit to cgwalters/ostree that referenced this pull request Mar 18, 2020
@cgwalters
tests: Rework tests/installed → tests/kola
f30b528 
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
coreos/coreos-assembler#1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
coreos/coreos-assembler#1252 (comment)

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.
cgwalters added a commit to cgwalters/ostree that referenced this pull request Mar 18, 2020
@cgwalters
tests: Rework tests/installed → tests/kola
56f85fc 
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
coreos/coreos-assembler#1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
coreos/coreos-assembler#1252 (comment)

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.
cgwalters added a commit to cgwalters/ostree that referenced this pull request Mar 19, 2020
@cgwalters
tests: Rework tests/installed → tests/kola
b93180a 
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
coreos/coreos-assembler#1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
coreos/coreos-assembler#1252 (comment)

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.
@jlebon jlebon mentioned this pull request Apr 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jlebon jlebon jlebon left review comments

@dustymabe dustymabe Awaiting requested review from dustymabe

@lucab lucab Awaiting requested review from lucab

+1 more reviewer

@arithx arithx arithx left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@jlebon jlebon

Labels

approved lgtm size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@cgwalters @jlebon @openshift-ci-robot @arithx @openshift-merge-robot