Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run PTF tests under different kernel versions #3535

Merged
merged 1 commit into from
Sep 30, 2022

Conversation

tatry
Copy link
Contributor

@tatry tatry commented Sep 23, 2022

This PR allows to run PTF-ebpf tests on different kernel versions using public GitHub runners.

Why this PR is needed:

  • Ensure some sort of backward compatibility for ebpf backend.
  • Test new features with older kernels (see below).

Because GitHub runners do not support KVM acceleration and tests last long (2-3h, see example workflow run in my fork), with @osinstom we made following decisions:

  • Do not run on pull requests or pushes, but in weekly manner using cron. When tests fails this would require more investigation to debug, but current work flow is not changed nor extended in time.
  • Each kernel is tested in a separate job, which should ensure that timeout 6h is not met. Testing more kernels in one may not be successful due to that timeout.
  • Cache is used to speed up jobs, but images might not always be available (i.e. approximately there is place for 6 images/kernels and is preserved for 7 days). The cache used is different from used in P4C already (or at least it looks like they are different).
  • Preserve existing PTF tests to test ebpf backend.

This PR doesn't provide script to run these tests locally, because they heavily use virtual machines. If anybody wants run test locally on a specified kernel it is recommended to just install that kernel on his/her machine and boot into it to run tests.

Example use would be during resolving issue #3453, where changes in the latest kernel cause problems for psabpf-ctl (we will rename it into nikss-ctl) with adding a new inner maps to an outer map.

This PR allows to run PTF-ebpf tests on different kernel versions using public GitHub runners.

Why this PR is needed:
- Ensure some sort of backward compatibility for ebpf backend.
- Test new features with older kernels (see below).

Because GitHub runners do not support KVM acceleration and tests last long (2-3h, see [example workflow run](https://github.com/tatry/p4c/actions/runs/3096196743/jobs/5011462027) in my fork), with @osinstom we made following decisions:
- Do not run on pull requests or pushes, but in weekly manner using `cron`. When tests fails this would require more investigation to debug, but current work flow is not changed nor extended in time.
- Each kernel is tested in a separate job, which should ensure that timeout 6h is not met. Testing more kernels in one may not be successful due to that timeout.
- Cache is used to speed up jobs, but images might not always be available (i.e. approximately there is place for 6 images/kernels and is preserved for 7 days). The cache used is different from used in P4C already (or at least it looks like they are different).
- Preserve existing PTF tests to test ebpf backend.

This PR doesn't provide script to run these tests locally, because they heavily use virtual machines. If anybody wants run test locally on a specified kernel it is recommended to just install that kernel on his/her machine and boot into it to run tests.

Example use would be during resolving issue p4lang#3453, where changes in the latest kernel cause problems for `psabpf-ctl` (we will rename it into `nikss-ctl`) with adding a new inner maps to an outer map.
@mihaibudiu
Copy link
Contributor

From my point of view this looks fine.
I don't know who pays for our CI and how much quota we have.
Running once a week sounds reasonable.
Anyone else wants to comment on this?

@fruffy
Copy link
Collaborator

fruffy commented Sep 28, 2022

I do not have a problem with this, considering it runs once a week.
To simplify CI a little I would also suggest to do the same with the ptf-ebpf action. Or, considering the repository is not part of the p4c back ends, move testing for it to a downstream repository (https://github.com/P4-Research/psabpf).

@osinstom
Copy link
Contributor

I do not have a problem with this, considering it runs once a week. To simplify CI a little I would also suggest to do the same with the ptf-ebpf action. Or, considering the repository is not part of the p4c back ends, move testing for it to a downstream repository (https://github.com/P4-Research/psabpf).

@fruffy I'm not getting your point:

  1. ptf-ebpf action's purpose is to be run as pre-merge job to validate if a new PR doesn't break PSA/eBPF backend functionality on the latest kernel, while the goal of weekly job running on different kernels is to validate backward compatibility/operability on different (older) kernels. The purpose of these 2 jobs is different.
  2. Which repository is not part of the p4c back ends? If you mean psabpf, the CI jobs are not testing psabpf - they test PSA/eBPF compiler but use psabpf to populate P4 table entries for PTF tests

@fruffy
Copy link
Collaborator

fruffy commented Sep 30, 2022

I see, I assumed psabpf to be a compiler extension but it is completely independent. I introduced changes in the past that were breaking it, but they seem related to dependencies instead. Nevermind then!

@mihaibudiu mihaibudiu merged commit e728127 into p4lang:main Sep 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants