feat: run most frequent flavours in the PR stage by mdelapenya · Pull Request #873 · elastic/e2e-testing

mdelapenya · 2021-03-10T12:34:55Z

What does this PR do?

This PR refactors how the BDD feature files are structured, separating the existing examples for a scenario into multiple Examples block, annotated with the example type.

I.e., instead of:

Examples:
| image   |
| default |
| ubi8    |

we will have:

@default
Examples: default
| image   |
| default |

@ubi8
Examples: Ubi8
| image   |
| ubi8    |

To instruct the CI we are adding a new pullRequestFilter key in the E2E YAML descriptor for CI, which is read by the pipeline to configure the build, including the Gherkin tags to be appended for a PR. Therefore, the CI will append the pullRequestFilter tags to the tags to be added to the current execution, only for PRs.

For Fleet's endpoint integration, the PR filter will add ~debian, indicating that debian scenarios will be skipped.

we added both OSs (centos and debian) to the execution, as it only used centos for the tests. WIth this strategy, now it's possible to run both in the nightly builds, but centos in the PR stage.

another improvement is that we moved/copied the scenarios affecting processes for endpoint, to the feature file for processes.

For Fleet's fleet mode, the PR filter will add ~debian, indicating that debian scenarios will be skipped.
For Fleet's Standalone mode, the PR filter will add ~ubi8, indicating that UBI8 scenarios will be skipped.

If the key is empty, the pipeline will add an empty string, which means keeping old behaviour.

We are using same approach for metricbeat's MySQL versions: moving most used versions to the PR phase (latest MariaDB, and MySQL 8 and 5.7) and the rest (old versions and Percona) to the nightly.

Why is it important?

We will be able to instruct the CI to add the selected tag/annotation to be run for PRs, speeding up the PR build (actually halfing it). On the other hand, nightly builds should include those skipped scenarios.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's checklist

Select Centos or Debian as the OS to be skipped in the PR phase for the Fleet mode tests
Select UBI8 or the default one as the flavour to be skipped in the PR phase for the Standalone tests

@michalpristas @EricDavisX could you please help me out to choose the platform?

Related issues

Closes Separate centos, debian and tar executions using BDD tags #869

Use cases

For fleet mode:

Given the Fleet-mode features are executed
When a PR is executed
Then debian scenarios are skipped

Given the Fleet-mode features are executed
When a branch is executed
Then debian scenarios are included

Given the Fleet-mode features are executed
When a nightly build is executed
Then debian scenarios are included

For stand-alone mode:

Given the Standalone-mode features are executed
When a PR is executed
Then ubi8 scenarios are skipped

Given the Standalone-mode features are executed
When a branch is executed
Then ubi8 scenarios are included

Given the Standalone-mode features are executed
When a nightly build is executed
Then ubi8 scenarios are included

Follow-up

Document the connection between Gherkin tags and the I descriptor

This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build

mdelapenya · 2021-03-10T12:38:20Z

      - "ubuntu-18.04"
    scenarios:
      - name: "Fleet"
+        pullRequestFilter: "&& ~debian"


Adding the BDD filters here will simplify the logic in the pipeline

mdelapenya · 2021-03-10T12:39:08Z

  def platform = args.get('platform')
  def suite = args.get('suite')
  def tags = args.get('tags')
+  def pullRequestFilter = args.get('pullRequestFilter')?.trim() ? args.get('pullRequestFilter') : ''


Do not add a filter if the key is not present or empty

mdelapenya · 2021-03-10T12:39:39Z

  def suite = item.suite
  def platforms = item.platforms
  item.scenarios.each { scenario ->
+    def pullRequestFilter = scenario.pullRequestFilter


If the key is not present, what value will be passed here? Should we add defensive code here too?

elasticmachine · 2021-03-10T12:43:31Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Started by user Manuel de la Peña
Start Time: 2021-03-11T17:37:19.042+0000
Duration: 20 min 35 sec
Commit: 546d77f

Test stats 🧪

Test	Results
Failed	0
Passed	109
Skipped	0
Total	109

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	109
Skipped	0
Total	109

mdelapenya · 2021-03-10T13:24:23Z

+        pullRequestFilter: "&& ~ubi8"
        tags: "stand_alone_agent"
      - name: "Backend Processes"
+        pullRequestFilter: "&& ~debian"


Suggested change

pullRequestFilter: "&& ~debian"

pullRequestFilter: " && ~debian"

@variant

Because there is only one variant, we are removing the @variant tag

EricDavisX

yeah, i like this - thanks Manu.

mdelapenya · 2021-03-10T21:28:48Z

Mmm... I'm thinking we need to cover the use case where a PR on beats triggers the master branch here. We could pass a special flag from the PR so that we could identify the triggered build.

@kuisathaverat @v1v do you know if it's possible to identify that the upstream job is a PR with the build causes?

mdelapenya · 2021-03-11T18:03:33Z

Now that we support detecting the upstream cause for PRs, I'm going to merge this one. Thanks!

@variant

* chore: decouple centos from debian in BDD examples This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build * feat: run centos in the PR phase * feat: do not run ubi8 in the PR phase * fix: negate tags to skip only those ones * fix: enclose variable in double quotes * fix: add whitespaces * fix: defensive programming for the new key * chore: move old MySQL versions to the nightly build Because there is only one variant, we are removing the @variant tag * chore: use PR filter for MySQL YAML descritor * chore: move MySQL 5.7 to the PR phase * chore: move Percona out to the PR phase * chore: append PR filter for upstream PRs * chore: support running endpoint tests for debian * chore: move endpoint processes to the process feature file * chore: do not add Debian tests for PRs in endpoint test suite

@variant

* chore: decouple centos from debian in BDD examples This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build * feat: run centos in the PR phase * feat: do not run ubi8 in the PR phase * fix: negate tags to skip only those ones * fix: enclose variable in double quotes * fix: add whitespaces * fix: defensive programming for the new key * chore: move old MySQL versions to the nightly build Because there is only one variant, we are removing the @variant tag * chore: use PR filter for MySQL YAML descritor * chore: move MySQL 5.7 to the PR phase * chore: move Percona out to the PR phase * chore: append PR filter for upstream PRs * chore: support running endpoint tests for debian * chore: move endpoint processes to the process feature file * chore: do not add Debian tests for PRs in endpoint test suite

@variant

* chore: decouple centos from debian in BDD examples This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build * feat: run centos in the PR phase * feat: do not run ubi8 in the PR phase * fix: negate tags to skip only those ones * fix: enclose variable in double quotes * fix: add whitespaces * fix: defensive programming for the new key * chore: move old MySQL versions to the nightly build Because there is only one variant, we are removing the @variant tag * chore: use PR filter for MySQL YAML descritor * chore: move MySQL 5.7 to the PR phase * chore: move Percona out to the PR phase * chore: append PR filter for upstream PRs * chore: support running endpoint tests for debian * chore: move endpoint processes to the process feature file * chore: do not add Debian tests for PRs in endpoint test suite

@variant

…7.11.x (#882) * feat: run most frequent flavours in the PR stage (#873) * chore: decouple centos from debian in BDD examples This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build * feat: run centos in the PR phase * feat: do not run ubi8 in the PR phase * fix: negate tags to skip only those ones * fix: enclose variable in double quotes * fix: add whitespaces * fix: defensive programming for the new key * chore: move old MySQL versions to the nightly build Because there is only one variant, we are removing the @variant tag * chore: use PR filter for MySQL YAML descritor * chore: move MySQL 5.7 to the PR phase * chore: move Percona out to the PR phase * chore: append PR filter for upstream PRs * chore: support running endpoint tests for debian * chore: move endpoint processes to the process feature file * chore: do not add Debian tests for PRs in endpoint test suite * fix: proper usage of step (#883)

@variant

…7.12.x (#881) * feat: run most frequent flavours in the PR stage (#873) * chore: decouple centos from debian in BDD examples This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build * feat: run centos in the PR phase * feat: do not run ubi8 in the PR phase * fix: negate tags to skip only those ones * fix: enclose variable in double quotes * fix: add whitespaces * fix: defensive programming for the new key * chore: move old MySQL versions to the nightly build Because there is only one variant, we are removing the @variant tag * chore: use PR filter for MySQL YAML descritor * chore: move MySQL 5.7 to the PR phase * chore: move Percona out to the PR phase * chore: append PR filter for upstream PRs * chore: support running endpoint tests for debian * chore: move endpoint processes to the process feature file * chore: do not add Debian tests for PRs in endpoint test suite * fix: proper usage of step (#883)

@variant

…7.x (#880) * feat: run most frequent flavours in the PR stage (#873) * chore: decouple centos from debian in BDD examples This change will support the selection of centos, debian or both when running the tests. Therefore the CI will be able to run only tests for centos in the PR phase, speeding up the PR build: The other OSs could be run in the nightly build * feat: run centos in the PR phase * feat: do not run ubi8 in the PR phase * fix: negate tags to skip only those ones * fix: enclose variable in double quotes * fix: add whitespaces * fix: defensive programming for the new key * chore: move old MySQL versions to the nightly build Because there is only one variant, we are removing the @variant tag * chore: use PR filter for MySQL YAML descritor * chore: move MySQL 5.7 to the PR phase * chore: move Percona out to the PR phase * chore: append PR filter for upstream PRs * chore: support running endpoint tests for debian * chore: move endpoint processes to the process feature file * chore: do not add Debian tests for PRs in endpoint test suite * fix: proper usage of step (#883)

…hings * upstream/master: [CI] tear down the workspace (elastic#885) docs: add Make as build system (elastic#886) fix: proper usage of step (elastic#883) feat: run most frequent flavours in the PR stage (elastic#873) break: move from "pull-requests" to "commits" GCP bucket (elastic#866) fix: use proper flags (elastic#868) feat: instrument Helm charts test suite (elastic#858)

mdelapenya added 4 commits March 10, 2021 12:59

feat: run centos in the PR phase

b3b895d

feat: do not run ubi8 in the PR phase

340f987

fix: negate tags to skip only those ones

5d33580

mdelapenya self-assigned this Mar 10, 2021

mdelapenya requested review from a team, EricDavisX and michalpristas March 10, 2021 12:34

mdelapenya commented Mar 10, 2021

View reviewed changes

fix: enclose variable in double quotes

0b8a140

kuisathaverat approved these changes Mar 10, 2021

View reviewed changes

mdelapenya commented Mar 10, 2021

View reviewed changes

Comment thread .ci/.e2e-tests.yaml Outdated

Comment thread .ci/.e2e-tests.yaml Outdated

Comment thread .ci/.e2e-tests.yaml Outdated

mdelapenya commented Mar 10, 2021

View reviewed changes

mdelapenya added 6 commits March 10, 2021 14:26

fix: add whitespaces

97db516

fix: defensive programming for the new key

a7c6544

chore: move old MySQL versions to the nightly build

d8958e3

Because there is only one variant, we are removing the @variant tag

chore: use PR filter for MySQL YAML descritor

b3e7134

chore: move MySQL 5.7 to the PR phase

3a759dc

chore: move Percona out to the PR phase

93d11e4

mdelapenya changed the title ~~feat: run most frequent OSs in the PR stage~~ feat: run most frequent flavours in the PR stage Mar 10, 2021

EricDavisX approved these changes Mar 10, 2021

View reviewed changes

mdelapenya marked this pull request as ready for review March 10, 2021 21:24

mdelapenya mentioned this pull request Mar 11, 2021

chore: support filtering the upstream cause in the isUpstreamTrigger step elastic/apm-pipeline-library#1031

Merged

cachedout approved these changes Mar 11, 2021

View reviewed changes

mdelapenya added 2 commits March 11, 2021 13:41

chore: append PR filter for upstream PRs

c8ab765

chore: support running endpoint tests for debian

cf3d043

mdelapenya added 2 commits March 11, 2021 16:32

chore: move endpoint processes to the process feature file

ab3e85b

chore: do not add Debian tests for PRs in endpoint test suite

546d77f

mdelapenya merged commit a01d0a6 into elastic:master Mar 11, 2021

mdelapenya mentioned this pull request Mar 11, 2021

feat: run most frequent flavours in the PR stage (#873) backport for 7.x #880

Merged

mdelapenya mentioned this pull request Mar 11, 2021

feat: run most frequent flavours in the PR stage (#873) backport for 7.12.x #881

Merged

mdelapenya mentioned this pull request Mar 11, 2021

feat: run most frequent flavours in the PR stage (#873) backport for 7.11.x #882

Merged

mdelapenya deleted the 869-installer-annotations branch March 11, 2021 18:05

mdelapenya mentioned this pull request Mar 12, 2021

fix: proper usage of step #883

Merged

8 tasks

mdelapenya mentioned this pull request Oct 26, 2021

chore: bump elastic-package to v0.25.0 #1721

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: run most frequent flavours in the PR stage#873

feat: run most frequent flavours in the PR stage#873
mdelapenya merged 15 commits intoelastic:masterfrom
mdelapenya:869-installer-annotations

mdelapenya commented Mar 10, 2021 •

edited

Loading

Uh oh!

mdelapenya Mar 10, 2021 •

edited

Loading

Uh oh!

mdelapenya Mar 10, 2021

Uh oh!

mdelapenya Mar 10, 2021

Uh oh!

elasticmachine commented Mar 10, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdelapenya Mar 10, 2021

Uh oh!

EricDavisX left a comment

Uh oh!

mdelapenya commented Mar 10, 2021

Uh oh!

mdelapenya commented Mar 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	pullRequestFilter: "&& ~debian"
	pullRequestFilter: " && ~debian"

Conversation

mdelapenya commented Mar 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's checklist

Related issues

Use cases

Follow-up

Uh oh!

mdelapenya Mar 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdelapenya Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

mdelapenya Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdelapenya Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

EricDavisX left a comment

Choose a reason for hiding this comment

Uh oh!

mdelapenya commented Mar 10, 2021

Uh oh!

mdelapenya commented Mar 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mdelapenya commented Mar 10, 2021 •

edited

Loading

mdelapenya Mar 10, 2021 •

edited

Loading

elasticmachine commented Mar 10, 2021 •

edited

Loading