[FEATURE] Metrics to track the integration test failures at the tests level

[prudhvigodithi]

Is your feature request related to a problem?

Coming from Central Release Dashboard, track the integration test failures at the tests level (displaying the exact failed tests at component level) and give a report of the failed/flaky tests.

Today we use the libraries publishDistributionBuildResults and publishIntegTestResults to index the build and integration test failures for both OpenSearch and OpenSearch Dashboards. Extend this to ensure the failed integration tests has the information displaying the exact failed tests. Once the failed tests are indexed create visualizations to surface the failures at RC level and find the flaky test trends. Having the failed tests would help the release manager and plugin teams to quickly identify the exact failures of the integration test.

What solution would you like?

For dashboards the cypress reports are stored in cypress/results/ and for OpenSearch plugins in build/reports/tests/integTest/.

• Create a library to parse these reports and index to the cluster periodically for every component during the integration test execution.
  (or)
• Use the test report workflow to add the failing tests and classes information to test manifest and update the library publishIntegTestResults to parse the yaml file and index the results.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[peterzhuamazon]

Hi,

I do prefer to keep adding the information to test-report.yml as it could be the single source of truth, instead of directly reading from the cypress/results dir. In the end if we upgrade cypress version (which is likely based on how old cypress9 is), or change to a different test framework later on, we do not need to change the metrics code that much.

Thanks.

[prudhvigodithi]

[Triage]
Thanks @peterzhuamazon, yes today we upload all the the gradle and cypress test reports to the build s3 bucket, example

integ-test/2.16.0/10154/linux/x64/tar/test-results/8503/integ-test/k-NN/with-security/opensearch-integ-test/

integ-test-opensearch-dashboards/2.16.0/7856/linux/x64/tar/test-results/6186/integ-test/OpenSearch-Dashboards-ci-group-9/with-security/cypress-report/

Extend the report workflow to collect the above information and update the test-report.yml with all the details.

./report.sh manifests/1.3.19/opensearch-1.3.19-test.yml --artifact-paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.19/10212/linux/arm64/rpm --test-run-id 8574 --test-type integ-test --base-path https://ci.opensearch.org/ci/dbc/integ-test/1.3.19/10212/linux/arm64/rpm --release-candidate 3 https://ci.opensearch.org/ci/dbc/integ-test/1.3.19/10212/linux/arm64/rpm/test-results/8574/integ-test/k-NN/without-security/k-NN.yml

[peterzhuamazon]

Just have a offline conversation with @prudhvigodithi , and we both agrees that processing the opensearch-integ-test and cypress results dir for testclass and testname is beneficial to catching the failure, similar to what we did with gradle check failure index.

I will add this as part of the test report manifest soon.

Thanks.

[peterzhuamazon]

OpenSearch:

• Add OpenSearch failed Test Class and Name fields to test-report.yml opensearch-build#5101
• Integtest failed test case check will skip infobox class opensearch-build#5108
• Record OpenSearch-Dashboards failed test cases in test-report.yml opensearch-build#5111
• Use basename with extension as OSD test case class name opensearch-build#5116

[prudhvigodithi]

Closing this issue as today in OpenSearch Distribution Build and Integration Test Analytics dashboard we have all the details about the failing tests and its related classes. Created some data table visualizations for specific platform, architecture and distribution, these visualizations can be filtered by release version, RC, component etc. We can add more trend graph visualizations and pie charts as required. Thanks to @peterzhuamazon for adding this to test manifest.
Thank you
@getsaurabh02