Assert pipeline results in performance tests

[Abacn]

Fixes #23026

• Fix possible false possitive test status

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @johnjcasey for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[Abacn]

Run Java KafkaIO Performance Test

[Abacn]

Run Java JdbcIO Performance Test

[Abacn]

Run SQLBigQueryIO Batch Performance Test Java

[Abacn]

Run BigQueryIO Batch Performance Test Java Avro

[Abacn]

Run BigQueryIO Batch Performance Test Java Json

[Abacn]

Run Java CdapIO Performance Test

[Abacn]

Run Java AvroIO Performance Test

[Abacn]

Run Java ParquetIO Performance Test

[Abacn]

Run Java TextIO Performance Test

[Abacn]

Run Java TFRecordIO Performance Test

[Abacn]

Run Java XmlIO Performance Test

[Abacn]

Run Java HadoopFormatIO Performance Test

[Abacn]

Run Java MongoDBIO Performance Test

[Abacn]

Kafka IO performance test has "failed correctly", previous success prior to https://ci-beam.apache.org/view/PerformanceTests/job/beam_PerformanceTests_Kafka_IO/3005 was false positive [runs between run 3006 to 3026 there was another cause of failure (#23020)].
That is, the pipeline always fails with the following log message but previously not detected as fail test:

SEVERE: 2022-08-29T13:06:00.196Z: Workflow failed. Causes: S01:Generate records+Measure write time+Write to Kafka/Kafka ProducerRecord/Map+Write to Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter) failed., The job failed because a work item has failed 4 times.

[bvolpato]

Just curious, can the job end with another terminal state other than DONE or FAILED? e.g., if executed in a runner that supports cancellation.
If so, we could expect for DONE explicitly, instead of checking for not FAILED.

[Abacn]

Just curious, can the job end with another terminal state other than DONE or FAILED? e.g., if executed in a runner that supports cancellation.

If so, we could expect for DONE explicitly, instead of checking for not FAILED.

Yes you are right, cancel is missed in this assertion. I thought about if assert DONE or assert not FAILED. There exists some streaming tests that set to cancel the pipeline after certain time (because streaming pipelines generally does not DONE by themself); For simplicity I add same assertions for all performance tests.

Yes if we investigate each tests, for some tests including cancel as failure sounds more accurate; even more accurate the test should be marked as Aborted because a cancellation is initiated outside of the pipeline execution and done by runner (e.g. someone cancelled the pipeline at Dataflow). I just did not go this far. Assert not FAIL should suffice for the testing of Beam SDK.

[johnjcasey]

LGTM

[Abacn]

The two failed tests: SQL BigQueryIO is already perma-red; KafkaIO is failed "correctly" and would be fixed seperately

[Abacn]

Kafka Performance test is running two exact same pipelines (after a pipeline option removed in #14168). Removed one of them.

[johnjcasey]

Gotcha. LGTM

[Abacn]

Run Java_Examples_Dataflow_Java17 PreCommit