Add streaming test for Write API sink

[AlexZMLyu]

Add streaming test for BigQuery Write API

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[AlexZMLyu]

@chamikaramj Hi Cham, please kindly review the patch.

[yirutang]

R: @chamikaramj

[AlexZMLyu]

R: @pabloem

[pabloem]

Run Java PostCommit

[pabloem]

@AlexZMLyu please run ./gradlew :sdks:java:io:google-cloud-platform:spotlessApply to format the code and fix spotless issue

[pabloem]

Running postcommit that exercises these tests: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV1/1765/

[AlexZMLyu]

Talked to @pabloem offline. The TestStream class that I used here as the streaming source is not compatible with Dataflow runner. I will edit the code using the GenerateSequence and overriding the expand method to serve as the streaming source instead.

[robertwb]

TestStream should work fine on Runner v2, and is preferable.

[AlexZMLyu]

Robert, thanks for letting me know.
Do I need to change my code to utilize the Runner V2?

[chamikaramj]

This test is disabled for Runner v2 currently:

beam/runners/google-cloud-dataflow-java/build.gradle

Line 642 in 67e6726

exclude '**/BigQueryIOStorageWriteIT.class'

Trying to re-enable here: #21814

(you can try it on this PR similarly)

[AlexZMLyu]

Run PostCommit_Java_DataflowV2

[AlexZMLyu]

There was no failed test in https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2_PR/112/testReport/
Why the Dataflow V2 Java Post Commit Tests failed?

[AlexZMLyu]

Do I need to wait for approval of PR #21814 to run Runner v2 here?

[AlexZMLyu]

Run PostCommit_Java_DataflowV2

[chamikaramj]

Seems like Storage Write API tests are still failing for Runner v2: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2_PR/111/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOStorageWriteIT/testBigQueryStorageWrite30MProto/

So we can just get the Runner v1 version in (this PR).

[chamikaramj]

Retest this please

[chamikaramj]

Run PostCommit_Java_DataflowV1

[chamikaramj]

Run PostCommit_Java_Dataflow

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Seems like the new tests failed the post-commit test suite: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV1_PR/124/

[AlexZMLyu]

Then I still need to change the streaming source from TestStream to something else, right?

[chamikaramj]

Yeah, I think it's probably ok to use GenerateSequence (a bounded source) for a simple integration test that use the streaming Runner v1. Separately we should check why this still failed for Runner v2.

[AlexZMLyu]

Run PostCommit_Java_DataflowV1

[AlexZMLyu]

Run PostCommit_Java_DataflowV1

[chamikaramj]

Run Spotless PreCommit

[AlexZMLyu]

Spotless test passed.

[chamikaramj]

Run Java_GCP_IO_Direct PreCommit

[chamikaramj]

Seems like there's still one failing test suite. Probably unrelated. Retrying.

[AlexZMLyu]

Looks like all test passed

[chamikaramj]

Thanks. Merging.