Read API Source v2 #25392

vachan-shetty · 2023-02-08T23:02:24Z

Adding the new Read API Source which supports bundling of Streams within Read Session. Customers can enable/disable this feature using the setEnableBundling BigQuery pipeline option (the option flag currently defaults to false).

The design doc for the changes can be found here. This PR addresses #24260.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…o explain how OffsetBasedSource + RangeTracker is used.

vachan-shetty · 2023-02-08T23:53:16Z

R: @johnjcasey, @lukecwik, @kmjung

github-actions · 2023-02-09T01:23:02Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

...google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java

...orm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamBundleSource.java

…it test coverage.

…PI sources.

vachan-shetty · 2023-02-13T18:59:54Z

R: @ahmedabu98

vachan-shetty · 2023-02-15T00:54:53Z

Retest this please.

vachan-shetty · 2023-02-15T22:51:13Z

Retest this please.

ahmedabu98

Left a few comments. You can ignore my logging suggestions if you like another alternative, but I think we should improve some of those.
Still haven't looked at test class, but submitting this review now to give you some time before next release cut.

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java

...orm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamBundleSource.java

ahmedabu98 · 2023-02-16T23:26:53Z

...orm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamBundleSource.java

+    // of BigQueryStorageSourceBase, all splits here will be handled by `splitAtFraction()`. As a
+    // result, this is a no-op.


FYI because it was mentioned previously as TODO: Implement dynamic work rebalancing, splitAtFraction() is dynamic work rebalancing

ahmedabu98

LGTM! Hope these changes unblock users with large read operations

ahmedabu98 · 2023-02-16T23:40:34Z

Failing test org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testTriggeredFileLoadsWithTempTablesToExistingNullSchemaTable[1] in Java_GCP_IO_Direct is a known issue #25207. Can be ignored

johnjcasey · 2023-02-22T14:57:33Z

retest this please

…o explain how OffsetBasedSource + RangeTracker is used.

…it test coverage.

…PI sources.

…eadapi-v2 Merging latest changes from `upstream/master` to ensure Test runs.

…to readapi-v2 Merging latest changes from upstream.

damccorm · 2023-02-22T18:59:08Z

Test suites aren't statusing, but they've all completed successfully except for https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/1778/ (Java_GCP_IO_Direct) which is still running and seems stuck at the start

damccorm · 2023-02-22T18:59:16Z

Run Java_GCP_IO_Direct PreCommit

damccorm · 2023-02-22T19:48:38Z

That succeeded - https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/1778/

damccorm

LGTM

* ReadAPI Source v2 * Renamed the Source V2. Also added tests for the same. * v2 using OffsetBasedSource and OffsetBasedReader * Updating tests to have more sensible mock values. * Updated BqOption flag. * Simplifying `fractionConsumed` calculation. * Better variable names. * Minor refactoring. * Added a synchronized block in readNextRecord(). Also added comments to explain how OffsetBasedSource + RangeTracker is used. * Removed unnecessary synchronized block, added Javadoc and improved unit test coverage. * Consolidated code paths in `BigQueryIO` for Bundled and regular ReadAPI sources. * Lint fixes. * Minor Javadoc fix. * Fix StreamBundle creation logic and some minor code comment updates. * Updated logging. * Lint fixes. * ReadAPI Source v2 * Renamed the Source V2. Also added tests for the same. * v2 using OffsetBasedSource and OffsetBasedReader * Updating tests to have more sensible mock values. * Updated BqOption flag. * Simplifying `fractionConsumed` calculation. * Better variable names. * Minor refactoring. * Added a synchronized block in readNextRecord(). Also added comments to explain how OffsetBasedSource + RangeTracker is used. * Removed unnecessary synchronized block, added Javadoc and improved unit test coverage. * Consolidated code paths in `BigQueryIO` for Bundled and regular ReadAPI sources. * Lint fixes. * Minor Javadoc fix. * Fix StreamBundle creation logic and some minor code comment updates. * Updated logging. * Lint fixes.

This reverts commit 4ce8eed.

* Revert "Read API Source v2 (#25392)" This reverts commit 4ce8eed. * Preserve the ablility to use read API backend feature * Add log about num of stream * Add to CHANGES.md

vachan-shetty added 9 commits December 5, 2022 18:20

ReadAPI Source v2

3ec1d08

Renamed the Source V2. Also added tests for the same.

0a2fc4f

v2 using OffsetBasedSource and OffsetBasedReader

fc55a4f

Updating tests to have more sensible mock values.

0b09d51

Updated BqOption flag.

409afac

Simplifying fractionConsumed calculation.

ca02997

Better variable names.

a6e2d73

Minor refactoring.

f4677fa

Added a synchronized block in readNextRecord(). Also added comments t…

139e317

…o explain how OffsetBasedSource + RangeTracker is used.

github-actions bot added gcp io java labels Feb 9, 2023

kmjung reviewed Feb 9, 2023

View reviewed changes

vachan-shetty added 2 commits February 10, 2023 01:27

Removed unnecessary synchronized block, added Javadoc and improved un…

7e1f974

…it test coverage.

Consolidated code paths in BigQueryIO for Bundled and regular ReadA…

cf2cc08

…PI sources.

vachan-shetty added 2 commits February 13, 2023 21:49

Lint fixes.

54e80ae

Minor Javadoc fix.

dac8cfe

ahmedabu98 requested changes Feb 16, 2023

View reviewed changes

vachan-shetty added 2 commits February 16, 2023 22:54

Fix StreamBundle creation logic and some minor code comment updates.

d902326

Updated logging.

b45bdd7

ahmedabu98 reviewed Feb 16, 2023

View reviewed changes

ahmedabu98 approved these changes Feb 16, 2023

View reviewed changes

Lint fixes.

323a080

vachan-shetty added 15 commits February 22, 2023 17:24

Updating tests to have more sensible mock values.

d71c304

Updated BqOption flag.

a87b56b

Simplifying fractionConsumed calculation.

572467c

Better variable names.

9778b35

Minor refactoring.

e780266

Added a synchronized block in readNextRecord(). Also added comments t…

03ca8d6

…o explain how OffsetBasedSource + RangeTracker is used.

Removed unnecessary synchronized block, added Javadoc and improved un…

b337c57

…it test coverage.

Consolidated code paths in BigQueryIO for Bundled and regular ReadA…

e02665f

…PI sources.

Lint fixes.

d248d66

Minor Javadoc fix.

1432cbc

Fix StreamBundle creation logic and some minor code comment updates.

3adc815

Updated logging.

5eb1aa4

Lint fixes.

d1a6d43

Merge branch 'master' of https://github.com/vachan-shetty/beam into r…

0dd2aa6

…eadapi-v2 Merging latest changes from `upstream/master` to ensure Test runs.

Merge branch 'readapi-v2' of https://github.com/vachan-shetty/beam in…

d536f42

…to readapi-v2 Merging latest changes from upstream.

johnjcasey merged commit 4ce8eed into apache:master Feb 22, 2023

damccorm approved these changes Feb 22, 2023

View reviewed changes

Abacn mentioned this pull request Apr 19, 2023

[Bug]: BigQueryIO direct read not reading all rows when set --setEnableBundling=true #26354

Closed

15 tasks

Abacn added a commit to Abacn/beam that referenced this pull request Feb 26, 2024

Revert "Read API Source v2 (apache#25392)"

973be6c

This reverts commit 4ce8eed.

Abacn mentioned this pull request Feb 27, 2024

Disable BigQueryStorageStreamSource.splitAtFraction when read api v2 used #30443

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read API Source v2 #25392

Read API Source v2 #25392

vachan-shetty commented Feb 8, 2023

vachan-shetty commented Feb 8, 2023

github-actions bot commented Feb 9, 2023

vachan-shetty commented Feb 13, 2023

vachan-shetty commented Feb 15, 2023

vachan-shetty commented Feb 15, 2023

ahmedabu98 left a comment

ahmedabu98 Feb 16, 2023

ahmedabu98 left a comment

ahmedabu98 commented Feb 16, 2023

johnjcasey commented Feb 22, 2023

damccorm commented Feb 22, 2023

damccorm commented Feb 22, 2023

damccorm commented Feb 22, 2023

damccorm left a comment

		// of BigQueryStorageSourceBase, all splits here will be handled by `splitAtFraction()`. As a
		// result, this is a no-op.

Read API Source v2 #25392

Read API Source v2 #25392

Conversation

vachan-shetty commented Feb 8, 2023

GitHub Actions Tests Status (on master branch)

vachan-shetty commented Feb 8, 2023

github-actions bot commented Feb 9, 2023

vachan-shetty commented Feb 13, 2023

vachan-shetty commented Feb 15, 2023

vachan-shetty commented Feb 15, 2023

ahmedabu98 left a comment

Choose a reason for hiding this comment

ahmedabu98 Feb 16, 2023

Choose a reason for hiding this comment

ahmedabu98 left a comment

Choose a reason for hiding this comment

ahmedabu98 commented Feb 16, 2023

johnjcasey commented Feb 22, 2023

damccorm commented Feb 22, 2023

damccorm commented Feb 22, 2023

damccorm commented Feb 22, 2023

damccorm left a comment

Choose a reason for hiding this comment