[Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0"

[joar]

In short,

• if targetParallelism ≥ BigtableSource#getEstimatedSizeBytes; then
• desiredBundleSizeBytes is set to 0; which
• makes BigtableSource#splitKeyRangeIntoBundleSizedSubranges angry.

What happened?

Imagine a case where in:

beam/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java

Lines 215 to 217 in 282d027

	long estimatedBytes = source.getEstimatedSizeBytes(options);
	long bytesPerBundle = estimatedBytes / targetParallelism;
	List<? extends BoundedSource<T>> bundles = source.split(bytesPerBundle, options);

• targetParallelism is 32; and
• source.getEstimatedByteSize() is 10

then

• bytesPerBundle will be 0

so

beam/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java

Line 217 in 282d027

List<? extends BoundedSource<T>> bundles = source.split(bytesPerBundle, options);

will be called with the values: split.source(0L, options)

In OffsetBasedSource#split, this desired-0-sized split is handled:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/io/OffsetBasedSource.java

Lines 115 to 116 in 282d027

	long desiredBundleSizeOffsetUnits =
	Math.max(Math.max(1, desiredBundleSizeBytes / getBytesPerOffset()), minBundleSize);

But BigtableSource#split does not seem to handle the desired-0-sized split:

beam/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java

Lines 1328 to 1333 in 282d027

	desiredBundleSizeBytes =
	Math.max(sizeEstimate / maximumNumberOfSplits, desiredBundleSizeBytes);
	// Delegate to testable helper.
	List<BigtableSource> splits =
	splitBasedOnSamples(desiredBundleSizeBytes, getSampleRowKeys(options));

so a few frames down the road from BigtableSource#split you'll end up violating this checkArgument in BigtableSource#splitKeyRangeIntoBundleSizedSubranges:

beam/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java

Lines 1623 to 1626 in 71c8459

	checkArgument(
	desiredBundleSizeBytes > 0,
	"Desired bundle size %s bytes must be greater than 0.",
	desiredBundleSizeBytes);

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Thanks for reporting the issue. Interesting edge case, since it has been root caused, would you mind entering a PR for fix, thanks for the contribution!

[joar]

The scenario that makes this edge case pop up for us is running pipelines in CI against the Bigtable emulator. The key of course being tiny tables.

I'll enter a PR!

[joar]

The question is, should it be fixed in BigtableSource, BoundedReadEvaluatorFactory or both?

[Abacn]

Fix BoundedReadEvaluatorFactory

• Pro: prevent errorprone behavior on all implementation like this
• Con: changed the behavior of java core, the scope of impact is unknown

Fix BigtableSource

• Pro: safe
• Con: just fix one use case

Based on this understanding I would vote for Fix BigtableSource for now.