[Bug]: Wierd Limit error on TextToPubsub batching

[hoangn-athena]

Related Template(s)

TextToPubsub

What happened?

It only happen on SDK version 2.48.0

Beam Version

Newer than 2.46.0

Relevant log output

Error message from worker: javax.naming.SizeLimitExceededException: Pubsub message of length 264 exceeds maximum of 100 bytes, when considering the payload and attributes. See https://cloud.google.com/pubsub/quotas#resource_limits org.apache.beam.sdk.io.gcp.pubsub.PreparePubsubWriteDoFn.validatePubsubMessageSize(PreparePubsubWriteDoFn.java:101) org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Write$PubsubBoundedWriter.processElement(PubsubIO.java:1413)

[mohamed-ali24]

@hoangn-athena , which version of this template are you using?

[hoangn-athena]

@hoangn-athena , which version of this template are you using?
@mohamed-ali24, I am using the latest version which is located at
gs://dataflow-templates-libraries/2023-06-13-00_RC00

[mohamed-ali24]

Can you try the prior versiongs://dataflow-templates-libraries/2023-06-06-00_rc00?

[mohamed-ali24]

There is a message available at:
https://cloud.google.com/dataflow/docs/support/sdk-version-support-status
Warning: In version 2.48.0, when you use Pub/Sub sinks with batch jobs, your pipeline might incorrectly receive the error SizeLimitExceededException for any messages above 100 bytes. For more information, see the Apache Beam GitHub issue #27000.
apache/beam#27000

[hoangn-athena]

@mohamed-ali24, Thank you for the information.

[bvolpato]

Yes, sorry, there was a bug released on Beam 2.48.0 and templates updated to that version recently -- we do have tests for this template but this happened in an edge case that wasn't covered.

We are improving our test coverage too: #810

Thanks @mohamed-ali24 and @hoangn-athena!

[bvolpato]

Added a note in the release too.