Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate SnsIO.write to async implementation #21366

Open
damccorm opened this issue Jun 4, 2022 · 1 comment
Open

Migrate SnsIO.write to async implementation #21366

damccorm opened this issue Jun 4, 2022 · 1 comment

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

In  BEAM-13203 SnsIO.writeAsync was deprecated to address the risk of data loss in that transform. Unfortunately, it has considerable issues in design and implementation with respect to the beam programming model as well as general async programming:

  • Output for a bundle may be emitted even after the bundle is finished (lack of join / waiting for pending futures), additionally output is emitted from multiple threads risking concurrent modification issues on underlying data structures.
  • Exceptions disappear into the void (respective tests are implemented incorrectly as they don't throw asynchronously and could make one think this works).
  • Parallelism is unlimited causing the writer to instantly accumulate everything into memory.

With BEAM-13563 it will be possible to change SnsIO.write into a more performant async implementation without any breaking changes to the public user facing API making SnsIO.writeAsync obsolete.

Imported from Jira BEAM-13824. Original Jira may contain additional context.
Reported by: mosche.

@mosche
Copy link
Member

mosche commented May 5, 2023

A new async implementation should also use the batch API, see related
#21429

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants