fix(destination-bigquery): retry typing+deduping queries on concurrent-update aborts - #76440
Draft
devin-ai-integration[bot] wants to merge 2 commits into
Draft
fix(destination-bigquery): retry typing+deduping queries on concurrent-update aborts#76440devin-ai-integration[bot] wants to merge 2 commits into
devin-ai-integration[bot] wants to merge 2 commits into
Conversation
…t-update aborts BigQuery uses optimistic concurrency control for DML. When two transactions modify the same table in overlapping windows, BigQuery aborts one of them with "Transaction is aborted due to concurrent update". The typing+deduping SQL executed by the destination (MERGE, CREATE OR REPLACE TABLE) is idempotent, so the canonical mitigation is to retry. Retry up to 5 times with exponential backoff and jitter when we detect the concurrent-update error text. Other errors continue to surface immediately. Resolves airbytehq/oncall#11993. Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
Contributor
|
Contributor
|
↪️ Triggering Reason: Draft PR with CI green, no prior |
Contributor
|
Contributor
Author
Fix Validation Evidence — In ProgressStarting Status: Investigating Current phase: Context gathering and pre-flight checks I will update this same comment as I progress through:
|
3 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resolves https://github.com/airbytehq/oncall/issues/11993:
BigQuery uses optimistic concurrency control for DML. When two transactions modify the same table in overlapping windows, BigQuery aborts one of them with:
The destination-bigquery connector's typing+deduping path wraps its
MERGE/CREATE OR REPLACE TABLEstatements inBEGIN TRANSACTION ... COMMIT TRANSACTION, which makes them eligible for this abort whenever another writer (a second sync, a dbt job, a scheduled query, an analyst query, etc.) touches the same final table. Because Sentry sees a unique transaction ID + table name on every occurrence, each hit groups as a new issue and generates oncall noise even though the canonical mitigation, per BigQuery docs on concurrent DML statements, is simply to retry the aborted transaction.The typing+deduping operations we run here are idempotent, so retrying is safe.
How
In
BigQueryDatabaseHandler.execute(sql):runQueryWithConcurrentUpdateRetries.job.status.error, if the error (or any of itsexecutionErrors) contains"Transaction is aborted due to concurrent update", retry with exponential backoff + jitter. Up to 5 total attempts, starting at 1s delay, capped at 60s.ConfigErrorExceptionwrapping for billing errors).This keeps the fix scoped to the destination connector; no CDK changes are needed. Callers from the CDK's
TypingDedupingFinalTableOperations(createFinalTable,softResetFinalTable,overwriteFinalTable,typeAndDedupe) transparently benefit because they all go throughdatabaseHandler.execute(...).Review guide
airbyte-integrations/connectors/destination-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/write/typing_deduping/BigQueryDatabaseHandler.kt— retry logic.airbyte-integrations/connectors/destination-bigquery/src/test/kotlin/io/airbyte/integrations/destination/bigquery/BigQueryDatabaseHandlerTest.kt— new unit tests for the retry path and error matcher.airbyte-integrations/connectors/destination-bigquery/metadata.yaml— version bump to 3.0.19.docs/integrations/destinations/bigquery.md— changelog entry.Test Coverage
Unit tests added to
BigQueryDatabaseHandlerTest:isConcurrentUpdateError matches by top-level messageisConcurrentUpdateError matches by nested error messageisConcurrentUpdateError does not match unrelated errorsexecute retries on concurrent-update error then succeeds— verifies the job is submitted 3 times when the first two attempts abort and the third succeeds.execute rethrows concurrent-update error after exhausting retries— verifies the job is submitted exactlyCONCURRENT_UPDATE_MAX_ATTEMPTS(5) times when every attempt aborts, and that the final exception is re-thrown.execute does not retry on non-concurrent-update errors— regression guard so unrelated BigQuery errors still fail fast.These tests fail against the prior
execute()implementation (which did not retry) and pass with the new implementation.Breaking Change Evaluation Gate:
3.0.18→3.0.19.enableProgressiveRollout: false— no-rc.xsuffix needed.User Impact
Transaction is aborted due to concurrent updateduring typing+deduping will now automatically retry up to 5 times before surfacing the error. In the common case (a transient concurrent write from a dbt job, another sync, or an analyst query), the retry will succeed and the sync will complete normally instead of failing.Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/8af8f94c618c4117b2cb6690605305c2