fix(destination-bigquery): retry transient timeouts and concurrent-update aborts during typing+deduping - #76902
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
|
|
↪️ Triggering Reason: Draft connector PR has passing CI and no prior |
|
Fix Validation EvidenceStatus: Evidence plan ready; approval requested for live validation. Current outcome: Pre-flight passed. Pre-release image exists: Next gate: explicit human approval is required before pinning any live connection to the pre-release. Approval request sent via Slack: https://airbytehq-team.slack.com/archives/C0AEXV81Q7N/p1778415224833789 Pre-flight checks
Evidence planProving criteria A live destination-bigquery sync using typing/deduping completes successfully on the pre-release, with logs confirming it ran Disproving criteria A pinned sync fails in the same typing/deduping phase with Inconclusive criteria Source-side/check failures, unrelated destination config errors, no typing/deduping activity, or cancellation/timeouts unrelated to BigQuery query execution. Inconclusive pins will be removed before trying another case. Testing strategy
Candidate selection summary Private oncall candidates were reviewed in the private issue. Several directly affected connections are currently pinned to older destination versions, so they are not safe as first-choice prove-fix candidates without broader coordination. The best first candidate found so far is an affected, unpinned, recent-success connection from the private oncall list. A safe internal fallback is also available if approval is limited to Airbyte-owned workspace testing. Pre-release publish details
|
|
…date aborts during typing+deduping Co-Authored-By: bot_apk <apk@cognition.ai>
b1b2529 to
8380f28
Compare
Detected
|
|
🙋 Escalating to human via #human-in-the-loop per Hands-Free AI Triage Project triage next step. Reason: Draft PR, Target: Aaron ("AJ") Steers (@aaronsteers) (cc Aaron ("AJ") Steers (@aaronsteers) if not target) |
|
Triggering |
|
🧪 Fix Validation EvidenceStatus: ⏳ Pre-release published. Awaiting human approval to pin sandbox connection for live validation. Connector: Current outcome: Pre-flight passed. Pre-release image published: Next gate: Explicit human approval required before pinning any live connection to the pre-release. Pre-flight checks
Evidence planProving criteria A live destination-bigquery sync using typing/deduping completes successfully on the pre-release, with logs confirming it ran Disproving criteria A pinned sync fails in the same typing/deduping phase with Inconclusive criteria Source-side/check failures, unrelated destination config errors, no typing/deduping activity, or cancellation/timeouts unrelated to BigQuery query execution. Inconclusive pins will be removed before trying another case. Testing strategy
|
|
|
|
↪️ Triggering Reason: this fix PR is Separately, its |
|
❌ Cannot revive Devin session - the session is too old. Please start a new session instead. |
|
…y-transient-timeout Co-Authored-By: bot_apk <apk@cognition.ai>
|
🔀 Resolved merge conflict with Resolution: only version/changelog collisions conflicted — Pushed the resolved branch; its CI and review gates will re-run on the new HEAD. |
|
↪️ Triggering |
|
🤖 /ai-prove-fix — destination-bigquery transient-retry validation: ⚪ Could Not Test (live validation not approved in time)Validating this PR's fix (retry transient Outcome: ⚪ Could Not TestLive validation could not be executed: the required human (Slack HITL) approval for pinning one internal connection to the pre-release was requested twice (initial + reminder) but did not arrive within ~24h, so no live pin/sync was performed. No production connection was modified. Status
Supporting (non-live) evidence
Recommended next steps
Pre-flight assessment (current HEAD b1bc4f8)
Evidence plan
This comment is updated in place as validation progresses. |
|
|
🙋 Escalated to #human-in-the-loop per Hands-Free AI Triage Project triage next step. Reason: |
2 participants
What
Resolves https://github.com/airbytehq/oncall/issues/10855:
BigQuery occasionally surfaces transient
BigQueryExceptionerrors to the destination connector during the typing+deduping job submit-and-poll loop inBigQueryDatabaseHandler.execute(Sql). Two distinct manifestations were observed in customer syncs:job.reload()throws aBigQueryException(client-side HTTP timeout, 5xx, or network blip).job.status.erroris populated with aBigQueryErrorwhose message isRequest timed out. Please try again. at [2:1]— BigQuery's server-side transient timeout. The[2:1]suffix is BigQuery's standard SQL position indicator; the failure has nothing to do with the SQL itself.Neither case is retried today. The existing
executeWithRetrieswrapper only handles HTTP 403rateLimitExceeded, and Sentry grouping by the unique transaction ID / position suffix means each occurrence lands as a new issue. Review of the oncall Sentry workspace list shows multiple distinct workspaces affected with recurring events per connection, consistent with transient server-side timeouts that would succeed on retry.This PR introduces a single unified retry wrapper that handles both error classes — transient timeouts and concurrent-update aborts.
How
In
BigQueryDatabaseHandler:submitAndPoll(statement).runQueryWithTransientRetries(queryId, statement), which:BigQueryExceptionfrombq.create(...)/job.reload()and classifies it viaisTransientException(e):e.isRetryable == true, ORe.messagecontains"Request timed out. Please try again."or"Transaction is aborted due to concurrent update", ORBigQueryErrorine.errorsmatches the same message substrings.job.status.erroron terminal jobs and classifies viaisTransientError(errors)against the same message substrings.attemptNumber/numAttemptson each retry.wrapWithConfigExceptionIfNeeded) on any non-transient error — syntax errors, schema errors, billing errors, etc. all still surface on the first attempt with no additional delay.No CDK-level changes are needed; all callers (
createFinalTable,softResetFinalTable,overwriteFinalTable,typeAndDedupein the CDK'sTypingDedupingFinalTableOperations) go throughdatabaseHandler.execute(...)and benefit transparently.Review guide
airbyte-integrations/connectors/destination-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/write/typing_deduping/BigQueryDatabaseHandler.kt— newrunQueryWithTransientRetries,submitAndPoll, andisTransient*helpers.execute(sql)now delegates to the retry wrapper.airbyte-integrations/connectors/destination-bigquery/src/test/kotlin/io/airbyte/integrations/destination/bigquery/BigQueryDatabaseHandlerTest.kt— new unit tests covering the classifier functions and end-to-end retry behavior throughexecute(sql).airbyte-integrations/connectors/destination-bigquery/metadata.yaml— version bump3.0.19→3.0.20.docs/integrations/destinations/bigquery.md— changelog entry.Test Coverage
New unit tests in
BigQueryDatabaseHandlerTest:Classifier tests:
isTransientError matches request-timeout messageisTransientError matches concurrent-update messageisTransientError does not match syntax errorisTransientException classifies isRetryable as transientisTransientException classifies message-based timeout as transientisTransientException classifies 5xx as transientisTransientException does not classify syntax error as transientEnd-to-end retry behavior tests (exercising
handler.execute(sql)):execute retries when job status reports transient timeout and succeeds on retry— fails 2x onjob.status.error, succeeds on 3rd; verifiesbq.createwas called 3 times.execute retries when BigQuery create throws transient exception then succeeds—bq.createthrowsBigQueryException(isRetryable=true, code=503)then returns a clean job; verifies 2 create calls.execute fails fast on non-transient job-status error without retrying— syntax error surfaces after exactly 1 create call.execute throws after exhausting all transient-retry attempts— concurrent-update error every attempt; verifies exactlyTRANSIENT_RETRY_MAX_ATTEMPTS(5) create calls and that the final exception preserves the transient error.Breaking Change Evaluation
3.0.19→3.0.20.enableProgressiveRollout: falseinmetadata.yaml— no-rc.xsuffix needed.User Impact
com.google.cloud.bigquery.BigQueryException: Request timed out. Please try again.during typing+deduping will now automatically retry up to 5 times before surfacing the error. In the common case (a brief BigQuery-side timeout), the retry will succeed and the sync will complete normally.1 + 2 + 4 + 8 + 16 = 31seconds of backoff plus jitter — negligible next to the typing+deduping queries themselves.Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/ea8e6ca7dcb54e99bb2ef7f5ce9d848a
Important
Active progressive rollout warning for destination-bigquery.
Important
Autopilot Progressive Rollout Enabled
Autopilot progressive rollouts are enabled for one or more connector(s) modified in this PR. Check the box below if you need to bypass normal rollout safety processes and release to all users immediately upon merge:
Note: