fix(destination-bigquery): retry transient backend errors when opening write channel - #84296
Draft
devin-ai-integration[bot] wants to merge 4 commits into
Conversation
Co-Authored-By: bot_apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR targets PR #83276 (same function, open draft):
What
Related to https://github.com/airbytehq/oncall/issues/13286:
A single transient
BigQueryException: 503 Service Unavailablefrom BigQuery's backend fails the whole destination attempt and gets paged assystem_error. The exception comes from opening the resumable upload session inBigqueryBatchStandardInsertsLoader.switchToWriteChannel()—bigquery.writer(job, writeChannelConfiguration), i.e.POST /upload/bigquery/v2/projects/<project>/jobs?uploadType=resumable.That call is effectively not retried by the Google client, despite the connector configuring
RetrySettings(maxAttempts = 15)inBigqueryBeansFactory. Reading the client source:TableDataWriteChannel.open()does go throughBigQueryRetryHelper.runWithRetries, but the callable isHttpBigQueryRpc.openSkipExceptionTranslation(), which ends in a barehttpRequest.execute()and therefore throwsHttpResponseException(anIOException) rather than aBaseServiceException.BigQueryOptionsdefaults toBigQueryBaseService.DEFAULT_BIGQUERY_EXCEPTION_HANDLER, which retries onlyBaseServiceException.isRetryable()plusConnectException/UnknownHostException/SocketException— soExceptionHandler.shouldRetryreturnsNO_RETRYfor a 5xx on this path.BigQuery's error reference documents
backendErroras 500/502/503/504, service-side, and says the client should retry with truncated exponential backoff (and thatjobs.insertshould be retried).The loader's
catchblock then converted only 403/404 intoConfigErrorExceptionand rethrew everything else as a plainBigQueryException, which the bulk CDK's default classifier turns intosystem_error— that is what paged Sentry (airbyte-destination-bigquery@3.0.22).Caveat on the reported job: for the customer job in the oncall issue, 15 of 16 attempts failed source-side (TikTok Marketing "transient server-side error (code 50000)"); the BigQuery 503 only hit the final attempt. This PR closes a real destination-side gap (no retry + wrong failure classification), but it would not have made that particular job succeed. Prior low-volume reports of the same Sentry issue: https://github.com/airbytehq/oncall/issues/9724, https://github.com/airbytehq/oncall/issues/10592, https://github.com/airbytehq/oncall/issues/12881, https://github.com/airbytehq/oncall/issues/13046.
How
In
switchToWriteChannel(), the write-channel open is now attempted in a bounded retry loop:warnwith the attempt number, HTTP code, and message. Backoff/jitter style follows the transient-retry loop in the (also open) fix(destination-bigquery): retry transient timeouts and concurrent-update aborts during typing+deduping #76902 / fix(destination-bigquery): retry typing+deduping queries on concurrent-update aborts #76440 work onBigQueryDatabaseHandler.ConfigErrorException(CONFIG_ERROR_MSG + e), and any other non-retryable code is still rethrown immediately asBigQueryException(e.code, e.message, e)— unchanged behavior.TransientErrorExceptionwith the lastBigQueryExceptionas cause, so the bulk CDK classifies it transient instead ofsystem_error.JobId; initiating a resumable upload session does not create the load job, so re-initiating is well-defined.BigqueryBatchStandardInsertsLoaderFactory.create()is unchanged.Relationship to other open work on this same function. This PR is stacked on #83276, which wraps the blocking BigQuery calls in
BigQueryUtils.executeBigQueryOperation { ... }(interrupt →TransientErrorException) and preserves the cause when rethrowing. The retry loop keeps the open call inside that utility, so the interrupt semantics from #83276 are preserved and the two changes don't conflict. #77873 ("handle missing load job") also touches this loader; it editsfinish(), not the open path. If #83276 lands or closes, this branch can be rebased and retargeted tomaster.Review guide
write/standard_insert/BigqueryBatchStandardInsertLoader.kt— retry loop, status-code classification, exhaustion →TransientErrorException, retry-parameter seamswrite/standard_insert/BigqueryBatchStandardInsertsLoaderTest.kt— retry-then-success, 403/404 fail-fast (no retries), exhaustion, non-retryable 400 rethrowmetadata.yaml/docs/integrations/destinations/bigquery.md—3.0.25-rc.1(progressive rollout enabled) + changelogTest Coverage
New mockk unit tests in
BigqueryBatchStandardInsertsLoaderTest:bigquery.writercalled 3 timesConfigErrorException,bigquery.writercalled exactly onceTransientErrorExceptionwith theBigQueryExceptionas cause, called exactlymaxOpenAttemptstimesBigQueryExceptionrethrown immediately with cause preserved, called onceNot verified locally: Gradle could not resolve dependencies on this machine — Maven Central returned HTTP 429 for
software.amazon.awssdk:{s3,sts,sso,ssooidc}:2.22.10across several spaced retries — socompileKotlin/compileTestKotlin/testnever ran. Compilation and test execution are relying on CI here; please treat the CI unit-test result as the first real signal.Declarative-First Evaluation
Not applicable — Java/Kotlin bulk-load destination, not a declarative/manifest connector.
Breaking Change Evaluation
Not breaking: no schema, spec, state, stream, or data-scope change. Retrying a failed open and reclassifying an exhausted retry as transient only affects failure paths. Versioned as
3.0.25-rc.1because this connector hasenableProgressiveRollout: true.User Impact
A transient BigQuery backend blip while opening a standard-inserts load job no longer fails the sync attempt outright — it is retried for up to ~45s of backoff. If BigQuery is unavailable for longer than that, the attempt fails as a transient error with a readable message instead of a raw 503 classified as
system_error, which should also stop this paging Sentry as a system error.Can this PR be safely reverted and rolled back?
Devin session — requested via
/ai-fixon https://github.com/airbytehq/oncall/issues/13286 by the Airbyte oncall workflow.