fix(destination-bigquery): surface interrupted standard-inserts operations as transient errors - #83276
Conversation
Co-Authored-By: bot_apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: bot_apk <apk@cognition.ai>
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
Co-Authored-By: bot_apk <apk@cognition.ai>
|
|
↪️ Triggering Reason: Draft |
|
🧪 Fix Validation Evidence⚪ Outcome: Could Not Test (live validation pending human approval)Pre-flight checks passed and pre-release 🚦 Next Steps
📋 Connector & PR DetailsConnector: 📝 Evidence PlanProving CriteriaA cancelled/interrupted sync on the pre-release surfaces the new message "The BigQuery operation was interrupted, likely because the sync was cancelled..." classified as Disproving CriteriaInterrupted syncs still surface raw Cases Attempted
✅ Pre-flight Checks
📊 Detailed Evidence Log
Note: Connection IDs and detailed logs are recorded in the linked private issue. |
Airbyte Support Bot (airbyte-support-bot)
added
the
hyd-prove
|
|
🙋 Escalating to a human via #human-in-the-loop per Hands-Free AI Triage Project triage next step. Reason: |
2 participants
What
Resolves https://github.com/airbytehq/oncall/issues/13206:
When a sync is cancelled/torn down (e.g. because the source failed — in the reported job it was a HubSpot 401), the destination's direct-load worker thread is interrupted while inside
BigqueryBatchStandardInsertsLoader.switchToWriteChannel(). The BigQuery SDK wraps the interrupt inBigQueryException(cause=InterruptedException), and the loader rethrew it as a brand-newBigQueryException(e.code, e.message)— dropping the cause. The bulk CDK'sDefaultExceptionClassifierthen falls through toSystemError, so users see a rawjava.lang.InterruptedExceptionclassified assystem_error(Sentry: https://airbytehq.sentry.io/issues/7643588179/).How
BigQueryUtils.executeBigQueryOperation { ... }: catchesBigQueryException, walks the full cause chain forInterruptedException; if found, restores the thread interrupt flag and throwsTransientErrorException("The BigQuery operation was interrupted, likely because the sync was cancelled. ...", cause = original). Same semantics as the utility introduced in the (still-open) fix(destination-bigquery): handle InterruptedException during sync cancellation #79168, so the two changes merge cleanly.BigqueryBatchStandardInsertsLoaderwith the utility:bigquery.writer(...),writer.write(...)(both call sites),writer.close(),BigQueryUtils.waitForJobFinish(...),writer.job.reload().switchToWriteChannel()keeps 403/404 →ConfigErrorExceptionunchanged, and now preserves the original exception as cause when rethrowing otherBigQueryExceptions:BigQueryException(e.code, e.message, e).Review guide
BigQueryUtils.kt— newexecuteBigQueryOperation()utilitywrite/standard_insert/BigqueryBatchStandardInsertLoader.kt— wrapped blocking calls, cause preservationBigQueryUtilsExecuteBigQueryOperationTest.kt— utility unit tests (nested cause chain, interrupt flag, passthrough)write/standard_insert/BigqueryBatchStandardInsertsLoaderTest.kt— loader tests: interrupt →TransientErrorException, 403/404 →ConfigErrorExceptionunchanged, cause preserved on other errorsmetadata.yaml/docs/integrations/destinations/bigquery.md— version bump to3.0.24-rc.1(progressive rollout enabled) + changelogDeclarative-First Evaluation
Not applicable — this is a Java/Kotlin bulk-load connector, not a declarative/manifest connector.
Test Coverage
New unit tests cover the interruption conversion (including nested cause chains and interrupt-flag restoration), the unchanged 403/404 config-error path, and cause preservation for other BigQuery errors. Local Gradle runs were blocked by Maven Central HTTP 429 rate limiting during dependency resolution; relying on CI unit tests for execution.
User Impact
An interrupted BigQuery operation during sync teardown now surfaces as a transient error with a readable message instead of a raw
java.lang.InterruptedExceptionclassified assystem_error. Reduces Sentry noise; no behavior change for real config or system errors.Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/9da8f5a89a5c472083d9203bec6c5d3d