Skip to content

Add transactional batch support for Cosmos DB Spark connector#47478

Merged
TheovanKraay merged 16 commits into
Azure:mainfrom
TheovanKraay:spark-transactional-batch-support-v2
Jan 13, 2026
Merged

Add transactional batch support for Cosmos DB Spark connector#47478
TheovanKraay merged 16 commits into
Azure:mainfrom
TheovanKraay:spark-transactional-batch-support-v2

fix broken link since doc refresh

ee7de52
Select commit
Loading
Failed to load commit list.
Azure Pipelines / java - cosmos - ci succeeded Jan 12, 2026 in 1h 0m 46s

Build #20260112.3 had test failures

Details

Tests

  • Failed: 2 (0.00%)
  • Passed: 71,386 (99.01%)
  • Other: 715 (0.99%)
  • Total: 72,103
Code coverage

  • 2392 of 80740 lines covered (2.96%)

Annotations

Check failure on line 26 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / java - cosmos - ci

Build log #L26

PowerShell exited with code '1'.

Check failure on line 25 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / java - cosmos - ci

Build log #L25

PowerShell exited with code '1'.

Check failure on line 26 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / java - cosmos - ci

Build log #L26

PowerShell exited with code '1'.

Check failure on line 25 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / java - cosmos - ci

Build log #L25

PowerShell exited with code '1'.

Check failure on line 1 in c.a.c.s.PointWriterITest.Point Writer can create item with duplicates

See this annotation in the file changed.

@azure-pipelines azure-pipelines / java - cosmos - ci

c.a.c.s.PointWriterITest.Point Writer can create item with duplicates

{"innerErrorMessage":"{\"innerErrorMessage\":\"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x3f92e3c5, L:/127.0.0.1:56628 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests\"}","cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.77.0-snapshot.1 WindowsServer2022/10.0 JRE/11.0.29","activityId":"33c616f5-ac85-488e-9907-e500ed02cae4","requestLatencyInMs":2579,"requestStartTimeUTC":"2026-01-12T16:17:18.189597800Z","requestEndTimeUTC":"2026-01-12T16:17:20.769468Z","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer48/partitions/a4cb497c-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"quorumAckedLSN":-1,"currentReplicaSetSize":-1,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x3f92e3c5, L:/127.0.0.1:56628 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests","exceptionResponseHeaders":"{x-ms-substatus=0}","replicaStatusList":{"Ignoring":["10253:S:Connected"],"Attempting":["10253:P:Connected"]},"transportRequestTimeline":[{"eventName":"created","startTimeUTC":"2026-01-12T16:17:18.189597800Z","durationInMilliSecs":0.0},{"eventName":"queued","startTimeUTC":"2026-01-12T16:17:18.189597800Z","durationInMilliSecs":0.0},{"eventName":"channelAcquisitionStarted","startTimeUTC":"2026-01-12T16:17:18.189597800Z","durationInMilliSecs":171.0008},{"eventName":"pipelined","startTimeUTC":"2026-01-12T16:17:18.360598600Z","durationInMilliSecs":0.0},{"eventName":"transitTime","startTimeUTC":"2026-01-12T16:17:18.360598600Z","durationInMilliSecs":10.9962},{"eventName":"decodeTime","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"received","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"completed","startTimeUTC":"2026-01-12T16:17:18.371594800Z","durationInMilliSecs":0.0}],"rntbdRequestLengthInBytes":511,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":102,"responsePayloadLengthInBytes":-1,"channelStatistics":{"channelId":"3f92e3c5","channelTaskQueueSize":30,"pendingRequestsCount":0,"lastReadTime":"2026-01-12T16:17:18.211587700Z","waitForConnectionInit":true},"serviceEndpointStatistics":{"availableChannels":36,"acquiredChannels":0,"executorTaskQueueSize":36,"inflightRequests":67,"lastSuccessfulRequestTime":"2026-01-12T16:17:18.166Z","lastRequestTime":"2026-01-12T16:17:18.190Z","createdTime":"2026-01-12T16:16:54.469881600Z","isClosed":false,"cerMetrics":{}}},"requestResponseTimeUTC":"2026-01-12T16:17:18.374605700Z","requestStartTimeUTC":"2026-01-12T16:17:18.189597800Z","requestResourceType":"Document","requestOperationType":"Create","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"perPartitionCircuitBreakerInfoHolder":null,"perPartitionFailoverInfoHolder":null}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{"829ab7e3-0921-4c52-89c7-61651b6840fb":{"startTimeUTC":"2026-01-12T16:17:18.418594Z","endTimeUTC":"2026-01-12T16:17:18.919596600Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2F4XYiAA%3D%3D%2Fcolls%2F4XYiAPDL5W4%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","exceptionMessage":"io.netty.handler.timeout.ReadTimeoutException","forceRefresh":true,"forceCollectionRoutingMapRefresh":false,"inflightRequest":false},"8b2121ec-71b6-4d9a-ab3d-6ee20841db7f":{"startTimeUTC":"2026-01-12T16:17:18.966592700Z","endTimeUTC":"2026-01-12T16:17:20.769468Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2F4XYiAA%3D%3D%2Fcolls%2F4XYiAPDL5W4%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","exceptionMessage":null,"forceRefresh":true,"forceCol
Raw output
      {"ClassName":"CosmosException","userAgent":"azsdk-java-cosmos/4.77.0-snapshot.1 WindowsServer2022/10.0 JRE/11.0.29","statusCode":408,"resourceAddress":null,"innerErrorMessage":"{\"innerErrorMessage\":\"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x3f92e3c5, L:/127.0.0.1:56628 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests\"}","causeInfo":"[class: class com.azure.cosmos.implementation.GoneException, message: {\"innerErrorMessage\":\"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x3f92e3c5, L:/127.0.0.1:56628 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests\"}]","responseHeaders":"{x-ms-substatus=0}","cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.77.0-snapshot.1 WindowsServer2022/10.0 JRE/11.0.29","activityId":"33c616f5-ac85-488e-9907-e500ed02cae4","requestLatencyInMs":2579,"requestStartTimeUTC":"2026-01-12T16:17:18.189597800Z","requestEndTimeUTC":"2026-01-12T16:17:20.769468Z","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer48/partitions/a4cb497c-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"quorumAckedLSN":-1,"currentReplicaSetSize":-1,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x3f92e3c5, L:/127.0.0.1:56628 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests","exceptionResponseHeaders":"{x-ms-substatus=0}","replicaStatusList":{"Ignoring":["10253:S:Connected"],"Attempting":["10253:P:Connected"]},"transportRequestTimeline":[{"eventName":"created","startTimeUTC":"2026-01-12T16:17:18.189597800Z","durationInMilliSecs":0.0},{"eventName":"queued","startTimeUTC":"2026-01-12T16:17:18.189597800Z","durationInMilliSecs":0.0},{"eventName":"channelAcquisitionStarted","startTimeUTC":"2026-01-12T16:17:18.189597800Z","durationInMilliSecs":171.0008},{"eventName":"pipelined","startTimeUTC":"2026-01-12T16:17:18.360598600Z","durationInMilliSecs":0.0},{"eventName":"transitTime","startTimeUTC":"2026-01-12T16:17:18.360598600Z","durationInMilliSecs":10.9962},{"eventName":"decodeTime","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"received","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"completed","startTimeUTC":"2026-01-12T16:17:18.371594800Z","durationInMilliSecs":0.0}],"rntbdRequestLengthInBytes":511,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":102,"responsePayloadLengthInBytes":-1,"channelStatistics":{"channelId":"3f92e3c5","channelTaskQueueSize":30,"pendingRequestsCount":0,"lastReadTime":"2026-01-12T16:17:18.211587700Z","waitForConnectionInit":true},"serviceEndpointStatistics":{"availableChannels":36,"acquiredChannels":0,"executorTaskQueueSize":36,"inflightRequests":67,"lastSuccessfulRequestTime":"2026-01-12T16:17:18.166Z","lastRequestTime":"2026-01-12T16:17:18.190Z","createdTime":"2026-01-12T16:16:54.469881600Z","isClosed":false,"cerMetrics":{}}},"requestResponseTimeUTC":"2026-01-12T16:17:18.374605700Z","requestStartTimeUTC":"2026-01-12T16:17:18.189597800Z","requestResourceType":"Document","requestOperationType":"Create","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"perPartitionCircuitBreakerInfoHolder":null,"perPartitionFailoverInfoHolder":null}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{"829ab7e3-0921-4c52-89c7-61651b6840fb":{"startTimeUTC":"2026-01-12T16:17:18.418594Z","endTimeUTC":"2026-01-12T16:17:18.919596600Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2F4XYiAA%3D%3D%2Fcolls%2F4XYiAPDL5W4%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","exceptionMessage":"io.netty.handler.timeout

Check failure on line 1 in c.a.c.s.PointWriterITest.Point Writer can create item with duplicates

See this annotation in the file changed.

@azure-pipelines azure-pipelines / java - cosmos - ci

c.a.c.s.PointWriterITest.Point Writer can create item with duplicates

{"innerErrorMessage":"{\"innerErrorMessage\":\"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x27a885cc, L:/127.0.0.1:59598 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests\"}","cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.77.0-snapshot.1 WindowsServer2022/10.0 JRE/1.8.0_472","activityId":"5639ee98-fb44-4f04-8c65-ed47fc07e894","requestLatencyInMs":671,"requestStartTimeUTC":"2026-01-12T16:23:32.284Z","requestEndTimeUTC":"2026-01-12T16:23:32.955Z","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer37/partitions/a4cb4971-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"quorumAckedLSN":-1,"currentReplicaSetSize":-1,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x27a885cc, L:/127.0.0.1:59598 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests","exceptionResponseHeaders":"{x-ms-substatus=0}","replicaStatusList":{"Ignoring":["10253:S:Connected"],"Attempting":["10253:P:Connected"]},"transportRequestTimeline":[{"eventName":"created","startTimeUTC":"2026-01-12T16:23:32.284Z","durationInMilliSecs":0.0},{"eventName":"queued","startTimeUTC":"2026-01-12T16:23:32.284Z","durationInMilliSecs":0.0},{"eventName":"channelAcquisitionStarted","startTimeUTC":"2026-01-12T16:23:32.284Z","durationInMilliSecs":63.0},{"eventName":"pipelined","startTimeUTC":"2026-01-12T16:23:32.347Z","durationInMilliSecs":68.0},{"eventName":"transitTime","startTimeUTC":"2026-01-12T16:23:32.415Z","durationInMilliSecs":15.0},{"eventName":"decodeTime","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"received","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"completed","startTimeUTC":"2026-01-12T16:23:32.430Z","durationInMilliSecs":0.0}],"rntbdRequestLengthInBytes":512,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":101,"responsePayloadLengthInBytes":-1,"channelStatistics":{"channelId":"27a885cc","channelTaskQueueSize":0,"pendingRequestsCount":0,"lastReadTime":"2026-01-12T16:23:32.347Z","waitForConnectionInit":false},"serviceEndpointStatistics":{"availableChannels":1,"acquiredChannels":23,"executorTaskQueueSize":2,"inflightRequests":40,"lastSuccessfulRequestTime":"2026-01-12T16:23:32.275Z","lastRequestTime":"2026-01-12T16:23:32.277Z","createdTime":"2026-01-12T16:23:05.921Z","isClosed":false,"cerMetrics":{}}},"requestResponseTimeUTC":"2026-01-12T16:23:32.431Z","requestStartTimeUTC":"2026-01-12T16:23:32.284Z","requestResourceType":"Document","requestOperationType":"Create","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"perPartitionCircuitBreakerInfoHolder":null,"perPartitionFailoverInfoHolder":null}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{"67843097-01a0-4418-b8e9-f8ed17083e06":{"startTimeUTC":"2026-01-12T16:23:32.454Z","endTimeUTC":"2026-01-12T16:23:32.955Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2F-oBBAA%3D%3D%2Fcolls%2F-oBBAMpKVpE%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","exceptionMessage":"io.netty.handler.timeout.ReadTimeoutException","forceRefresh":true,"forceCollectionRoutingMapRefresh":false,"inflightRequest":false},"ce05afb9-66d5-46d4-adf5-63f510281f28":{"startTimeUTC":"2026-01-12T16:23:32.980Z","endTimeUTC":null,"targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2F-oBBAA%3D%3D%2Fcolls%2F-oBBAMpKVpE%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","exceptionMessage":null,"forceRefresh":true,"forceCollectionRoutingMapRefresh":false,"inflightRequest":true}},"regionsContacted":["south central us"],"retryContext":{"s
Raw output
      {"ClassName":"CosmosException","userAgent":"azsdk-java-cosmos/4.77.0-snapshot.1 WindowsServer2022/10.0 JRE/1.8.0_472","statusCode":408,"resourceAddress":null,"innerErrorMessage":"{\"innerErrorMessage\":\"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x27a885cc, L:/127.0.0.1:59598 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests\"}","causeInfo":"[class: class com.azure.cosmos.implementation.GoneException, message: {\"innerErrorMessage\":\"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x27a885cc, L:/127.0.0.1:59598 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests\"}]","responseHeaders":"{x-ms-substatus=0}","cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.77.0-snapshot.1 WindowsServer2022/10.0 JRE/1.8.0_472","activityId":"5639ee98-fb44-4f04-8c65-ed47fc07e894","requestLatencyInMs":2643,"requestStartTimeUTC":"2026-01-12T16:23:32.284Z","requestEndTimeUTC":"2026-01-12T16:23:34.927Z","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer37/partitions/a4cb4971-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":0,"quorumAckedLSN":-1,"currentReplicaSetSize":-1,"globalCommittedLsn":-1,"partitionKeyRangeId":null,"isValid":true,"statusCode":410,"subStatusCode":0,"isGone":true,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"ChannelHandlerContext(RntbdRequestManager#0, [id: 0x27a885cc, L:/127.0.0.1:59598 - R:/127.0.0.1:10253]) closed exceptionally with 1 pending requests","exceptionResponseHeaders":"{x-ms-substatus=0}","replicaStatusList":{"Ignoring":["10253:S:Connected"],"Attempting":["10253:P:Connected"]},"transportRequestTimeline":[{"eventName":"created","startTimeUTC":"2026-01-12T16:23:32.284Z","durationInMilliSecs":0.0},{"eventName":"queued","startTimeUTC":"2026-01-12T16:23:32.284Z","durationInMilliSecs":0.0},{"eventName":"channelAcquisitionStarted","startTimeUTC":"2026-01-12T16:23:32.284Z","durationInMilliSecs":63.0},{"eventName":"pipelined","startTimeUTC":"2026-01-12T16:23:32.347Z","durationInMilliSecs":68.0},{"eventName":"transitTime","startTimeUTC":"2026-01-12T16:23:32.415Z","durationInMilliSecs":15.0},{"eventName":"decodeTime","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"received","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"completed","startTimeUTC":"2026-01-12T16:23:32.430Z","durationInMilliSecs":0.0}],"rntbdRequestLengthInBytes":512,"rntbdResponseLengthInBytes":-1,"requestPayloadLengthInBytes":101,"responsePayloadLengthInBytes":-1,"channelStatistics":{"channelId":"27a885cc","channelTaskQueueSize":0,"pendingRequestsCount":0,"lastReadTime":"2026-01-12T16:23:32.347Z","waitForConnectionInit":false},"serviceEndpointStatistics":{"availableChannels":1,"acquiredChannels":23,"executorTaskQueueSize":2,"inflightRequests":40,"lastSuccessfulRequestTime":"2026-01-12T16:23:32.275Z","lastRequestTime":"2026-01-12T16:23:32.277Z","createdTime":"2026-01-12T16:23:05.921Z","isClosed":false,"cerMetrics":{}}},"requestResponseTimeUTC":"2026-01-12T16:23:32.431Z","requestStartTimeUTC":"2026-01-12T16:23:32.284Z","requestResourceType":"Document","requestOperationType":"Create","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"perPartitionCircuitBreakerInfoHolder":null,"perPartitionFailoverInfoHolder":null}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{"67843097-01a0-4418-b8e9-f8ed17083e06":{"startTimeUTC":"2026-01-12T16:23:32.454Z","endTimeUTC":"2026-01-12T16:23:32.955Z","targetEndpoint":"https://127.0.0.1:8081/addresses/?$resolveFor=dbs%2F-oBBAA%3D%3D%2Fcolls%2F-oBBAMpKVpE%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0","exceptionMessage":"io.netty.handler.timeout.ReadTimeoutException","forceRefresh":true,"forceCollectionRoutingMapRefresh":fal