Skip to content

DRIVERS-2344: add disambiguatedPaths to change streams update document #1292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
durran merged 6 commits into from
Aug 19, 2022
Merged

DRIVERS-2344: add disambiguatedPaths to change streams update document #1292

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
72dc6ea
add disambiguatedPaths to change streams update document
baileympearson Aug 17, 2022
4c9b173
chore: update changelog and bump spec version
baileympearson Aug 17, 2022
fcca0f8
chore: remove unintentional change
baileympearson Aug 17, 2022
2cd8cff
chore: remove unintentional change
baileympearson Aug 17, 2022
3023642
Address Benji's comment
baileympearson Aug 18, 2022
f898171
chore: bump updated at date
durran Aug 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion source/change-streams/change-streams.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Change Streams
:Type: Standards
:Minimum Server Version: 3.6
:Last Modified: 2022-05-19
Comment thread
durran marked this conversation as resolved.
Outdated
:Version: 1.16
:Version: 1.17

.. contents::

Expand Down Expand Up @@ -307,6 +307,26 @@ If an aggregate command with a ``$changeStream`` stage completes successfully, t
* @since 4.7.0
*/
truncatedArrays: Array<Document>;

/**
* A document containing a map that associates an update path to an array containing the path components used in the update document. This data
Comment thread
benjirewis marked this conversation as resolved.
Outdated
* can be used in combination with the other fields in an `UpdateDescription` to determine the actual path in the document that was updated. This is
* necessary in cases where a key contains a dot-separated strings (i.e., `{ "a.b": "c" }`) or a document contains a numeric literal string key
Comment thread
benjirewis marked this conversation as resolved.
Outdated
* (i.e., `{ "a": { "0": "a" } }`. Note that in this scenario, the numeric key can't be the top level key, because `{ "0": "a" }` is not ambiguous -
Comment thread
benjirewis marked this conversation as resolved.
Outdated
* update paths would simply be `'0'` which is unambiguous because BSON documents cannot have arrays at the top level.).
*
* Each entry in the document maps an update path to an array which contains the actual path used when the document was updated.
* For example, given a document with the following shape `{ "a": { "0": 0 } }` and an update of `{ $inc: { "a.0": 1 } }`, `disambiguatedPaths` would
* look like the following:
* {
* "a.0": ["a", "0"]
* }
*
* In each array, all elements will be returned as strings with the exception of array indices, which will be returned as 32 bit integers.
*
* @since 6.1.0
*/
disambiguatedPaths?: Document
Comment thread
benjirewis marked this conversation as resolved.
Outdated
}

The responses to a change stream aggregate or getMore have the following structures:
Expand Down Expand Up @@ -1070,3 +1090,5 @@ Changelog
+------------+------------------------------------------------------------+
| 2022-05-19 | Support new change stream events with showExpandedEvents. |
+------------+------------------------------------------------------------+
| 2022-08-17 | Support `disambiguatedPaths` in `UpdateDescription`. |
+------------+------------------------------------------------------------+
250 changes: 250 additions & 0 deletions source/change-streams/tests/unified/change-streams-disambiguatedPaths.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
{
"description": "disambiguatedPaths",
"schemaVersion": "1.9",
"createEntities": [
{
"client": {
"id": "client0",
"useMultipleMongoses": false
}
},
{
"database": {
"id": "database0",
"client": "client0",
"databaseName": "database0"
}
},
{
"collection": {
"id": "collection0",
"database": "database0",
"collectionName": "collection0"
}
}
],
"runOnRequirements": [
{
"minServerVersion": "6.1.0",
"topologies": [
"replicaset",
"sharded-replicaset",
"load-balanced"
]
}
],
"initialData": [
{
"collectionName": "collection0",
"databaseName": "database0",
"documents": []
}
],
"tests": [
{
"description": "disambiguatedPaths is not present when showExpandedEvents is false/unset",
"operations": [
{
"name": "insertOne",
"object": "collection0",
"arguments": {
"document": {
"_id": 1,
"a": {
"1": 1
}
}
}
},
{
"name": "createChangeStream",
"object": "collection0",
"arguments": {
"pipeline": []
},
"saveResultAsEntity": "changeStream0"
},
{
"name": "updateOne",
"object": "collection0",
"arguments": {
"filter": {
"_id": 1
},
"update": {
"$set": {
"a.1": 2
}
}
}
},
{
"name": "iterateUntilDocumentOrError",
"object": "changeStream0",
"expectResult": {
"operationType": "update",
"ns": {
"db": "database0",
"coll": "collection0"
},
"updateDescription": {
"updatedFields": {
"$$exists": true
},
"removedFields": {
"$$exists": true
},
"truncatedArrays": {
"$$exists": true
},
"disambiguatedPaths": {
"$$exists": false
}
}
}
}
]
},
{
"description": "disambiguatedPaths is present on updateDescription when an ambiguous path is present",
"operations": [
{
"name": "insertOne",
"object": "collection0",
"arguments": {
"document": {
"_id": 1,
"a": {
"1": 1
}
}
}
},
{
"name": "createChangeStream",
"object": "collection0",
"arguments": {
"pipeline": [],
"showExpandedEvents": true
},
"saveResultAsEntity": "changeStream0"
},
{
"name": "updateOne",
"object": "collection0",
"arguments": {
"filter": {
"_id": 1
},
"update": {
"$set": {
"a.1": 2
}
}
}
},
{
"name": "iterateUntilDocumentOrError",
"object": "changeStream0",
"expectResult": {
"operationType": "update",
"ns": {
"db": "database0",
"coll": "collection0"
},
"updateDescription": {
"updatedFields": {
"$$exists": true
},
"removedFields": {
"$$exists": true
},
"truncatedArrays": {
"$$exists": true
},
"disambiguatedPaths": {
"a.1": [
"a",
"1"
]
}
}
}
}
]
},
{
"description": "disambiguatedPaths returns array indices as integers",
"operations": [
{
"name": "insertOne",
"object": "collection0",
"arguments": {
"document": {
"_id": 1,
"a": [
{
"1": 1
}
]
}
}
},
{
"name": "createChangeStream",
"object": "collection0",
"arguments": {
"pipeline": [],
"showExpandedEvents": true
},
"saveResultAsEntity": "changeStream0"
},
{
"name": "updateOne",
"object": "collection0",
"arguments": {
"filter": {
"_id": 1
},
"update": {
"$set": {
"a.0.1": 2
}
}
}
},
{
"name": "iterateUntilDocumentOrError",
"object": "changeStream0",
"expectResult": {
"operationType": "update",
"ns": {
"db": "database0",
"coll": "collection0"
},
"updateDescription": {
"updatedFields": {
"$$exists": true
},
"removedFields": {
"$$exists": true
},
"truncatedArrays": {
"$$exists": true
},
"disambiguatedPaths": {
"a.0.1": [
"a",
{
"$$type": "int"
},
"1"
]
}
}
}
}
]
}
]
}
102 changes: 102 additions & 0 deletions source/change-streams/tests/unified/change-streams-disambiguatedPaths.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
description: "disambiguatedPaths"
schemaVersion: "1.9"
Comment thread
benjirewis marked this conversation as resolved.
Outdated
createEntities:
- client:
id: &client0 client0
useMultipleMongoses: false
- database:
id: &database0 database0
client: *client0
databaseName: *database0
- collection:
id: &collection0 collection0
database: *database0
collectionName: *collection0

runOnRequirements:
- minServerVersion: "6.1.0"
topologies: [ replicaset, sharded-replicaset, load-balanced ]

@benjirewis benjirewis Aug 18, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these new tests pass on sharded clusters + serverless? I would hope so, but we've had issues in the past with change streams tests on those topologies.

@baileympearson baileympearson Aug 18, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They pass on sharded replica sets - https://evergreen.mongodb.com/task_log_raw/mongo_node_driver_next_ubuntu_18.04_erbium_test_latest_sharded_cluster_patch_f28ba0de3bc925267a185965eb971ecf91bc2680_62fd236be3c3313be42d354a_22_08_17_17_20_44/0?type=T#L835
They pass on load balanced - https://evergreen.mongodb.com/task_log_raw/mongo_node_driver_next_ubuntu_18.04_erbium_test_latest_load_balanced_patch_f28ba0de3bc925267a185965eb971ecf91bc2680_62fd236be3c3313be42d354a_22_08_17_17_20_44/0?type=T#L751
They don't run on serverless - https://github.com/mongodb/specifications/tree/master/source/serverless-testing#other-tests

Sharded is odd because sharded clusters must be backed by replica sets as of 3.6 (https://www.mongodb.com/docs/manual/core/sharded-cluster-components/#sharded-cluster-components). So effectively they're the same now, I can add it as well to the list of topologies.

@benjirewis benjirewis Aug 18, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't run on serverless

The spec says:

Unified spec tests from all specifications MUST be run against Atlas Serverless.

Thanks for adding sharded; it's sort of a strange topology, but might as well add.


initialData:
- collectionName: *collection0
databaseName: *database0
documents: []

tests:
- description: "disambiguatedPaths is not present when showExpandedEvents is false/unset"
operations:
- name: insertOne
object: *collection0
arguments:
document: { _id: 1, 'a': { '1': 1 } }

@benjirewis benjirewis Aug 18, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[opt] You could use " to be consistent with the other spec tests.

- name: createChangeStream
object: *collection0
arguments: { pipeline: [] }
saveResultAsEntity: &changeStream0 changeStream0
- name: updateOne
object: *collection0
arguments:
filter: { _id: 1 }
update: { $set: { 'a.1': 2 } }
- name: iterateUntilDocumentOrError
object: *changeStream0
expectResult:
operationType: "update"
ns: { db: *database0, coll: *collection0 }
updateDescription:
updatedFields: { $$exists: true }
removedFields: { $$exists: true }
truncatedArrays: { $$exists: true }
disambiguatedPaths: { $$exists: false }

- description: "disambiguatedPaths is present on updateDescription when an ambiguous path is present"
operations:
- name: insertOne
object: *collection0
arguments:
document: { _id: 1, 'a': { '1': 1 } }
- name: createChangeStream
object: *collection0
arguments: { pipeline: [], showExpandedEvents: true }
saveResultAsEntity: &changeStream0 changeStream0
- name: updateOne
object: *collection0
arguments:
filter: { _id: 1 }
update: { $set: { 'a.1': 2 } }
- name: iterateUntilDocumentOrError
object: *changeStream0
expectResult:
operationType: "update"
ns: { db: *database0, coll: *collection0 }
updateDescription:
updatedFields: { $$exists: true }
removedFields: { $$exists: true }
truncatedArrays: { $$exists: true }
disambiguatedPaths: { 'a.1': ['a', '1'] }

- description: "disambiguatedPaths returns array indices as integers"
operations:
- name: insertOne
object: *collection0
arguments:
document: { _id: 1, 'a': [{'1': 1 }] }
- name: createChangeStream
object: *collection0
arguments: { pipeline: [], showExpandedEvents: true }
saveResultAsEntity: &changeStream0 changeStream0
- name: updateOne
object: *collection0
arguments:
filter: { _id: 1 }
update: { $set: { 'a.0.1': 2 } }
- name: iterateUntilDocumentOrError
object: *changeStream0
expectResult:
operationType: "update"
ns: { db: *database0, coll: *collection0 }
updateDescription:
updatedFields: { $$exists: true }
removedFields: { $$exists: true }
truncatedArrays: { $$exists: true }
disambiguatedPaths: { 'a.0.1': ['a', { $$type: 'int' }, '1'] }