Use optimistic concurrency in SO migrations#231406
Merged
jesuswr merged 32 commits intoelastic:mainfrom Sep 1, 2025
Merged
Conversation
Member
Author
|
/ci 🙏 |
Member
Author
|
/ci |
Member
Author
|
/ci |
Member
Author
|
/ci |
Member
Author
|
/ci |
Member
Author
|
/ci |
rudolf
approved these changes
Sep 1, 2025
.../migration-server-internal/src/zdt/model/stages/outdated_documents_search_bulk_index.test.ts
Outdated
Show resolved
Hide resolved
|
|
||
| const originalBulkImplementation = client.bulk; | ||
| const spy = jest.spyOn(client, 'bulk'); | ||
| spy.mockImplementation(function (this: typeof client, ...args) { |
Contributor
There was a problem hiding this comment.
clever way to introduce the updates at just the right time in the migration algorithm
…zdt/model/stages/outdated_documents_search_bulk_index.test.ts Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
gsoldevila
approved these changes
Sep 1, 2025
Contributor
There was a problem hiding this comment.
Fantastic job! 🚀
So IIUC Optimistic Concurrency Control was simply NOT WORKING because we were not passing the seqNoPrimaryTerm: true, to the Search query 🤦🏼 .
I see that at the moment the "conflict" errors are simply filtered out, and we don't have any visibility at all if/when they happen. I wonder if there are scenarios where we would want to log the count of documents that we tried to update and failed due to version conflicts (can be discussed async).
Contributor
|
Starting backport for target branches: 8.18, 8.19, 9.0, 9.1 https://github.com/elastic/kibana/actions/runs/17383231253 |
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsTest FailuresMetrics [docs]
History
cc @jesuswr |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 1, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> (cherry picked from commit a5805f2)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 1, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> (cherry picked from commit a5805f2)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 1, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> (cherry picked from commit a5805f2)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 1, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> (cherry picked from commit a5805f2)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Sep 1, 2025
# Backport This will backport the following commits from `main` to `9.1`: - [Use optimistic concurrency in SO migrations (#231406)](#231406) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jesus Wahrman","email":"41008968+jesuswr@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-09-01T16:42:54Z","message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Feature:Saved Objects","release_note:skip","backport:prev-major","backport:current-major","v9.2.0"],"title":"Use optimistic concurrency in SO migrations","number":231406,"url":"https://github.com/elastic/kibana/pull/231406","mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/231406","number":231406,"mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}}]}] BACKPORT--> Co-authored-by: Jesus Wahrman <41008968+jesuswr@users.noreply.github.com> Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
8 tasks
kibanamachine
added a commit
that referenced
this pull request
Sep 2, 2025
# Backport This will backport the following commits from `main` to `8.19`: - [Use optimistic concurrency in SO migrations (#231406)](#231406) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jesus Wahrman","email":"41008968+jesuswr@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-09-01T16:42:54Z","message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Feature:Saved Objects","release_note:skip","backport:prev-major","backport:current-major","v9.2.0"],"title":"Use optimistic concurrency in SO migrations","number":231406,"url":"https://github.com/elastic/kibana/pull/231406","mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/231406","number":231406,"mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}}]}] BACKPORT--> --------- Co-authored-by: Jesus Wahrman <41008968+jesuswr@users.noreply.github.com> Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> Co-authored-by: Jesus Wahrman <jesus.wahrman@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Sep 2, 2025
# Backport This will backport the following commits from `main` to `9.0`: - [Use optimistic concurrency in SO migrations (#231406)](#231406) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jesus Wahrman","email":"41008968+jesuswr@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-09-01T16:42:54Z","message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Feature:Saved Objects","release_note:skip","backport:prev-major","backport:current-major","v9.2.0"],"title":"Use optimistic concurrency in SO migrations","number":231406,"url":"https://github.com/elastic/kibana/pull/231406","mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/231406","number":231406,"mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}}]}] BACKPORT--> --------- Co-authored-by: Jesus Wahrman <41008968+jesuswr@users.noreply.github.com> Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> Co-authored-by: Jesus Wahrman <jesus.wahrman@elastic.co>
ymao1
pushed a commit
to ymao1/kibana
that referenced
this pull request
Sep 2, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
kibanamachine
added a commit
that referenced
this pull request
Sep 2, 2025
# Backport This will backport the following commits from `main` to `8.18`: - [Use optimistic concurrency in SO migrations (#231406)](#231406) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jesus Wahrman","email":"41008968+jesuswr@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-09-01T16:42:54Z","message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Feature:Saved Objects","release_note:skip","backport:prev-major","backport:current-major","v9.2.0"],"title":"Use optimistic concurrency in SO migrations","number":231406,"url":"https://github.com/elastic/kibana/pull/231406","mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/231406","number":231406,"mergeCommit":{"message":"Use optimistic concurrency in SO migrations (#231406)\n\n## Summary\n\nResolves https://github.com/elastic/kibana/issues/226518\n\nAdd `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic\nconcurrency logic. Also added some tests for this.\n\nWhile doing this, some tests broke, these tests were moving a SO from\nsingle namespace to multi namespace. We can't use the optimistic\nconcurrency logic in this case because we are using the `seq no` and\n`primary term` from the old SO, so when it tries to create a new SO it\nwill fail. That's why we are checking if `document._id !==\ndocument._source.originId;` before deciding if we'll use the optimistic\nconcurrency logic.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n---------\n\nCo-authored-by: Rudolf Meijering <skaapgif@gmail.com>","sha":"a5805f23728374dc6ec6ee241d7c38aca19762f6"}}]}] BACKPORT--> --------- Co-authored-by: Jesus Wahrman <41008968+jesuswr@users.noreply.github.com> Co-authored-by: Rudolf Meijering <skaapgif@gmail.com> Co-authored-by: Jesus Wahrman <jesus.wahrman@elastic.co>
MichelLosier
pushed a commit
to MichelLosier/kibana
that referenced
this pull request
Sep 2, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
kowalczyk-krzysztof
pushed a commit
to kowalczyk-krzysztof/kibana
that referenced
this pull request
Sep 3, 2025
## Summary Resolves elastic#226518 Add `seqNoPrimaryTerm: true,` in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this. While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the `seq no` and `primary term` from the old SO, so when it tries to create a new SO it will fail. That's why we are checking if `document._id !== document._source.originId;` before deciding if we'll use the optimistic concurrency logic. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #226518
Add
seqNoPrimaryTerm: true,in V2 and ZDT so it uses the optimistic concurrency logic. Also added some tests for this.While doing this, some tests broke, these tests were moving a SO from single namespace to multi namespace. We can't use the optimistic concurrency logic in this case because we are using the
seq noandprimary termfrom the old SO, so when it tries to create a new SO it will fail. That's why we are checking ifdocument._id !== document._source.originId;before deciding if we'll use the optimistic concurrency logic.Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:*label is applied per the guidelinesbackport:*labels.