[data views] data views + rollup index referenced by alias#212592
Merged
lukasolson merged 5 commits intoelastic:mainfrom Mar 11, 2025
Merged
[data views] data views + rollup index referenced by alias#212592lukasolson merged 5 commits intoelastic:mainfrom
lukasolson merged 5 commits intoelastic:mainfrom
Conversation
mattkime
changed the title
mattkime
added
Team:Kibana Management
Contributor
|
Pinging @elastic/kibana-management (Team:Kibana Management) |
ElenaStoeva
approved these changes
Feb 27, 2025
Contributor
ElenaStoeva
left a comment
ElenaStoeva
left a comment
There was a problem hiding this comment.
Tested locally - verified that aliases of rollup indices are identified as rollup in Data views and that rollup data views referencing these aliases can be created. Code changes also lgtm.
davismcphee
reviewed
Feb 28, 2025
Contributor
davismcphee
left a comment
davismcphee
left a comment
There was a problem hiding this comment.
I might be misunderstanding what should be happening here, but is it expected that I can't create a rollup data view using the rollup index directly, and that Discover blows up when creating one using the alias? Unfortunately I'm not really familiar enough with the previous behaviour to know what to expect here.
Index:
Alias:
| (matchedIndex) => | ||
| rollupIndices.includes(matchedIndex.name) || | ||
| // alias case | ||
| (matchedIndex.item.indices?.length === 1 && |
Contributor
There was a problem hiding this comment.
Bear with me because I don't know much about rollups, but why do we check that the alias only points to one index? I see the comment that a rollup pattern can only match one rollup index, but is it possible the alias could cover multiple indices and only one of them is a rollup index?
Contributor
Author
There was a problem hiding this comment.
Rollup data views work with one and only one rollup index. I think this is because the rollup meta data is saved on the data view.
Yes, I think its technically possible for an alias to point to more than one rollup index - but that would be due to an alias created by an ES user rather than through Upgrade Assistant. I don't think there's a graceful way of handling that case so I think only working with the first returned rollup is fine.
Contributor
Author
|
@davismcphee Looks like you might have found a couple of bugs. I'll try to reproduce them. |
lukasolson
approved these changes
Mar 10, 2025
Contributor
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Async chunks
Public APIs missing exports
History
cc @mattkime |
Contributor
|
Starting backport for target branches: 9.0 https://github.com/elastic/kibana/actions/runs/13778189826 |
kibanamachine
added a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 11, 2025
…12592) ## Summary Upgrading to 9.x involves reindexing indices created in 7.x, which does include rollup indices. Reindexing means relying on aliases to preserve existing index names. As it turns out, our existing code did not work with rollups that referenced aliases, rather than indices. This is because the index name is used as an object key even if it was retrieved via alias. Note - I need to verify this on 9.0 from scratch. I used upgraded data and need to verify the steps to make this work when testing. To test 1. Add sample data 2. Create a rollup job that references the sample data. 3. Create a data view that references the rollup index. It may take a few minutes for the rollup index to be populated. 4. Create an alias from the dev console, like such - ``` POST _aliases { "actions": [ { "add": { "index": "rollup", "alias": "my-alias" } } ] } ``` 5. Create a rollup data view based in the alias you just created. Part of elastic#211850 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 5b6dbf2)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
mattkime
added a commit
to mattkime/kibana
that referenced
this pull request
Mar 11, 2025
…12592) ## Summary Upgrading to 9.x involves reindexing indices created in 7.x, which does include rollup indices. Reindexing means relying on aliases to preserve existing index names. As it turns out, our existing code did not work with rollups that referenced aliases, rather than indices. This is because the index name is used as an object key even if it was retrieved via alias. Note - I need to verify this on 9.0 from scratch. I used upgraded data and need to verify the steps to make this work when testing. To test 1. Add sample data 2. Create a rollup job that references the sample data. 3. Create a data view that references the rollup index. It may take a few minutes for the rollup index to be populated. 4. Create an alias from the dev console, like such - ``` POST _aliases { "actions": [ { "add": { "index": "rollup", "alias": "my-alias" } } ] } ``` 5. Create a rollup data view based in the alias you just created. Part of elastic#211850 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 5b6dbf2)
mattkime
added a commit
to mattkime/kibana
that referenced
this pull request
Mar 11, 2025
…12592) ## Summary Upgrading to 9.x involves reindexing indices created in 7.x, which does include rollup indices. Reindexing means relying on aliases to preserve existing index names. As it turns out, our existing code did not work with rollups that referenced aliases, rather than indices. This is because the index name is used as an object key even if it was retrieved via alias. Note - I need to verify this on 9.0 from scratch. I used upgraded data and need to verify the steps to make this work when testing. To test 1. Add sample data 2. Create a rollup job that references the sample data. 3. Create a data view that references the rollup index. It may take a few minutes for the rollup index to be populated. 4. Create an alias from the dev console, like such - ``` POST _aliases { "actions": [ { "add": { "index": "rollup", "alias": "my-alias" } } ] } ``` 5. Create a rollup data view based in the alias you just created. Part of elastic#211850 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 5b6dbf2)
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
mattkime
added a commit
to mattkime/kibana
that referenced
this pull request
Mar 11, 2025
…12592) ## Summary Upgrading to 9.x involves reindexing indices created in 7.x, which does include rollup indices. Reindexing means relying on aliases to preserve existing index names. As it turns out, our existing code did not work with rollups that referenced aliases, rather than indices. This is because the index name is used as an object key even if it was retrieved via alias. Note - I need to verify this on 9.0 from scratch. I used upgraded data and need to verify the steps to make this work when testing. To test 1. Add sample data 2. Create a rollup job that references the sample data. 3. Create a data view that references the rollup index. It may take a few minutes for the rollup index to be populated. 4. Create an alias from the dev console, like such - ``` POST _aliases { "actions": [ { "add": { "index": "rollup", "alias": "my-alias" } } ] } ``` 5. Create a rollup data view based in the alias you just created. Part of elastic#211850 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 5b6dbf2)
kibanamachine
added a commit
that referenced
this pull request
Mar 11, 2025
…2592) (#213852) # Backport This will backport the following commits from `main` to `9.0`: - [[data views] data views + rollup index referenced by alias (#212592)](#212592) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"matt@mattki.me"},"sourceCommit":{"committedDate":"2025-03-11T01:16:04Z","message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:Kibana Management","Feature:Rollups","backport:prev-minor","v9.1.0"],"title":"[data views] data views + rollup index referenced by alias","number":212592,"url":"https://github.com/elastic/kibana/pull/212592","mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212592","number":212592,"mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}}]}] BACKPORT--> Co-authored-by: Matthew Kime <matt@mattki.me>
mattkime
added a commit
that referenced
this pull request
Mar 11, 2025
…2592) (#213856) # Backport This will backport the following commits from `main` to `8.x`: - [[data views] data views + rollup index referenced by alias (#212592)](#212592) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"matt@mattki.me"},"sourceCommit":{"committedDate":"2025-03-11T01:16:04Z","message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:Kibana Management","Feature:Rollups","backport:prev-minor","v9.1.0"],"title":"[data views] data views + rollup index referenced by alias","number":212592,"url":"https://github.com/elastic/kibana/pull/212592","mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212592","number":212592,"mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},{"url":"https://github.com/elastic/kibana/pull/213852","number":213852,"branch":"9.0","state":"OPEN"}]}] BACKPORT-->
mattkime
added a commit
that referenced
this pull request
Mar 11, 2025
…12592) (#213857) # Backport This will backport the following commits from `main` to `8.18`: - [[data views] data views + rollup index referenced by alias (#212592)](#212592) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"matt@mattki.me"},"sourceCommit":{"committedDate":"2025-03-11T01:16:04Z","message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:Kibana Management","Feature:Rollups","backport:prev-minor","v9.1.0"],"title":"[data views] data views + rollup index referenced by alias","number":212592,"url":"https://github.com/elastic/kibana/pull/212592","mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212592","number":212592,"mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},{"url":"https://github.com/elastic/kibana/pull/213852","number":213852,"branch":"9.0","state":"OPEN"}]}] BACKPORT-->
mattkime
added a commit
that referenced
this pull request
Mar 11, 2025
…12592) (#213858) # Backport This will backport the following commits from `main` to `8.17`: - [[data views] data views + rollup index referenced by alias (#212592)](#212592) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"matt@mattki.me"},"sourceCommit":{"committedDate":"2025-03-11T01:16:04Z","message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:Kibana Management","Feature:Rollups","backport:prev-minor","v9.1.0"],"title":"[data views] data views + rollup index referenced by alias","number":212592,"url":"https://github.com/elastic/kibana/pull/212592","mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212592","number":212592,"mergeCommit":{"message":"[data views] data views + rollup index referenced by alias (#212592)\n\n## Summary\n\nUpgrading to 9.x involves reindexing indices created in 7.x, which does\ninclude rollup indices. Reindexing means relying on aliases to preserve\nexisting index names. As it turns out, our existing code did not work\nwith rollups that referenced aliases, rather than indices. This is\nbecause the index name is used as an object key even if it was retrieved\nvia alias.\n\n\nNote - I need to verify this on 9.0 from scratch. I used upgraded data\nand need to verify the steps to make this work when testing.\n\nTo test\n1. Add sample data\n2. Create a rollup job that references the sample data. \n3. Create a data view that references the rollup index. It may take a\nfew minutes for the rollup index to be populated.\n4. Create an alias from the dev console, like such - \n\n```\nPOST _aliases\n{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"rollup\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}\n``` \n5. Create a rollup data view based in the alias you just created.\n\nPart of https://github.com/elastic/kibana/issues/211850\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"5b6dbf2b2711c94cb32a39e6487f95abd128433c"}},{"url":"https://github.com/elastic/kibana/pull/213852","number":213852,"branch":"9.0","state":"OPEN"}]}] BACKPORT-->
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Mar 22, 2025
…12592) ## Summary Upgrading to 9.x involves reindexing indices created in 7.x, which does include rollup indices. Reindexing means relying on aliases to preserve existing index names. As it turns out, our existing code did not work with rollups that referenced aliases, rather than indices. This is because the index name is used as an object key even if it was retrieved via alias. Note - I need to verify this on 9.0 from scratch. I used upgraded data and need to verify the steps to make this work when testing. To test 1. Add sample data 2. Create a rollup job that references the sample data. 3. Create a data view that references the rollup index. It may take a few minutes for the rollup index to be populated. 4. Create an alias from the dev console, like such - ``` POST _aliases { "actions": [ { "add": { "index": "rollup", "alias": "my-alias" } } ] } ``` 5. Create a rollup data view based in the alias you just created. Part of elastic#211850 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
mattkime
added a commit
that referenced
this pull request
Apr 14, 2025
This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped. To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index. Follow up to #212592 and #214656 Closes: #211850 --------- Co-authored-by: Ignacio Rivas <rivasign@gmail.com>
mattkime
added a commit
that referenced
this pull request
Apr 21, 2025
…to 9.1 (#218049) ## Summary forward port of #212815 --- This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped. To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index. Follow up to #212592 and #214656 Closes: #211850
pgayvallet
pushed a commit
to pgayvallet/kibana
that referenced
this pull request
Apr 22, 2025
…to 9.1 (elastic#218049) ## Summary forward port of elastic#212815 --- This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped. To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index. Follow up to elastic#212592 and elastic#214656 Closes: elastic#211850
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
May 29, 2025
…to 9.1 (elastic#218049) ## Summary forward port of elastic#212815 --- This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped. To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index. Follow up to elastic#212592 and elastic#214656 Closes: elastic#211850
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrading to 9.x involves reindexing indices created in 7.x, which does include rollup indices. Reindexing means relying on aliases to preserve existing index names. As it turns out, our existing code did not work with rollups that referenced aliases, rather than indices. This is because the index name is used as an object key even if it was retrieved via alias.
Note - I need to verify this on 9.0 from scratch. I used upgraded data and need to verify the steps to make this work when testing.
To test
Part of #211850