[Fleet] Save package policy previous revision on package upgrade#222779
Merged
jillguyonnet merged 27 commits intoelastic:mainfrom Jun 20, 2025
Merged
Conversation
jillguyonnet
added
release_note:skip
Member
Author
|
@juliaElastic Would you be able to give me early feedback on the approach? I would do something similar for the |
juliaElastic
reviewed
Jun 5, 2025
juliaElastic
reviewed
Jun 5, 2025
Member
Author
|
@elasticmachine merge upstream |
2 similar comments
Member
Author
|
@elasticmachine merge upstream |
Member
Author
|
@elasticmachine merge upstream |
Contributor
|
merge conflict between base and head |
jillguyonnet
force-pushed
the
fleet/keep-package-policy-previous-revisions
branch
from
e9aa99e to
ed9dbb5
Compare
Member
Author
|
@elasticmachine merge upstream |
Member
Author
|
@juliaElastic I'm still troubleshooting the remaining failing tests, which are not obvious to me so far 🤔 If you can check that the implementation satisfies the requirements of https://github.com/elastic/ingest-dev/issues/5444 that would be super helpful 🙏 |
Member
Author
|
@elasticmachine merge upstream |
… src/core/server/integration_tests/ci_checks'
… src/core/server/integration_tests/ci_checks'
…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --update'
… src/core/server/integration_tests/ci_checks'
jillguyonnet
force-pushed
the
fleet/keep-package-policy-previous-revisions
branch
from
dcb2e20 to
7cfea07
Compare
…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --update'
Member
Author
|
@elasticmachine merge upstream |
gsoldevila
reviewed
Jun 19, 2025
| "job.job_id", | ||
| "model_id" | ||
| ], | ||
| "monitoring-entity-source": [ |
Member
There was a problem hiding this comment.
You're removing all fields from this SO type? Not sure I understand where this change is coming from, I see this type was introduced 2 weeks ago with #219680.
Member
Author
There was a problem hiding this comment.
As discussed on Slack, #219680 renamed monitoring-entity-source to entity-analytics-monitoring-entity-source. I've deleted the entire definition of monitoring-entity-source accordingly.
Contributor
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Page load bundle
History
|
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
Jun 25, 2025
…stic#222779) ## Summary Closes elastic/ingest-dev#5444 * Add `enablePackageRollback` feature flag * Save package policy previous revision on package upgrade * Add `latest_revision` boolean property to `ingest-package-policies/fleet-package-policies` saved object type * Package policy SO are created with `latest_revision: true` * When a package policy is updated with a new package version, the previous SO is saved to ES with id `{id}:prev` and `latest_revision: false` * Backfill existing SO with `latest_revision: true` * GET logic filters for `latest_revision: true` * Save package previous version * Add `previous_version` property to `epm-packages` saved object type * When a package is upgraded to a new version, set `previous_version` ### Testing * Install an integration on an outdated version (edit the version in the URL and add the integration). * Check the package policy SO: it should have been created with `latest_revision: true`. * Check the package SO: the `previous_version` property should not be set. * Upgrade the integration and upgrade package policies. * Check the package policy SO: there should now be 2 SO for this package policy: * The updated one with `latest_revision: true` and policy id * The previous one with `latest_revision: false` and `{policy_id}:prev` * Check the package SO: the `previous_version` property should be set to the old version Note: it seems Fleet only allows upgrading packages to the latest version (please correct me if that's wrong); for testing two consecutive updates (e.g. check that only the most recent revision is saved), it might be necessary to run a custom EPR. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Risk of bad requests across Fleet wherever packages or package policies are queried. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
5 tasks
jillguyonnet
added a commit
that referenced
this pull request
Jul 18, 2025
## Summary Closes elastic/ingest-dev#5445 This PR adds a public API endpoint `POST kbn:/api/fleet/epm/packages/{packageName}/rollback` that rolls back an integration to the previous installed version, provided that previous version had been saved and that all integration policies (in all spaces) have corresponding saved previous revisions. This endpoint updates integration policies across all spaces. For example, given the following: * The `system` integration was installed on 2.2.0 with integration policies (also on 2.2.0) in multiple spaces. * The integration was upgraded to 2.3.2 and all integration policies were also upgraded. Then rolling back the integration will install 2.2.0 and update integration policies in all spaces to the latest revision they were on for 2.2.0. Rollback will fail if: * The integration has no `previous_version` property set (set on upgrade, see #222779). * At least one integration policy (in any space) doesn't have a saved previous revision (SO with id `{id}:prev`). * At least one integration policy (in any space) has a saved previous revision with a different previous version. ### Implementation details 1. Package policies rollback step 1 1. Create temporary SO copies in order to reverse the rollback in case of failure. 2. Update SO with data from saved previous revisions. 2. Package rollback 3. Package policies rollback step 2 * If package rollback succeeded, delete temporary SO copies and previous revision SO + bump agent policy revisions. * If package rollback failed, restore SO to pre-rollback (using temporary SO copies) and delete these copies. ### Testing 1. Create a custom space with default accesses. 5. In the default space install a package on an older version and make sure to have a package policy (edit the version in the browser URL and add the package). Choose a version old enough to have a few more recent versions to test with. 6. In the custom space, add the package on the same older version in order to create a package policy. You should now have a package policy on this older version on both spaces. 7. Go to Dev Tools and issue a rollback request: it should fail with `No previous version found for package {pkgName}`. 8. Force upgrade the package to a newer (not the latest) version with ``` POST kbn:/api/fleet/epm/packages/{pkgName}/{pkgVersion} { "force": true } ``` This doesn't upgrade the package policies. 9. Issue a rollback request: it should fail with `No previous version found for package policies: [policyId1, policyId2]`. 10. Force upgrade the package again to a newer version. 11. In the UI, upgrade both package policies. 12. Go to Dev Tools and issue a rollback request: it should fail with `Wrong previous version for package policies: [policyId1, policyId2]`. This is because the package was upgraded from v1 to v2 to v3, but policies were directly upgraded from v1 to v3. 13. Force upgrade the package again to a newer version. 14. In the UI, upgrade both package policies. 15. Take note of the revisions the agent policies are on. 16. Go to Dev Tools and issue a rollback request: this time, it should succeed and roll back the package and its policies in both spaces to the previous version. 17. Check in the UI that the rollback looks correct (agent policy revisions should have been bumped). For completeness, you can check the saved objects of the package and the package policies: * the package SO should have the correct (older) version and `previous_version: null` * the package policy SO storing the revision with the previous version (with id ending in `:prev`) should have been deleted * the current package policy SO should have the correct (older) version, `latest_revision: true` and revision number bumped by 1 * there should be no temporary SO (with id ending in `:copy`) ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Risk of incorrect integration policy data after rolling back the integration to its previous version. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Bluefinger
pushed a commit
to Bluefinger/kibana
that referenced
this pull request
Jul 22, 2025
## Summary Closes elastic/ingest-dev#5445 This PR adds a public API endpoint `POST kbn:/api/fleet/epm/packages/{packageName}/rollback` that rolls back an integration to the previous installed version, provided that previous version had been saved and that all integration policies (in all spaces) have corresponding saved previous revisions. This endpoint updates integration policies across all spaces. For example, given the following: * The `system` integration was installed on 2.2.0 with integration policies (also on 2.2.0) in multiple spaces. * The integration was upgraded to 2.3.2 and all integration policies were also upgraded. Then rolling back the integration will install 2.2.0 and update integration policies in all spaces to the latest revision they were on for 2.2.0. Rollback will fail if: * The integration has no `previous_version` property set (set on upgrade, see elastic#222779). * At least one integration policy (in any space) doesn't have a saved previous revision (SO with id `{id}:prev`). * At least one integration policy (in any space) has a saved previous revision with a different previous version. ### Implementation details 1. Package policies rollback step 1 1. Create temporary SO copies in order to reverse the rollback in case of failure. 2. Update SO with data from saved previous revisions. 2. Package rollback 3. Package policies rollback step 2 * If package rollback succeeded, delete temporary SO copies and previous revision SO + bump agent policy revisions. * If package rollback failed, restore SO to pre-rollback (using temporary SO copies) and delete these copies. ### Testing 1. Create a custom space with default accesses. 5. In the default space install a package on an older version and make sure to have a package policy (edit the version in the browser URL and add the package). Choose a version old enough to have a few more recent versions to test with. 6. In the custom space, add the package on the same older version in order to create a package policy. You should now have a package policy on this older version on both spaces. 7. Go to Dev Tools and issue a rollback request: it should fail with `No previous version found for package {pkgName}`. 8. Force upgrade the package to a newer (not the latest) version with ``` POST kbn:/api/fleet/epm/packages/{pkgName}/{pkgVersion} { "force": true } ``` This doesn't upgrade the package policies. 9. Issue a rollback request: it should fail with `No previous version found for package policies: [policyId1, policyId2]`. 10. Force upgrade the package again to a newer version. 11. In the UI, upgrade both package policies. 12. Go to Dev Tools and issue a rollback request: it should fail with `Wrong previous version for package policies: [policyId1, policyId2]`. This is because the package was upgraded from v1 to v2 to v3, but policies were directly upgraded from v1 to v3. 13. Force upgrade the package again to a newer version. 14. In the UI, upgrade both package policies. 15. Take note of the revisions the agent policies are on. 16. Go to Dev Tools and issue a rollback request: this time, it should succeed and roll back the package and its policies in both spaces to the previous version. 17. Check in the UI that the rollback looks correct (agent policy revisions should have been bumped). For completeness, you can check the saved objects of the package and the package policies: * the package SO should have the correct (older) version and `previous_version: null` * the package policy SO storing the revision with the previous version (with id ending in `:prev`) should have been deleted * the current package policy SO should have the correct (older) version, `latest_revision: true` and revision number bumped by 1 * there should be no temporary SO (with id ending in `:copy`) ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Risk of incorrect integration policy data after rolling back the integration to its previous version. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kertal
pushed a commit
to kertal/kibana
that referenced
this pull request
Jul 25, 2025
## Summary Closes elastic/ingest-dev#5445 This PR adds a public API endpoint `POST kbn:/api/fleet/epm/packages/{packageName}/rollback` that rolls back an integration to the previous installed version, provided that previous version had been saved and that all integration policies (in all spaces) have corresponding saved previous revisions. This endpoint updates integration policies across all spaces. For example, given the following: * The `system` integration was installed on 2.2.0 with integration policies (also on 2.2.0) in multiple spaces. * The integration was upgraded to 2.3.2 and all integration policies were also upgraded. Then rolling back the integration will install 2.2.0 and update integration policies in all spaces to the latest revision they were on for 2.2.0. Rollback will fail if: * The integration has no `previous_version` property set (set on upgrade, see elastic#222779). * At least one integration policy (in any space) doesn't have a saved previous revision (SO with id `{id}:prev`). * At least one integration policy (in any space) has a saved previous revision with a different previous version. ### Implementation details 1. Package policies rollback step 1 1. Create temporary SO copies in order to reverse the rollback in case of failure. 2. Update SO with data from saved previous revisions. 2. Package rollback 3. Package policies rollback step 2 * If package rollback succeeded, delete temporary SO copies and previous revision SO + bump agent policy revisions. * If package rollback failed, restore SO to pre-rollback (using temporary SO copies) and delete these copies. ### Testing 1. Create a custom space with default accesses. 5. In the default space install a package on an older version and make sure to have a package policy (edit the version in the browser URL and add the package). Choose a version old enough to have a few more recent versions to test with. 6. In the custom space, add the package on the same older version in order to create a package policy. You should now have a package policy on this older version on both spaces. 7. Go to Dev Tools and issue a rollback request: it should fail with `No previous version found for package {pkgName}`. 8. Force upgrade the package to a newer (not the latest) version with ``` POST kbn:/api/fleet/epm/packages/{pkgName}/{pkgVersion} { "force": true } ``` This doesn't upgrade the package policies. 9. Issue a rollback request: it should fail with `No previous version found for package policies: [policyId1, policyId2]`. 10. Force upgrade the package again to a newer version. 11. In the UI, upgrade both package policies. 12. Go to Dev Tools and issue a rollback request: it should fail with `Wrong previous version for package policies: [policyId1, policyId2]`. This is because the package was upgraded from v1 to v2 to v3, but policies were directly upgraded from v1 to v3. 13. Force upgrade the package again to a newer version. 14. In the UI, upgrade both package policies. 15. Take note of the revisions the agent policies are on. 16. Go to Dev Tools and issue a rollback request: this time, it should succeed and roll back the package and its policies in both spaces to the previous version. 17. Check in the UI that the rollback looks correct (agent policy revisions should have been bumped). For completeness, you can check the saved objects of the package and the package policies: * the package SO should have the correct (older) version and `previous_version: null` * the package policy SO storing the revision with the previous version (with id ending in `:prev`) should have been deleted * the current package policy SO should have the correct (older) version, `latest_revision: true` and revision number bumped by 1 * there should be no temporary SO (with id ending in `:copy`) ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Risk of incorrect integration policy data after rolling back the integration to its previous version. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kpollich
added
backport:version
Contributor
|
Starting backport for target branches: 9.1 https://github.com/elastic/kibana/actions/runs/20795913596 |
Contributor
|
Starting backport for target branches: 9.1 https://github.com/elastic/kibana/actions/runs/20795913578 |
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
1 similar comment
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kpollich
added
backport:skip
juliaElastic
added a commit
that referenced
this pull request
Apr 17, 2026
…se instead of latest_revision:true (#263717) ## Summary Fixes the `latest_revision` filter used when listing and counting package policies. The previous filter (`latest_revision:true`) excluded 8.x policies that never had `latest_revision` persisted to Elasticsearch, causing them to appear with a count of 0 in the Installed Integrations list after upgrading from 8.x to 9.x. The fix changes all filters from `latest_revision:true` to `NOT latest_revision:false`. This treats a missing field as equivalent to `true` (i.e. a current revision), which is the correct semantic — a policy without the field is a current revision, not a previous one. ### Root cause The `latest_revision` field was introduced in SO model version 19 (package rollback feature, PR #222779). For policies created in 8.x, the field is absent in ES because: - The v2 migration backfill only runs on one node in a multi-node cluster; other nodes skip document transforms - Patch upgrades (e.g. 9.2.7 → 9.2.8) skip the backfill entirely since no outdated documents exist - The list aggregation uses `perPage: 0`, bypassing the read-time in-memory backfill path entirely ### Changes - **`package_policies_aggregation.ts`** (Bug 1 + Bug 2): Change filter to `NOT :false`; add `size: SO_SEARCH_LIMIT` to terms aggregation to prevent ES default truncation at 10 buckets - **`services/epm/packages/get.ts`** (Bug 3): Add `NOT latest_revision:false` to `getPackageUsageStats` so `:prev` rollback docs are excluded from stats counts - **`services/package_policy.ts`**: Apply same filter fix to `list()`, `listIds()`, `getAllItemsIds()`, `getCompiledVersionsForAgentPolicy()`, and `findAllForAgentPolicy()` Closes #262491 Closes elastic/ingest-dev#7478 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Apr 17, 2026
…se instead of latest_revision:true (elastic#263717) ## Summary Fixes the `latest_revision` filter used when listing and counting package policies. The previous filter (`latest_revision:true`) excluded 8.x policies that never had `latest_revision` persisted to Elasticsearch, causing them to appear with a count of 0 in the Installed Integrations list after upgrading from 8.x to 9.x. The fix changes all filters from `latest_revision:true` to `NOT latest_revision:false`. This treats a missing field as equivalent to `true` (i.e. a current revision), which is the correct semantic — a policy without the field is a current revision, not a previous one. ### Root cause The `latest_revision` field was introduced in SO model version 19 (package rollback feature, PR elastic#222779). For policies created in 8.x, the field is absent in ES because: - The v2 migration backfill only runs on one node in a multi-node cluster; other nodes skip document transforms - Patch upgrades (e.g. 9.2.7 → 9.2.8) skip the backfill entirely since no outdated documents exist - The list aggregation uses `perPage: 0`, bypassing the read-time in-memory backfill path entirely ### Changes - **`package_policies_aggregation.ts`** (Bug 1 + Bug 2): Change filter to `NOT :false`; add `size: SO_SEARCH_LIMIT` to terms aggregation to prevent ES default truncation at 10 buckets - **`services/epm/packages/get.ts`** (Bug 3): Add `NOT latest_revision:false` to `getPackageUsageStats` so `:prev` rollback docs are excluded from stats counts - **`services/package_policy.ts`**: Apply same filter fix to `list()`, `listIds()`, `getAllItemsIds()`, `getCompiledVersionsForAgentPolicy()`, and `findAllForAgentPolicy()` Closes elastic#262491 Closes elastic/ingest-dev#7478 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit 222e630)
kibanamachine
added a commit
that referenced
this pull request
Apr 17, 2026
…on:false instead of latest_revision:true (#263717) (#264033) # Backport This will backport the following commits from `main` to `9.4`: - [[Fleet] Fix package policy count filters: use NOT latest_revision:false instead of latest_revision:true (#263717)](#263717) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2026-04-17T11:41:48Z","message":"[Fleet] Fix package policy count filters: use NOT latest_revision:false instead of latest_revision:true (#263717)\n\n## Summary\n\nFixes the `latest_revision` filter used when listing and counting\npackage policies. The previous filter (`latest_revision:true`) excluded\n8.x policies that never had `latest_revision` persisted to\nElasticsearch, causing them to appear with a count of 0 in the Installed\nIntegrations list after upgrading from 8.x to 9.x.\n\nThe fix changes all filters from `latest_revision:true` to `NOT\nlatest_revision:false`. This treats a missing field as equivalent to\n`true` (i.e. a current revision), which is the correct semantic — a\npolicy without the field is a current revision, not a previous one.\n\n### Root cause\n\nThe `latest_revision` field was introduced in SO model version 19\n(package rollback feature, PR #222779). For policies created in 8.x, the\nfield is absent in ES because:\n- The v2 migration backfill only runs on one node in a multi-node\ncluster; other nodes skip document transforms\n- Patch upgrades (e.g. 9.2.7 → 9.2.8) skip the backfill entirely since\nno outdated documents exist\n- The list aggregation uses `perPage: 0`, bypassing the read-time\nin-memory backfill path entirely\n\n### Changes\n\n- **`package_policies_aggregation.ts`** (Bug 1 + Bug 2): Change filter\nto `NOT :false`; add `size: SO_SEARCH_LIMIT` to terms aggregation to\nprevent ES default truncation at 10 buckets\n- **`services/epm/packages/get.ts`** (Bug 3): Add `NOT\nlatest_revision:false` to `getPackageUsageStats` so `:prev` rollback\ndocs are excluded from stats counts\n- **`services/package_policy.ts`**: Apply same filter fix to `list()`,\n`listIds()`, `getAllItemsIds()`, `getCompiledVersionsForAgentPolicy()`,\nand `findAllForAgentPolicy()`\n\nCloses #262491\nCloses https://github.com/elastic/ingest-dev/issues/7478\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---------\n\nCo-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>","sha":"222e63005e6b113dd2ffffd7e75893a7be70e4be","branchLabelMapping":{"^v9.5.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:version","v9.4.0","v9.5.0","v9.3.4","v9.2.9"],"title":"[Fleet] Fix package policy count filters: use NOT latest_revision:false instead of latest_revision:true","number":263717,"url":"https://github.com/elastic/kibana/pull/263717","mergeCommit":{"message":"[Fleet] Fix package policy count filters: use NOT latest_revision:false instead of latest_revision:true (#263717)\n\n## Summary\n\nFixes the `latest_revision` filter used when listing and counting\npackage policies. The previous filter (`latest_revision:true`) excluded\n8.x policies that never had `latest_revision` persisted to\nElasticsearch, causing them to appear with a count of 0 in the Installed\nIntegrations list after upgrading from 8.x to 9.x.\n\nThe fix changes all filters from `latest_revision:true` to `NOT\nlatest_revision:false`. This treats a missing field as equivalent to\n`true` (i.e. a current revision), which is the correct semantic — a\npolicy without the field is a current revision, not a previous one.\n\n### Root cause\n\nThe `latest_revision` field was introduced in SO model version 19\n(package rollback feature, PR #222779). For policies created in 8.x, the\nfield is absent in ES because:\n- The v2 migration backfill only runs on one node in a multi-node\ncluster; other nodes skip document transforms\n- Patch upgrades (e.g. 9.2.7 → 9.2.8) skip the backfill entirely since\nno outdated documents exist\n- The list aggregation uses `perPage: 0`, bypassing the read-time\nin-memory backfill path entirely\n\n### Changes\n\n- **`package_policies_aggregation.ts`** (Bug 1 + Bug 2): Change filter\nto `NOT :false`; add `size: SO_SEARCH_LIMIT` to terms aggregation to\nprevent ES default truncation at 10 buckets\n- **`services/epm/packages/get.ts`** (Bug 3): Add `NOT\nlatest_revision:false` to `getPackageUsageStats` so `:prev` rollback\ndocs are excluded from stats counts\n- **`services/package_policy.ts`**: Apply same filter fix to `list()`,\n`listIds()`, `getAllItemsIds()`, `getCompiledVersionsForAgentPolicy()`,\nand `findAllForAgentPolicy()`\n\nCloses #262491\nCloses https://github.com/elastic/ingest-dev/issues/7478\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---------\n\nCo-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>","sha":"222e63005e6b113dd2ffffd7e75893a7be70e4be"}},"sourceBranch":"main","suggestedTargetBranches":["9.4","9.3","9.2"],"targetPullRequestStates":[{"branch":"9.4","label":"v9.4.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.5.0","branchLabelMappingKey":"^v9.5.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/263717","number":263717,"mergeCommit":{"message":"[Fleet] Fix package policy count filters: use NOT latest_revision:false instead of latest_revision:true (#263717)\n\n## Summary\n\nFixes the `latest_revision` filter used when listing and counting\npackage policies. The previous filter (`latest_revision:true`) excluded\n8.x policies that never had `latest_revision` persisted to\nElasticsearch, causing them to appear with a count of 0 in the Installed\nIntegrations list after upgrading from 8.x to 9.x.\n\nThe fix changes all filters from `latest_revision:true` to `NOT\nlatest_revision:false`. This treats a missing field as equivalent to\n`true` (i.e. a current revision), which is the correct semantic — a\npolicy without the field is a current revision, not a previous one.\n\n### Root cause\n\nThe `latest_revision` field was introduced in SO model version 19\n(package rollback feature, PR #222779). For policies created in 8.x, the\nfield is absent in ES because:\n- The v2 migration backfill only runs on one node in a multi-node\ncluster; other nodes skip document transforms\n- Patch upgrades (e.g. 9.2.7 → 9.2.8) skip the backfill entirely since\nno outdated documents exist\n- The list aggregation uses `perPage: 0`, bypassing the read-time\nin-memory backfill path entirely\n\n### Changes\n\n- **`package_policies_aggregation.ts`** (Bug 1 + Bug 2): Change filter\nto `NOT :false`; add `size: SO_SEARCH_LIMIT` to terms aggregation to\nprevent ES default truncation at 10 buckets\n- **`services/epm/packages/get.ts`** (Bug 3): Add `NOT\nlatest_revision:false` to `getPackageUsageStats` so `:prev` rollback\ndocs are excluded from stats counts\n- **`services/package_policy.ts`**: Apply same filter fix to `list()`,\n`listIds()`, `getAllItemsIds()`, `getCompiledVersionsForAgentPolicy()`,\nand `findAllForAgentPolicy()`\n\nCloses #262491\nCloses https://github.com/elastic/ingest-dev/issues/7478\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)\n\n---------\n\nCo-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>","sha":"222e63005e6b113dd2ffffd7e75893a7be70e4be"}},{"branch":"9.3","label":"v9.3.4","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes https://github.com/elastic/ingest-dev/issues/5444
enablePackageRollbackfeature flaglatest_revisionboolean property toingest-package-policies/fleet-package-policiessaved object typelatest_revision: true{id}:prevandlatest_revision: falselatest_revision: truelatest_revision: trueprevious_versionproperty toepm-packagessaved object typeprevious_versionTesting
latest_revision: true.previous_versionproperty should not be set.latest_revision: trueand policy idlatest_revision: falseand{policy_id}:prevprevious_versionproperty should be set to the old versionNote: it seems Fleet only allows upgrading packages to the latest version (please correct me if that's wrong); for testing two consecutive updates (e.g. check that only the most recent revision is saved), it might be necessary to run a custom EPR.
Checklist
release_note:*label is applied per the guidelinesIdentify risks
Risk of bad requests across Fleet wherever packages or package policies are queried.