Introduce install TTL markers

[pchila]

What does this PR do?

This PR introduces a registry of elastic-agent installs by maintaining a YAML structure in .installed installation marker.
As soon as a new version of agent is unpacked on disk during an upgrade, the installation is added to the registry, updated when the symlinks are rotated to such installation and removed after the installation is deleted from disk.
At @cmacknz 's request the install registry has been scrapped substituted by a more limited but simpler mechanism.

This PR introduces TTL marker files in versioned homes of Elastic Agent versions to keep track of available manual rollback targets and the time until those installs should remain on disk.
When upgrading, existing installs are marked with a .ttl YAML file, for example:

version: 9.3.0-SNAPSHOT
hash: abcdef
valid_until: 2025-10-17T14:01:13+02:00

This allows to determine which available rollbacks are still on disk even after the upgrade marker is removed at the end of a successful upgrade.

Why is it important?

This PR is a prerequisite for introducing manual rollbacks beyond grace period with PR #9643

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

• Relates Manual rollback after grace period #9643

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pchila]

Linter errors are already addressed in #9643

[pkoutsovasilis]

LGTM, I manually tested that:

• .installed keeps track of installed versions when I upgrade with correct TTL
• .installed keeps track of installed version when I manually rollback (removes old version)

It doesn't yet clean up entries in .installed - I did manually edit the ttl and restarted elastic-agent - but this I can see from the code that this is in #9643; @pchila keep me honest here

[pchila]

LGTM, I manually tested that:

* .installed keeps track of installed versions when I upgrade

* .installed keeps track of installed version when I manually rollback

It doesn't yet clean up entries in .installed - I did manually edit the ttl and restarted elastic-agent - but this I can see from the code that this is in #9643; @pchila keep me honest here

Correct. The handling for the expired installs is in the follow-up PR which allows for manual rollbacks while not in the grace period.
(Manually rolling back during the grace period does not need to rely on the install registry so the cleanup is still missing at this stage)

[ycombinator]

Starting to review this PR now but one general comment: could you please add godoc comments for any exported types, capturing what they represent / their purpose? Thanks.

[ycombinator]

@pchila Thanks for breaking out this PR from #9643 — I appreciate it as it makes each PR more focussed on a single logical change and easier to review (at least for me). Left some minor comments but this is looking good!

[pchila]

CI failures in build https://buildkite.com/elastic/elastic-agent/builds/28194 are due to TestSensitiveLogsESExporter integration test failing (untouched by this PR but recently introduced with #9341 )
Edit: fixed by a rebase on latest main

[cmacknz]

@pchila and I met today and came to the following conclusions:

• We should have a way to feature flag the rollback functionality, so that we can surface errors related to new code and fail the upgrade but also have a way to bypass unexpected bugs.
  • The simplest path to do this is to use the rollback window parameter we already support and will expose in the Fleet UI.
  • We can interpret having a rollback_window of 0 as a request to disable the rollback logic.
  • In concrete terms of the discussion of this PR, this would mean we treat updating the registry of available rollback versions as fatal errors unless rollback window is zero where we either ignore or simply don't update the registry at all.
• We should move the installation registry out of the .installed file.
  • The largest reason to do this is that it makes it much easier for us to change or relocate the list of registry versions by deleting or ignoring the new file.
    • For example if we had a local database available, we would prefer to track this there.
  • As part of this Paolo suggested we should only track versions available for rollback through the upgrade process, and remove the updating the registry at install time to further simplify things.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b introduce-install-registry upstream/introduce-install-registry
git merge upstream/main
git push upstream introduce-install-registry

[pchila]

@pchila and I met today and came to the following conclusions:

* We should have a way to feature flag the rollback functionality, so that we can surface errors related to new code and fail the upgrade but also have a way to bypass unexpected bugs.
  
  * The simplest path to do this is to use the rollback window parameter we already support and will expose in the Fleet UI.
  * We can interpret having a rollback_window of 0 as a request to disable the rollback logic.
  * In concrete terms of the discussion of this PR, this would mean we treat updating the registry of available rollback versions as fatal errors unless rollback window is zero where we either ignore or simply don't update the registry at all.

* We should move the installation registry out of the `.installed` file.
  
  * The largest reason to do this is that it makes it much easier for us to change or relocate the list of registry versions by deleting or ignoring the new file.
    
    * For example if we had a local database available, we would prefer to track this there.
  * As part of this Paolo suggested we should only track versions available for rollback through the upgrade process, and remove the updating the registry at install time to further simplify things.

@cmacknz see d4cc617

[cmacknz]

Thanks for simplifying, a couple of comments but direction overall LGTM

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9303e2b

History

• 💔 Build #29040 failed 2f565ff
• 💛 Build #28987 was flaky d8ff91f
• 💔 Build #28973 failed 11a5a70
• 💛 Build #28863 was flaky 377f80d
• 💛 Build #28841 was flaky d4cc617

cc @pchila