[Ingest Manager] Upgrade Agents in Fleet #78810

neptunian · 2020-09-29T15:59:45Z

Resolves #78469

Summary

Adds the ability to upgrade agents in the Fleet UI

Changes

Added POST /api/ingest_manager/fleet/agents/bulk_upgrade endpoint which expects payload:

{
  agents: [array of agent IDs | kuery string],
  version: string - version to update to,
  source_uri: string - optional download url 
}

Agent should not upgrade agents that are currently unenrolled or unenrolling.

Update GET api/ingest_manager/fleet/agents endpoint to accept showUpgradeable option. This will filter for agents that are less than this kibana version with local_metadata.elastic.agent.upgradeable: true
Agent Details page now reports release based on local_metadata.elastic.agents.snapshot being true/false. For older Agent versions that do not send this data, the value will be '-'. The snapshot version cannot be tested until we have a snapshot with this functionality.

Agent Details screenshot

UI for upgrading a single agent and bulk upgrading agents based on design spec . This design spec is based on new design and functionality for Fleet Status Improvements which is not in 7.10, so these changes do not incorporate that.

Agent Details for upgradeable agent

Agent Detail for non upgradeable agent

Agent List non upgradeable action menu

Agent List upgradeable action menu

upgradeable action menu

Agent List bulk upgrade

Agent List filter upgradeable

This will filter for agents that are less than this kibana version with `local_metadata.elastic.agent.upgradeable: true`

Agent List when upgrading

After a few seconds, Agent should turn green and version should be updated. Upgrade Available indication should be gone.

Testing

To test the upgrade flow through Kibana UI (agent gets the download file and no source_uri is provided to the api), follow these steps:

Setup a local server that fetches the download
- cd into beats/x-pack/elastic-agent
- build a version of the agent which defaults to 8.0.0. run DEV=true PLATFORMS=darwin mage package
- mkdir -p ~/server/beats/elastic-agent and copy the built files there cp build/distributions/*tar* ~/server/beats/elastic-agent
- cd ~/server/beats/elastic-agent and start the server python3 -m http.server
Build a new version to run
- cd back into beats/x-pack/elastic-agent
- make a small commit such as adding a comment (so commit inside a package changes and it is a valid upgrade)
- modify version to make it report as 7.9.0. vi ../../libbeat/version/version.go and change version to 7.9.0
- run DEV=true PLATFORMS=darwin mage package again
- cd into builds/distribution/elastic-agent and unpack the 7.9 *.tar.gz
- cd into 7.9 directory and enroll an agent but do not run it (make sure this PR is running in kibana)
- open elastic-agent.yml and uncomment and update sourceUri to where the server is running and save
- enroll the agent. ignore the CONFIG_CHANGE action errors in the log
- Find your agent in Fleet and use the menu to upgrade it

To simulate testing with the `source_uri`, see testing steps in this PR:

elastic/beats#21002

michalpristas · 2020-09-30T07:53:20Z

works ok with elastic/beats#21372
if source uri is not passed we use official store, otherwise we use what's coming.

what in case of error should agent ack the action? if it's not acked will it be resend indefinitely?

...c/applications/ingest_manager/sections/fleet/agent_details_page/components/agent_details.tsx

michalpristas · 2020-09-30T11:12:02Z

correctly detects snapshot, visualisation of non snapshot commented in the code

jfsiii · 2020-09-30T12:16:07Z

x-pack/plugins/ingest_manager/public/plugin.ts

What do you think about changing this to env? It seems similar to what we have in config.

useKibanaVersion could be updated to read the EnvContext and other hooks/functions could get branch, or dev vs prod, etc if/when it comes up.

Sounds good to me. I didn't put it in IngestManagerConfigType just because those are from the config file, so I think like you said something like "env" sounds good. I opened a follow up issue #79483

x-pack/plugins/ingest_manager/common/services/is_agent_upgradeable.ts

michalpristas · 2020-10-02T06:55:20Z

x-pack/plugins/ingest_manager/server/types/models/agent.ts needs to be updated as well, for model validation
also PostAgentCheckinRequestBodyJSONSchema in x-pack/plugins/ingest_manager/server/types/rest_spec/agent.ts

michalpristas · 2020-10-02T07:59:23Z

After suggested changes (i did them manually) i started agent without starting a file server so UPDATE fails i got this (as expected)

UPDATE is ACK-ed and Error there is reported, under ACK there is line saying STATE was switched to UPDATING

after i started server and rekicked UPDATE action

update was ACKed and state switched to RUNNING after it finished.

Status on list page was ONLINE then UPDATING and then back ONLINE. Even though there were errors, so i guess this will be an object of next iteration.

… showUpgradeable

elasticmachine · 2020-10-02T18:13:25Z

Pinging @elastic/ingest-management (Team:Ingest Management)

neptunian · 2020-10-02T18:18:05Z

x-pack/plugins/ingest_manager/server/services/agents/crud.ts

  });
+  // filtering for a range on the version string will not work,
+  // nor does filtering on a flattened field (local_metadata), so filter here
+  if (showUpgradeable) {


this is not ideal, but i decided to filter on the results of the soClient.find(). As we discussed earlier I was unable to filter through the flattened field local_metadata and even if I could I still need to use semver to compare local_metadata.elastic.agent.version which is a string so I could not do a range comparison in KQL. @jen-huang @nchaulet

nchaulet

LGTM 🚀 tested with API calls (not a real agent but work as expected)

michalpristas · 2020-10-05T07:33:50Z

Tested ok today with elastic/beats#21461

neptunian · 2020-10-05T13:45:37Z

@elasticmachine merge upstream

* add kibanaVersion context and hook, add upgrade available indications * add agent upgrade modals and action buttons * fix import * add bulk actions api and remove source_uri as required * add upgrading to AgentHealth status * buildKueryForUpgradingAgents * bulk actions UI * remove source_uri * add release type to agent details * don't allow upgrade of unenrolled/unenrolling agent * hide upgradeable button when not upgradeable * fix test * add udpating agent status * remove upgrade available filter button for now * update isUpgradeAvailable to use local_metadata upgradeable * add UPDATING to agent event subtype * use saved object for updating agent status * add updating badge type label * add upgrade available button and update agent list endpoint to accept showUpgradeable * add schema and type for UPDATING * fix type * dont try to upgrade local_metadata * exclude from AAD upgrade_started_at and upgraded_at Co-authored-by: Kibana Machine <[email protected]>

* add kibanaVersion context and hook, add upgrade available indications * add agent upgrade modals and action buttons * fix import * add bulk actions api and remove source_uri as required * add upgrading to AgentHealth status * buildKueryForUpgradingAgents * bulk actions UI * remove source_uri * add release type to agent details * don't allow upgrade of unenrolled/unenrolling agent * hide upgradeable button when not upgradeable * fix test * add udpating agent status * remove upgrade available filter button for now * update isUpgradeAvailable to use local_metadata upgradeable * add UPDATING to agent event subtype * use saved object for updating agent status * add updating badge type label * add upgrade available button and update agent list endpoint to accept showUpgradeable * add schema and type for UPDATING * fix type * dont try to upgrade local_metadata * exclude from AAD upgrade_started_at and upgraded_at Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2020-10-06T17:03:07Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ad4a186

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
`alerts`	-	26	+26
`apm`	-	1276	+1276
`beatsManagement`	-	302	+302
`canvas`	-	1132	+1132
`cloud`	-	6	+6
`crossClusterReplication`	-	159	+159
`dashboardEnhanced`	-	26	+26
`dashboardMode`	-	4	+4
`dataEnhanced`	-	17	+17
`discoverEnhanced`	-	11	+11
`embeddableEnhanced`	-	11	+11
`enterpriseSearch`	-	250	+250
`features`	-	11	+11
`fileUpload`	-	21	+21
`globalSearch`	-	23	+23
`globalSearchBar`	-	13	+13
`globalSearchProviders`	-	8	+8
`graph`	-	169	+169
`grokdebugger`	-	75	+75
`indexLifecycleManagement`	-	206	+206
`indexManagement`	-	509	+509
`infra`	-	1147	+1147
`ingestManager`	-	548	+548
`ingestPipelines`	-	493	+493
`lens`	-	563	+563
`licenseManagement`	-	145	+145
`licensing`	-	12	+12
`lists`	-	128	+128
`logstash`	-	190	+190
`maps`	-	618	+618
`mapsLegacyLicensing`	-	4	+4
`ml`	-	1222	+1222
`monitoring`	-	635	+635
`observability`	-	96	+96
`painlessLab`	-	40	+40
`remoteClusters`	-	141	+141
`reporting`	-	45	+45
`rollup`	-	179	+179
`searchprofiler`	-	87	+87
`security`	-	458	+458
`securitySolution`	-	1998	+1998
`snapshotRestore`	-	168	+168
`spaces`	-	237	+237
`transform`	-	335	+335
`triggers_actions_ui`	-	274	+274
`uiActionsEnhanced`	-	134	+134
`upgradeAssistant`	-	117	+117
`uptime`	-	655	+655
`urlDrilldown`	-	8	+8
`watcher`	-	237	+237
total			+15169

async chunks size

id	before	after	diff
`apm`	-	4.2MB	⚠️ +4.2MB
`beatsManagement`	-	683.5KB	⚠️ +683.5KB
`canvas`	-	1.5MB	⚠️ +1.5MB
`crossClusterReplication`	-	456.5KB	⚠️ +456.5KB
`enterpriseSearch`	-	428.6KB	⚠️ +428.6KB
`fileUpload`	-	574.9KB	⚠️ +574.9KB
`graph`	-	1.3MB	⚠️ +1.3MB
`grokdebugger`	-	568.8KB	⚠️ +568.8KB
`indexLifecycleManagement`	-	231.7KB	⚠️ +231.7KB
`indexManagement`	-	1.6MB	⚠️ +1.6MB
`infra`	-	3.8MB	⚠️ +3.8MB
`ingestManager`	-	1.1MB	⚠️ +1.1MB
`ingestPipelines`	-	812.9KB	⚠️ +812.9KB
`lens`	-	1.0MB	⚠️ +1.0MB
`licenseManagement`	-	167.5KB	+167.5KB
`logstash`	-	213.4KB	⚠️ +213.4KB
`maps`	-	3.2MB	⚠️ +3.2MB
`ml`	-	10.6MB	⚠️ +10.6MB
`monitoring`	-	1.2MB	⚠️ +1.2MB
`observability`	-	163.2KB	+163.2KB
`painlessLab`	-	38.7KB	+38.7KB
`remoteClusters`	-	227.8KB	⚠️ +227.8KB
`rollup`	-	323.3KB	⚠️ +323.3KB
`searchprofiler`	-	705.1KB	⚠️ +705.1KB
`security`	-	1.0MB	⚠️ +1.0MB
`securitySolution`	-	10.3MB	⚠️ +10.3MB
`snapshotRestore`	-	622.6KB	⚠️ +622.6KB
`spaces`	-	45.5KB	+45.5KB
`transform`	-	1.2MB	⚠️ +1.2MB
`triggers_actions_ui`	-	1.5MB	⚠️ +1.5MB
`upgradeAssistant`	-	179.8KB	+179.8KB
`uptime`	-	1.7MB	⚠️ +1.7MB
`watcher`	-	1.0MB	⚠️ +1.0MB
total			⚠️ +52.5MB

distributable file count

id	before	after	diff
`default`	-	47088	+47088

miscellaneous assets size

id	before	after	diff
`apm`	-	49.0KB	+49.0KB
`enterpriseSearch`	-	834.9KB	⚠️ +834.9KB
`maps`	-	978.6KB	⚠️ +978.6KB
`securitySolution`	-	326.0KB	⚠️ +326.0KB
`triggers_actions_ui`	-	27.7KB	+27.7KB
total			⚠️ +2.2MB

page load bundle size

id	before	after	diff
`alerts`	-	89.8KB	+89.8KB
`apm`	-	44.3KB	+44.3KB
`beatsManagement`	-	169.1KB	+169.1KB
`canvas`	-	1.0MB	⚠️ +1.0MB
`cloud`	-	5.9KB	+5.9KB
`crossClusterReplication`	-	49.2KB	+49.2KB
`dashboardEnhanced`	-	43.1KB	+43.1KB
`dashboardMode`	-	7.5KB	+7.5KB
`dataEnhanced`	-	34.3KB	+34.3KB
`discoverEnhanced`	-	29.2KB	+29.2KB
`embeddableEnhanced`	-	25.5KB	+25.5KB
`enterpriseSearch`	-	20.0KB	+20.0KB
`features`	-	15.8KB	+15.8KB
`fileUpload`	-	9.5KB	+9.5KB
`globalSearch`	-	27.9KB	+27.9KB
`globalSearchBar`	-	29.0KB	+29.0KB
`globalSearchProviders`	-	10.3KB	+10.3KB
`graph`	-	15.8KB	+15.8KB
`grokdebugger`	-	11.5KB	+11.5KB
`indexLifecycleManagement`	-	89.9KB	+89.9KB
`indexManagement`	-	122.8KB	+122.8KB
`infra`	-	178.6KB	+178.6KB
`ingestManager`	-	540.9KB	⚠️ +540.9KB
`ingestPipelines`	-	42.0KB	+42.0KB
`lens`	-	76.0KB	+76.0KB
`licenseManagement`	-	26.2KB	+26.2KB
`licensing`	-	23.1KB	+23.1KB
`lists`	-	164.7KB	+164.7KB
`logstash`	-	37.6KB	+37.6KB
`maps`	-	164.4KB	+164.4KB
`mapsLegacyLicensing`	-	5.1KB	+5.1KB
`ml`	-	58.5KB	+58.5KB
`monitoring`	-	222.7KB	⚠️ +222.7KB
`observability`	-	52.4KB	+52.4KB
`painlessLab`	-	160.9KB	+160.9KB
`remoteClusters`	-	35.5KB	+35.5KB
`reporting`	-	164.5KB	+164.5KB
`rollup`	-	80.3KB	+80.3KB
`searchprofiler`	-	50.9KB	+50.9KB
`security`	-	163.3KB	+163.3KB
`securitySolution`	-	587.1KB	⚠️ +587.1KB
`snapshotRestore`	-	62.5KB	+62.5KB
`spaces`	-	364.2KB	⚠️ +364.2KB
`transform`	-	25.4KB	+25.4KB
`triggers_actions_ui`	-	148.0KB	+148.0KB
`uiActionsEnhanced`	-	320.7KB	⚠️ +320.7KB
`upgradeAssistant`	-	64.7KB	+64.7KB
`uptime`	-	25.2KB	+25.2KB
`urlDrilldown`	-	18.7KB	+18.7KB
`watcher`	-	27.9KB	+27.9KB
total			⚠️ +5.6MB

Saved Objects .kibana field count

id	before	after	diff
`_data_stream_timestamp`	-	1	+1
`action`	-	5	+5
`action_task_params`	-	3	+3
`alert`	-	30	+30
`apm-indices`	-	8	+8
`application_usage_daily`	-	2	+2
`canvas-element`	-	8	+8
`canvas-workpad`	-	5	+5
`canvas-workpad-template`	-	8	+8
`cases`	-	32	+32
`cases-comments`	-	17	+17
`cases-configure`	-	14	+14
`cases-user-actions`	-	10	+10
`config`	-	2	+2
`dashboard`	-	17	+17
`endpoint:user-artifact`	-	10	+10
`endpoint:user-artifact-manifest`	-	5	+5
`epm-packages`	-	14	+14
`exception-list`	-	39	+39
`exception-list-agnostic`	-	39	+39
`file-upload-telemetry`	-	2	+2
`fleet-agent-actions`	-	9	+9
`fleet-agent-events`	-	11	+11
`fleet-agents`	-	25	+25
`fleet-enrollment-api-keys`	-	10	+10
`graph-workspace`	-	9	+9
`index-pattern`	-	3	+3
`ingest_manager_settings`	-	6	+6
`ingest-agent-policies`	-	11	+11
`ingest-outputs`	-	10	+10
`ingest-package-policies`	-	35	+35
`kql-telemetry`	-	3	+3
`lens`	-	7	+7
`lens-ui-telemetry`	-	5	+5
`map`	-	7	+7
`ml-telemetry`	-	3	+3
`monitoring-telemetry`	-	2	+2
`namespace`	-	1	+1
`namespaces`	-	1	+1
`originId`	-	1	+1
`query`	-	6	+6
`references`	-	4	+4
`sample-data-telemetry`	-	3	+3
`search`	-	9	+9
`siem-detection-engine-rule-actions`	-	8	+8
`siem-detection-engine-rule-status`	-	12	+12
`siem-ui-timeline`	-	90	+90
`siem-ui-timeline-note`	-	8	+8
`siem-ui-timeline-pinned-event`	-	7	+7
`space`	-	9	+9
`telemetry`	-	9	+9
`timelion-sheet`	-	13	+13
`tsvb-validation-telemetry`	-	2	+2
`type`	-	1	+1
`ui-metric`	-	2	+2
`updated_at`	-	1	+1
`upgrade-assistant-reindex-operation`	-	18	+18
`upgrade-assistant-telemetry`	-	13	+13
`url`	-	6	+6
`visualization`	-	9	+9
total			+660

History

💚 Build #79613 succeeded ad4a186
💚 Build #79295 succeeded 3f5be5b
💚 Build #79264 succeeded 8ce5091
💚 Build #79126 succeeded 8839cccaac26cdd4b32bb0bea7ecfdaf319f3026
💚 Build #79004 succeeded 011a0bc54e0f704c0c6b91b867b36de522cbd55b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

neptunian added release_note:enhancement v8.0.0 v7.10.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 29, 2020

neptunian self-assigned this Sep 29, 2020

neptunian force-pushed the 68310-upgrade-agents branch from d975ffe to 156e02f Compare September 29, 2020 20:41

michalpristas reviewed Sep 30, 2020

View reviewed changes

...c/applications/ingest_manager/sections/fleet/agent_details_page/components/agent_details.tsx Outdated Show resolved Hide resolved

jfsiii reviewed Sep 30, 2020

View reviewed changes

This was referenced Oct 2, 2020

[Ingest Manager] Send updating state elastic/beats#21461

Merged

[Elastic Agent] Agent Upgrade feature test elastic/e2e-testing#341

Closed

neptunian added 16 commits October 2, 2020 12:22

add kibanaVersion context and hook, add upgrade available indications

623ed8b

add agent upgrade modals and action buttons

d77784b

fix import

d0c1c9e

add bulk actions api and remove source_uri as required

951dc39

add upgrading to AgentHealth status

072f456

buildKueryForUpgradingAgents

301edb7

bulk actions UI

d831ec5

remove source_uri

09bb711

add release type to agent details

17520b1

don't allow upgrade of unenrolled/unenrolling agent

4864503

hide upgradeable button when not upgradeable

69fd965

fix test

107c427

add udpating agent status

203f195

remove upgrade available filter button for now

782e90f

update isUpgradeAvailable to use local_metadata upgradeable

44f330f

add UPDATING to agent event subtype

88bcc58

neptunian added 5 commits October 2, 2020 12:22

use saved object for updating agent status

821b767

add updating badge type label

5b08a92

add upgrade available button and update agent list endpoint to accept…

18c762f

… showUpgradeable

add schema and type for UPDATING

a24f164

fix type

76b2ba6

neptunian force-pushed the 68310-upgrade-agents branch from 8839ccc to 76b2ba6 Compare October 2, 2020 17:35

dont try to upgrade local_metadata

8ce5091

neptunian marked this pull request as ready for review October 2, 2020 18:13

neptunian requested a review from a team October 2, 2020 18:13

neptunian commented Oct 2, 2020

View reviewed changes

exclude from AAD upgrade_started_at and upgraded_at

3f5be5b

nchaulet approved these changes Oct 5, 2020

View reviewed changes

Merge branch 'master' into 68310-upgrade-agents

ad4a186

neptunian mentioned this pull request Oct 5, 2020

[Ingest Manager] Create an env context for Fleet app #79483

Open

neptunian merged commit 53f22dc into elastic:master Oct 5, 2020

neptunian mentioned this pull request Oct 5, 2020

[7.x] [Ingest Manager] Upgrade Agents in Fleet (#78810) #79488

Merged

michalpristas mentioned this pull request Oct 5, 2020

Cherry-pick #21461 to 7.x: Send updating state elastic/beats#21530

Merged

6 tasks

neptunian mentioned this pull request Oct 5, 2020

[Ingest Manager] Update open API spec and add condition to agent upgrade endpoint #79579

Merged

neptunian deleted the 68310-upgrade-agents branch October 6, 2020 14:22

ph mentioned this pull request Oct 6, 2020

Upgrade agent option is not available in Actions menu under Agents tab #79644

Closed

[Ingest Manager] Upgrade Agents in Fleet #78810

[Ingest Manager] Upgrade Agents in Fleet #78810

Uh oh!

Conversation

neptunian commented Sep 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

To test the upgrade flow through Kibana UI (agent gets the download file and no source_uri is provided to the api), follow these steps:

To simulate testing with the source_uri, see testing steps in this PR:

Uh oh!

michalpristas commented Sep 30, 2020

Uh oh!

Uh oh!

michalpristas commented Sep 30, 2020

Uh oh!

jfsiii Sep 30, 2020

Choose a reason for hiding this comment

Uh oh!

neptunian Oct 5, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

michalpristas commented Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michalpristas commented Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Oct 2, 2020

Uh oh!

neptunian Oct 2, 2020

Choose a reason for hiding this comment

Uh oh!

nchaulet left a comment

Choose a reason for hiding this comment

Uh oh!

michalpristas commented Oct 5, 2020

Uh oh!

neptunian commented Oct 5, 2020

Uh oh!

kibanamachine commented Oct 6, 2020

💚 Build Succeeded

Metrics [docs]

@kbn/optimizer bundle module count

async chunks size

distributable file count

miscellaneous assets size

page load bundle size

Saved Objects .kibana field count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

neptunian commented Sep 29, 2020 •

edited

Loading

To simulate testing with the `source_uri`, see testing steps in this PR:

michalpristas commented Oct 2, 2020 •

edited

Loading

michalpristas commented Oct 2, 2020 •

edited

Loading