Allow the Agent to retrieve configuration from the Fleet API. by ph · Pull Request #15027 · elastic/beats

ph · 2019-12-10T13:41:32Z

Correctly Fetch a configuration from the Fleet API and send everything to the operator.
This PR doesn't include errors reportings.

Ref: #14951

depends on: #15089

How to test?

Use the dev-tools/cmd/fakewebapi
sh push.sh to push a configuration to the API.
mage build:dev
./agent enroll http://localhost:8080 abc123
Adjust the path in agent.yml see [Agent] Cleanup the YAML #15167 for followup.
BEATS_VERSION=7.5.0 agent run

elasticmachine · 2019-12-10T13:41:35Z

Pinging @elastic/ingest (Project:fleet)

This PR add a few things - It takes the keystore encryption implementation and create an io.Reader and io.Writer for it, we currently work with variable lenght blocks and we are not compatible for now with the one in libbeat. - It implements a few storage mechanism and wrapper: - DiskStore: Save a io.reader directly to disk, the content is saved in a temporary file and the target is replace in an atomic operation. - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader and io.Writer. - A fleet configuration is created in _meta/agent.fleet.yml, the content of this file is packed in the binary. When a user enroll the agent, we do a backup of the current agent.yml and the content is replaced with the agent.fleet.yml - The enrollment information is saved into a "fleet.yml" - The managed mode now read the content of the fleet.yml and creates a Kibana API client. - A Separates Fleet Config struct is created. - Enroll will now ask for confirmation before replacing the user configuration.

implement the required gateway and the action handling to allow to start, restart or stops processes based on the received configuration. Ref: #14951

Add two kind of scheduler one which is a `Stepper` schedule were the unblock is manually triggered by an out of bound call and a `Periodic` schedule which is based on the time.Ticker.

ph · 2019-12-17T20:52:28Z

@michalpristas you can do a review on this.

michalpristas

LGTM small comments here and there but i think once you resolve formatting issues we're close to getting this in.

michalpristas · 2019-12-18T10:39:08Z

+}
+
+// WaitTick returns a channel to watch for ticks.
+func (s *Stepper) WaitTick() <-chan time.Time {


i'm thinking about the name, as this does not actually wait

@michalpristas what if we use Tick() instead?

michalpristas · 2019-12-18T10:42:49Z

 	"net/http"
 	"net/url"

+	"time"


i think format will complain

looks fine, I've ran make fmt and nothing was raised.

michalpristas · 2019-12-18T10:43:49Z

+	log     *logger.Logger
+	Config  FleetAgentConfig
+	api     apiClient
+	agentID string


this will be conflicting with my PR but let's push this one first. my will be easier to solve conflicts in

michalpristas · 2019-12-18T10:44:38Z

+	Dispatch(...action) error
+}
+
+type fleetGateway struct {


can you add a comment with the purpose of this? so it's easier to find when i forget it in a future :D

michalpristas · 2019-12-18T10:47:21Z

+		return nil, err
+	}
+
+	actionDispatcher.MustRegister(


this is ok, i'm thinking also about having

&handlerPolicyChange{...}.Register(actionDispatcher)

so handler register itself and it know which action it registers for. let's keep it like you have it for now, this is really minor thing which can be addressed sometimes in a future or not at all

michalpristas · 2019-12-18T10:53:37Z

+func (f *fleetGateway) execute() (*fleetapi.CheckinResponse, error) {
+	cmd := fleetapi.NewCheckinCmd(f.agentID, f.client)
+
+	req := &fleetapi.CheckinRequest{}


can you add TODO to provide events?

michalpristas · 2019-12-18T11:38:54Z

+				continue
+			}
+
+			actions := make([]action, len(resp.Actions))


why do you need to repack this?

Because the actionDispatcher doesn't have hard dependencies on fleetapi.Action

ph · 2019-12-18T18:52:44Z

@michalpristas updated with all the changes.

ph · 2019-12-18T21:55:18Z

@michalpristas I have addressed all the above, merging this to unblock agent uuid.

…c#15027) * Feature: Allow to persist AccessToken and other connection data on disk This PR add a few things - It takes the keystore encryption implementation and create an io.Reader and io.Writer for it, we currently work with variable lenght blocks and we are not compatible for now with the one in libbeat. - It implements a few storage mechanism and wrapper: - DiskStore: Save a io.reader directly to disk, the content is saved in a temporary file and the target is replace in an atomic operation. - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader and io.Writer. - A fleet configuration is created in _meta/agent.fleet.yml, the content of this file is packed in the binary. When a user enroll the agent, we do a backup of the current agent.yml and the content is replaced with the agent.fleet.yml - The enrollment information is saved into a "fleet.yml" - The managed mode now read the content of the fleet.yml and creates a Kibana API client. - A Separates Fleet Config struct is created. - Enroll will now ask for confirmation before replacing the user configuration. * allow to use the obfuscated store. * Allow the Agent to retrieve configuration from the Fleet API. implement the required gateway and the action handling to allow to start, restart or stops processes based on the received configuration. Ref: elastic#14951

ph added in progress Pull request is currently in progress. [zube]: In Progress Project:fleet labels Dec 10, 2019

ph changed the title ~~Allow the Agent to retrieve configuration from the Fleet API.~~ [WIP] Allow the Agent to retrieve configuration from the Fleet API. Dec 10, 2019

ph self-assigned this Dec 11, 2019

houndci-bot reviewed Dec 16, 2019

View reviewed changes

Comment thread x-pack/agent/pkg/agent/application/action_dispatch.go Outdated

houndci-bot reviewed Dec 16, 2019

View reviewed changes

Comment thread x-pack/agent/pkg/agent/application/action_dispatch.go Outdated

ph added 22 commits December 17, 2019 10:54

allow to use the obfuscated store.

c515916

Allow the Agent to retrieve configuration from the Fleet API.

e195d3a

implement the required gateway and the action handling to allow to start, restart or stops processes based on the received configuration. Ref: #14951

Add Scheduler concepts as a package

1ef046c

Add two kind of scheduler one which is a `Stepper` schedule were the unblock is manually triggered by an out of bound call and a `Periodic` schedule which is based on the time.Ticker.

more forward

f648c95

small typo

b8e5ec1

move with more testing.

6736653

ok its compile now

35ee856

more tests cases

f705ff9

🤦

6b0c1c7

fix test

e8988b4

more changes in managed

1b91951

periodic check on

6b27f67

oops

fca90b4

missing headers

7a6f522

trying to debug a mem leak.. sweet.

a74da4f

add guard for me

e7842e0

move to MustRegister

5f4ae4b

fix format

c6d5dfd

cleaning up the logging

39cca94

more debug trace

590bd3b

cleanup some log statement

ac43a3d

ph added 7 commits December 17, 2019 10:54

Allow an initial tick for the scheduler

617d2c7

logging

b790fa6

fix tests

ca3fa07

update format with Nicolas example

73d263c

remove the httpprof

1b0cf38

Make it work (tm)

d8df93c

use right format.

bc8c2a3

ph requested a review from michalpristas December 17, 2019 20:50

ph added review [zube]: In Review and removed [zube]: In Progress in progress Pull request is currently in progress. labels Dec 17, 2019

ph changed the title ~~[WIP] Allow the Agent to retrieve configuration from the Fleet API.~~ Allow the Agent to retrieve configuration from the Fleet API. Dec 17, 2019

michalpristas approved these changes Dec 18, 2019

View reviewed changes

ph added 2 commits December 18, 2019 13:38

buf channel

f39bf3b

adding comments / todo

30a05c1

ph merged commit 06a7a0f into elastic:fleet Dec 18, 2019

Conversation

ph commented Dec 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Dec 10, 2019

Uh oh!

Uh oh!

Uh oh!

ph commented Dec 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michalpristas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ph commented Dec 18, 2019

Uh oh!

ph commented Dec 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ph commented Dec 10, 2019 •

edited

Loading

ph commented Dec 17, 2019 •

edited

Loading