Merge agent to master by ph · Pull Request #16494 · elastic/beats

ph · 2020-02-21T19:24:45Z

What does this PR do?

This PR move the development of the Elastic Agent inside the master branch, the Agent can be configured locally or using the ingest manager in Kibana. The Agent will take care of starting or stopping the processes depending on the configuration needs.

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

* updated rules

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

* updated rules

* Refactor: Rename pipeline into stream To align the code base with the multiple output strategy we will rename the pipeline concept into the streams. See #14445

This PR updates the enroll command to use an Elasticsearch token instead of custom token for enrollment. See elastic/kibana#49639 for details

* Change the Authentication mechanism for fleet. Fleet changed how we should authenticate with them instead of using an api key and a special header fields we now use the normal "Authorization" key and with this format: ``` Authorization: ApiKey {accessApiKey} ``` See #49639 for details

Add support for multiples outputs and link it to the streams.

* Implements the require "Checkin" calls This PR implements the required structs and calls to retrieve the configuration from Fleet. This is step one before refactoring the Reporter to split the pushing event and fetching of configuration.

) * Fakewebapi simulare a Fleet API endpoint for testing the checkin This PR creates a crude fake webserver to allow a beat agent to enroll and receives action from a remote host.

* Feature: Allow to persist AccessToken and other connection data on disk This PR add a few things - It takes the keystore encryption implementation and create an io.Reader and io.Writer for it, we currently work with variable lenght blocks and we are not compatible for now with the one in libbeat. - It implements a few storage mechanism and wrapper: - DiskStore: Save a io.reader directly to disk, the content is saved in a temporary file and the target is replace in an atomic operation. - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader and io.Writer. - A fleet configuration is created in _meta/agent.fleet.yml, the content of this file is packed in the binary. When a user enroll the agent, we do a backup of the current agent.yml and the content is replaced with the agent.fleet.yml - The enrollment information is saved into a "fleet.yml" - The managed mode now read the content of the fleet.yml and creates a Kibana API client. - A Separates Fleet Config struct is created. - Enroll will now ask for confirmation before replacing the user configuration. * allow to use the obfuscated store. * fix build

This PR add the fix from urso/ecslog#17 Which fix the infinite loop when using parsing flags, the bug was making the library consume all the available memory until the OS killed the process.

This PR add support for these types to make sure the convertion is correctly done. The problem was the generated map had NIL values instead of having the unsigned int values. Fix: #15182

* fixed yml files * configuration embed

Include metricbeat and Filebeat inside the agent.

* Feature: Allow to persist AccessToken and other connection data on disk This PR add a few things - It takes the keystore encryption implementation and create an io.Reader and io.Writer for it, we currently work with variable lenght blocks and we are not compatible for now with the one in libbeat. - It implements a few storage mechanism and wrapper: - DiskStore: Save a io.reader directly to disk, the content is saved in a temporary file and the target is replace in an atomic operation. - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader and io.Writer. - A fleet configuration is created in _meta/agent.fleet.yml, the content of this file is packed in the binary. When a user enroll the agent, we do a backup of the current agent.yml and the content is replaced with the agent.fleet.yml - The enrollment information is saved into a "fleet.yml" - The managed mode now read the content of the fleet.yml and creates a Kibana API client. - A Separates Fleet Config struct is created. - Enroll will now ask for confirmation before replacing the user configuration. * allow to use the obfuscated store. * Allow the Agent to retrieve configuration from the Fleet API. implement the required gateway and the action handling to allow to start, restart or stops processes based on the received configuration. Ref: #14951

* uuid for agent id * fixed hound dog * pass agent info * pass agent info * agent infor comments * notice update

…ing components (#15213)

ph · 2020-03-03T19:11:55Z

 	github.com/stretchr/testify v1.4.0
 	github.com/tsg/go-daemon v0.0.0-20200207173439-e704b93fd89b
 	github.com/tsg/gopacket v0.0.0-20190320122513-dd3d0e41124a
-	github.com/urso/magetools v0.0.0-20200106130147-61080ed7b22b // indirect


@kvch I will need your help on theses two dependencies (objx, magetools), I can't explain why they get removed.
I've tested with the master brancher and I do not have this issue.

as recommended by @urso I've ran mage vendor

ph · 2020-03-03T19:13:58Z

@urso @ruflin I have updated this PRs, sadly this bring back the dependencies. with the module to go module all the dependencies in #16616 were removed. I've found a concurrency issue I've disabled the test and created an issue #16771

urso · 2020-03-04T11:48:35Z

 	github.com/tsg/go-daemon v0.0.0-20200207173439-e704b93fd89b
 	github.com/tsg/gopacket v0.0.0-20190320122513-dd3d0e41124a
-	github.com/urso/magetools v0.0.0-20200106130147-61080ed7b22b // indirect
+	github.com/urso/ecslog v0.0.0-20200210125449-b79160ce24a2


This one looks wrong (dependencies are missing). I created a v0.0.1 tag on ecslog. Please use it to get the correct version and dependencies.

@urso I've fixed the version of ecslog on v0.0.1

ph · 2020-03-05T14:16:07Z

@urso @ruflin ready to be reviewed again.

ph · 2020-03-05T15:01:11Z

fixed again because of transport move.

ph · 2020-03-05T19:56:08Z

I've skipped and fixed some of the windows tests the #16852 and created #16854 for a followup.

ph · 2020-03-05T21:14:46Z

related created issue #16860

ph · 2020-03-06T18:45:35Z

Merging into master 🤗

ruflin · 2020-03-09T06:51:08Z

Hooray, well done team! 👏

This PR move the development of the Elastic Agent inside the master branch, the Agent can be configured locally or using the ingest manager in Kibana. The Agent will take care of starting or stopping the processes depending on the configuration needs. Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> (cherry picked from commit a9a2779)

michalpristas and others added 30 commits October 25, 2019 09:43

Moved agent into beats repo (#14163)

c7b7b2b

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

ca cannot be shorthand (#14366)

56ffef0

revisiting sample configuration and bringin it up to date (#14367)

20cfcdd

Agent: updated transpile rules for metricbeat (#14403)

4389042

* updated rules

Moved agent into beats repo (#14163)

565c4ad

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

ca cannot be shorthand (#14366)

471be1d

revisiting sample configuration and bringin it up to date (#14367)

073aa0b

Agent: updated transpile rules for metricbeat (#14403)

9c124fc

* updated rules

conflicts resolved

c8c8c24

merged conflicts

e2ac4ab

Refactor: Rename pipeline into stream (#14593)

441bde5

* Refactor: Rename pipeline into stream To align the code base with the multiple output strategy we will rename the pipeline concept into the streams. See #14445

Merge branch 'master' into fleet

ce8bff4

Change from Enrollment token to use Elasticsearch API key (#14869)

b85351b

This PR updates the enroll command to use an Elasticsearch token instead of custom token for enrollment. See elastic/kibana#49639 for details

Agent add support for multiples outputs (#14830)

b17619f

Add support for multiples outputs and link it to the streams.

add delta wait to ensure bucket ticked (#14932)

fa0d309

[Fleet] Enable log introspections (#14390)

1bc5895

Fakewebapi simulare a Fleet API endpoint for testing the checkin (#14945

aef8334

) * Fakewebapi simulare a Fleet API endpoint for testing the checkin This PR creates a crude fake webserver to allow a beat agent to enroll and receives action from a remote host.

resolved conflicts with master

a379823

[Agent] Grpc support for agent to beat communication (#14835)

1743bbc

Update ecslog to fix an infinite loop (#15142)

38591b9

This PR add the fix from urso/ecslog#17 Which fix the infinite loop when using parsing flags, the bug was making the library consume all the available memory until the OS killed the process.

Update notice (#15189)

73fc672

The MapVisitor didn't handle correctly Uint and Uin64 (#15187)

4b479d2

This PR add support for these types to make sure the convertion is correctly done. The problem was the generated map had NIL values instead of having the unsigned int values. Fix: #15182

[Agent] Make default config congestable by agent (#15191)

0b978c7

* fixed yml files * configuration embed

[Agent] Include beats in agent package (#15046)

f307368

Include metricbeat and Filebeat inside the agent.

[Agent] Agent UUID step 1 (#15144)

e25649f

* uuid for agent id * fixed hound dog * pass agent info * pass agent info * agent infor comments * notice update

[Agent] Parse and pass correct configuration sections into correspond…

7104867

…ing components (#15213)

ph mentioned this pull request Mar 3, 2020

[Agent] Concurrency issue in fleet gateway #16771

Closed

ph added 2 commits March 3, 2020 14:08

Will need another pair of eye on this problem

8ab4c79

Disable suite followup in #16771

b8ec22b

ph commented Mar 3, 2020

View reviewed changes

mage vendor

38a913d

urso reviewed Mar 4, 2020

View reviewed changes

ph added 3 commits March 4, 2020 10:41

tag v0.0.1 for ecslog

388c906

notice

5887bdd

Fix windows build

84ee7cf

ph added 2 commits March 5, 2020 09:54

Merge remote-tracking branch 'origin/master' into merge-agent-in-master

f2bf6aa

package has been moved..

8538f4d

ph added 3 commits March 5, 2020 13:11

Merge remote-tracking branch 'origin/master' into merge-agent-in-master

05d37e0

Fixing windows tests #1

b9a239c

Fix and skipped more tests

462f2c6

urso approved these changes Mar 5, 2020

View reviewed changes

More skip on windows

309365c

ph added 2 commits March 6, 2020 11:42

placeholder for windows. system test.

4609c06

Fixit.

7baf137

ph merged commit a9a2779 into elastic:master Mar 6, 2020

ph mentioned this pull request Mar 6, 2020

Cherry-pick #16494 to 7.x: Merge agent to master #16885

Merged

5 tasks

ph added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge agent to master#16494

Merge agent to master#16494
ph merged 116 commits intoelastic:masterfrom
ph:merge-agent-in-master

ph commented Feb 21, 2020 •

edited

Loading

Uh oh!

ph Mar 3, 2020 •

edited

Loading

Uh oh!

ph Mar 3, 2020

Uh oh!

ph commented Mar 3, 2020

Uh oh!

urso Mar 4, 2020

Uh oh!

ph Mar 4, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 6, 2020

Uh oh!

ruflin commented Mar 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ph commented Feb 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

ph Mar 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ph Mar 3, 2020

Choose a reason for hiding this comment

Uh oh!

ph commented Mar 3, 2020

Uh oh!

urso Mar 4, 2020

Choose a reason for hiding this comment

Uh oh!

ph Mar 4, 2020

Choose a reason for hiding this comment

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 5, 2020

Uh oh!

ph commented Mar 6, 2020

Uh oh!

ruflin commented Mar 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ph commented Feb 21, 2020 •

edited

Loading

ph Mar 3, 2020 •

edited

Loading