Skip to content

Add helper to get hosts on Go Metricbeat integration tests#13055

Merged
jsoriano merged 38 commits intoelastic:masterfrom
jsoriano:metricbeat-integration-tests-host
Aug 5, 2019
Merged

Add helper to get hosts on Go Metricbeat integration tests#13055
jsoriano merged 38 commits intoelastic:masterfrom
jsoriano:metricbeat-integration-tests-host

Conversation

@jsoriano
Copy link
Copy Markdown
Member

@jsoriano jsoriano commented Jul 24, 2019

Takes from #13039 and #7957 the required changes to allow
Metricbeat integration tests to obtain the hosts of the containers
from the framework, instead of needing to use environment
variables and tricky setups. This allows to run integration tests
for specific modules with go run -tags=integration, without
previous setup.

This is similar to the change introduced for python tests in #11925.

<SERVICE>_HOST environment variable can be set to the
host:port of a running service to avoid using compose, similarly
to old NO_COMPOSE environment variable.

This PR modifies many files, but most of the relevant changes
are in libbeat, and in the metricbeat docker compose. In the docker
compose file, env files are removed and metricbeat is run in the host
network. The rest of changes are to adapt the integration tests to
don't depend on environment variables to get the hosts information.

@jsoriano jsoriano added in progress Pull request is currently in progress. Metricbeat Metricbeat :Testing [zube]: In Progress Team:Integrations Label for the Integrations team labels Jul 24, 2019
@jsoriano jsoriano self-assigned this Jul 24, 2019
@jsoriano jsoriano force-pushed the metricbeat-integration-tests-host branch from 04489a8 to 2d8cdaf Compare July 26, 2019 10:27
Comment thread libbeat/tests/compose/project.go Outdated
Comment thread libbeat/tests/compose/project.go Outdated
@jsoriano jsoriano force-pushed the metricbeat-integration-tests-host branch 5 times, most recently from 5522efd to a3ce2e2 Compare July 29, 2019 19:21
Comment thread libbeat/tests/compose/wrapper.go Outdated
Comment thread libbeat/tests/compose/wrapper.go Outdated
Comment thread libbeat/tests/compose/wrapper.go Outdated
@jsoriano
Copy link
Copy Markdown
Member Author

jenkins, test this again please

1 similar comment
@jsoriano
Copy link
Copy Markdown
Member Author

jenkins, test this again please

@jsoriano jsoriano force-pushed the metricbeat-integration-tests-host branch from f12a31b to 8284004 Compare July 30, 2019 18:53
@jsoriano
Copy link
Copy Markdown
Member Author

jenkins, test this again please

@jsoriano jsoriano force-pushed the metricbeat-integration-tests-host branch from 82ee7f4 to e8941df Compare July 31, 2019 09:12
@jsoriano jsoriano force-pushed the metricbeat-integration-tests-host branch from 1982ad0 to b523145 Compare July 31, 2019 11:57
@jsoriano jsoriano marked this pull request as ready for review July 31, 2019 14:43
@jsoriano jsoriano requested review from a team as code owners July 31, 2019 14:43
@jsoriano jsoriano added [zube]: In Review review and removed [zube]: In Progress in progress Pull request is currently in progress. labels Jul 31, 2019
@zube zube Bot added the in progress Pull request is currently in progress. label Jul 31, 2019
@jsoriano jsoriano removed the in progress Pull request is currently in progress. label Aug 1, 2019
Copy link
Copy Markdown
Contributor

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great enabler! did a first pass and left some questions, I will look into it more tomorrow

Comment thread libbeat/tests/compose/compose.go

// EnsureUpWithTimeout starts all the requested services (must be defined in docker-compose.yml)
// Wait for `timeout` seconds for health
func EnsureUpWithTimeout(t *testing.T, timeout int, services ...string) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what’s the reason to accept only one service? not saying it’s a bad idea, just want to know the reasoning

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two main reasons:

  • It simplifies the returned information. With an only service, only an object with the host information for a service is returned.
  • It doesn't seem to be needed, we only had an scenario with two services (Kibana, that also needs Elasticsearch), and it can also be done with docker compose dependencies. If access to both hosts were needed, we could have two EnsureUp.

If wouldn't be hard to implement it though, by returning a struct with all the services info and adding a method like HostForServicePort(service, port) HostInfo. But I didn't consider it neccessary by now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds good to me, thanks for the explanation!

Comment thread metricbeat/module/aerospike/namespace/namespace_integration_test.go Outdated
Comment thread libbeat/tests/compose/wrapper.go Outdated
@exekias
Copy link
Copy Markdown
Contributor

exekias commented Aug 2, 2019

I'm happy if CI is happy

@jsoriano
Copy link
Copy Markdown
Member Author

jsoriano commented Aug 2, 2019

jenkins, test this again please


// Wait ensures all wanted services are healthy. Wait loop (60s timeout)
func (c *Project) Wait(seconds int, services ...string) error {
func (c *Project) Wait(seconds time.Duration, services ...string) error {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@jsoriano
Copy link
Copy Markdown
Member Author

jsoriano commented Aug 5, 2019

CI is happy, we all are happy 🙂

@jsoriano jsoriano merged commit 01d7bfd into elastic:master Aug 5, 2019
@jsoriano jsoriano deleted the metricbeat-integration-tests-host branch August 5, 2019 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Metricbeat Metricbeat review Team:Integrations Label for the Integrations team :Testing v7.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants