[gcp] Migrate some gcp beat metricset to data streams by endorama · Pull Request #2707 · elastic/integrations

endorama · 2022-02-16T17:58:57Z

What does this PR do?

Metricsets from gcp beat module must be migrated to gcp integration package data streams, to support using them with the agent.

This PR will contain metricset migration and tests for each of them. It will be a long running PR, as I want to add tests for each data stream, something today completely missing, but I want to have it open as a draft to allow following progresses.

Partial PRs will target gcp-metrics branch to add to this feature allowing to break down the scope of work in manageable chunks.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Part of Migrate some Metricbeat modules to GCP package #490

Screenshots

endorama · 2022-02-17T15:22:54Z

There have been an issue identified in current gcp package that is present in related code too #2712

endorama · 2022-03-02T15:51:12Z

Adding tests requires elastic-package Terraform deployer to support gcloud SDK. This has been implemented with elastic/elastic-package#638
That implementation was partially working due to authentication issues, tackled in elastic/elastic-package#662

To complete elastic/elastic-package#662 has been considered necessary adding a test gcp package to elastic-package, tackled in elastic/elastic-package#701

elasticmachine · 2022-03-17T17:10:01Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-13T09:02:56.770+0000
Duration: 20 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	46
Skipped	0
Total	46

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

andrewkroh · 2022-03-22T20:23:45Z

packages/gcp/manifest.yml

+        title: "Collect Google Cloud Platform (GCP) firewall logs (input: gcp-pubsub)"
+        description: "Collecting firewall logs from Google Cloud Platform (GCP) instances (input: gcp-pubsub)"
+        input_group: logs
+  - name: vpcflow


I like the organization of the package, but have a concern for existing users. What will happen when existing users upgrade?

I was trying to so something similar using policy_template in the CrowdStrike module, but when I tested upgrading I have hit some issues (see #2806). So I recommend doing a test where you setup gcp with an existing version, then try to upgrade to this new version.

What are in particular the things that break? I don't know any way to assist users with upgrades that we may use to help in this use case.
We also cannot use input level variables as those are not yet supported by Kibana (see elastic/kibana#112272)

I think a possible workaround would be an upgrade guide and a major version bump to signal the breaking change. What do you think? Could this be a valid strategy for your use case too?

What are in particular the things that break?

I suspect the issue lies with creating separate policy_templates which IIUC basically creates different integrations from a user perspective. Then there is no upgrade path from the older config, but TBH I'm not really sure. That's why I would recommend doing an quick independent test with gcp to check if you hit upgrade problems. Better to find the issue earlier and we can discuss what we can do to solve it or workaround.

I tested the upgrade path and your fears were correct. I'm going to write down my investigation in a separate issue.

The upgrade issue was tracked in elastic/kibana#131251 and has been resolved, I confirmed the fix in latest 8.3 build candidate.

packages/gcp/changelog.yml

endorama · 2022-05-11T15:30:56Z

~~A note for my future self: GCP Firestore data stream (that should be in this branch after merging #2704) is not there, probably due to a force push.~~
Changes reintegrated

elasticmachine · 2022-06-09T13:24:11Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`4/4`)	💚
Files	100.0% (`4/4`)	💚 3.145
Classes	100.0% (`4/4`)	💚 3.145
Methods	93.939% (`62/66`)	👍 4.43
Lines	95.969% (`1119/1166`)	👍 4.903
Conditionals	100.0% (`0/0`)	💚

endorama · 2022-06-13T09:18:06Z

/test

packages/gcp/changelog.yml

packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs

endorama · 2022-06-17T09:51:26Z

There is some internal discussion going on about how to handle the breaking change that this PR would introduce. For the moment this is not going to be merged, so I'm reverting this to Draft waiting for the discussion to reach a conclusive point.

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842 Changes to packages/gcp/data_stream/audit/manifest.yml has been removed too.

This configuration is not working and break test execution, as seen in https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-2707/11/pipeline Error in logs: ``` Error: error running package system tests: could not complete test run: could not setup service: can't attach service container to the stack network: could not attach container to the stack network (stderr="Error response from daemon: No such container: elastic-package-service_gcp_1\n"): exit status 1 script returned exit code 1 ``` This test case is copied over from https://github.com/elastic/elastic-package/blob/79c22bca9ccd95c1b38861bcea5c4298057a78f8/test/packages/parallel/gcp/data_stream/compute/_dev/test/system/test-default-config.yml Removing to unblock moving forward. Tests to ensure the data stream works have been done manually.

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

endorama · 2022-07-12T15:37:22Z

Decision has been made and this change has been considered non breaking as there is an upgrade path and the upgrade is manually triggered by a user.

I've updated the Kibana constraint to 8.3.0 in edc3905 to prevent users from updating to new package version on Kibana affected by elastic/kibana#131251

efd6

Minor comments.

efd6 · 2022-07-12T23:12:56Z

packages/gcp/changelog.yml

+- version: 2.0.0
+  changes:
+    - description: |
+        Move configurations to support metrics. This change is breaking, as it move


Suggested change

Move configurations to support metrics. This change is breaking, as it move

Move configurations to support metrics. This change is breaking, as it moves

Fixed in 8d43275

efd6 · 2022-07-12T23:13:57Z

packages/gcp/changelog.yml

+        variables again when upgrading the policies to this version.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/2707
+    - description: Add GCP Billing Data Stream


Are there docs for this data stream? (if adding, it would be worth including a link to https://cloud.google.com/billing/docs/reports since there are a lot of details that help understand the documents).

Added documentation in dc10062

Thank you for pointing this out!

efd6 · 2022-07-12T23:18:17Z

packages/gcp/_dev/build/docs/compute.md

+
+## Metrics
+
+This is the `compute` dataset.


Is it worth fleshing these out for consistency with the other data_stream docs? They go into a little bit of detail about what they are collecting.

Added documentation in dc10062

efd6 · 2022-07-12T23:18:46Z

packages/gcp/_dev/build/docs/firestore.md

+
+## Metrics
+
+This is the `firestore` dataset.


... and here.

Added documentation in dc10062

efd6 · 2022-07-12T23:48:44Z

packages/gcp/data_stream/audit/manifest.yml

+        multi: false
+        required: false
+        show_user: false
+        description: "GCP Alternative host"


I think these could be fleshed out a little so the user know when to use it, or to not use it. From the beat code, "Overrides the default Pub/Sub service address and disables TLS. For testing."

Done in 71facbc

efd6 · 2022-07-13T00:03:48Z

packages/gcp/manifest.yml

+        title: filebeat gcp audit
+        size: 1702x996
+        type: image/png
+  - name: firwall


s/firwall/firewall/

Fixed in 8d43275

Before version 8.3.0 a bug in the Fleet UI configuration conflict resolution screen would prevent a successful update of this package to the new version. The constraint is updated on Kibana >= 8.3 where the related fix landed[1] so we can guarantee a working upgrade path. [1]: elastic/kibana#132068

After internal discussion, the fix to the Fleet UI conflict resolution has not been backported to 7.17, thus making this upgrade a breaking change without a valid upgrade path (it would still be possible to not upgrade policies and only create new ones but this is uncharted territory). Based on this we decided to remove support for 7.17 altogether from gcp 2.0.0

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

endorama · 2022-07-13T09:05:58Z

Please note that after internal discussion due to a bug in Kibana < 8.3 not backported to 7.17 we decided to remove support for 7.17 release track in dec0dc1

I'm discussing if the fix applied to 8.3 can/will be backported to 7.17 or if there are limitations that prevent that.

Update: fixes can be backported. I discussed with @tommyers-elastic (and on suggestion from @andresrc) and we decided to merge this as is it and re-introduce 7.17 compatibility in a later release of the package to unblock this and test 7.17 appropriately.

cc @andresrc

endorama added enhancement New feature or request Integration:gcp Google Cloud Platform Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Feb 16, 2022

endorama self-assigned this Feb 16, 2022

endorama mentioned this pull request Mar 2, 2022

Migrate some Metricbeat modules to GCP package #490

Closed

11 tasks

endorama linked an issue Mar 2, 2022 that may be closed by this pull request

Migrate some Metricbeat modules to GCP package #490

Closed

11 tasks

endorama force-pushed the gcp-metrics branch from ddbbb84 to c414665 Compare March 17, 2022 17:01

endorama force-pushed the gcp-metrics branch 3 times, most recently from 8cf80ec to fe54af3 Compare March 22, 2022 15:14

andrewkroh reviewed Mar 22, 2022

View reviewed changes

sayden reviewed Mar 29, 2022

View reviewed changes

packages/gcp/changelog.yml Outdated Show resolved Hide resolved

sayden reviewed Mar 29, 2022

View reviewed changes

packages/gcp/changelog.yml Outdated Show resolved Hide resolved

endorama mentioned this pull request Apr 4, 2022

[GCP] Configuration breaking changes due to metrics data streams addition #2987

Closed

endorama force-pushed the gcp-metrics branch from aaab6bd to cc98bd1 Compare April 5, 2022 15:08

endorama force-pushed the gcp-metrics branch from 471f0fc to fa4dc99 Compare June 9, 2022 13:07

endorama marked this pull request as ready for review June 13, 2022 13:59

endorama requested review from a team as code owners June 13, 2022 13:59

kaiyan-sheng reviewed Jun 14, 2022

View reviewed changes

packages/gcp/changelog.yml Outdated Show resolved Hide resolved

kaiyan-sheng reviewed Jun 14, 2022

View reviewed changes

packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs Show resolved Hide resolved

endorama marked this pull request as draft June 17, 2022 09:51

[GCP] Add GCP Billing Metricset (#2141)

25c29cd

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842 Changes to packages/gcp/data_stream/audit/manifest.yml has been removed too.

gpop63 and others added 8 commits July 12, 2022 17:15

[GCP] Add firestore data_stream (#2704)

c8f00a8

firestore changelog

dba820d

update docs

51552e4

billing,compute: fix credentials_json escaping

7b51b6b

gcp: update changelog with more details on breaking change

5e354b0

Update changelog.yml

a3da559

Update packages/gcp/changelog.yml

6b97c3a

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

endorama force-pushed the gcp-metrics branch from 812cade to 6b97c3a Compare July 12, 2022 15:30

endorama marked this pull request as ready for review July 12, 2022 15:30

endorama force-pushed the gcp-metrics branch from e966bcd to edc3905 Compare July 12, 2022 15:38

efd6 reviewed Jul 13, 2022

View reviewed changes

endorama force-pushed the gcp-metrics branch from edc3905 to 826b985 Compare July 13, 2022 07:54

endorama and others added 5 commits July 13, 2022 10:00

add metrics datasets docs

dc10062

update docs

7257e23

fix typos

8d43275

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

improve details on alterative_host usage

71facbc

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

tommyers-elastic approved these changes Jul 13, 2022

View reviewed changes

efd6 approved these changes Jul 13, 2022

View reviewed changes

endorama changed the title ~~[gcp] Migrate beat metricset to data streams~~ [gcp] Migrate some gcp beat metricset to data streams Jul 14, 2022

endorama merged commit 5500034 into main Jul 14, 2022

endorama deleted the gcp-metrics branch July 14, 2022 08:49

endorama mentioned this pull request Jul 14, 2022

[GCP] Re introduce compatibility with 7.17 #3714

Closed

andrewkroh mentioned this pull request Jul 18, 2022

[production] Promote packages from snapshot (gcp-2.0.0, zscaler_zia-2.3.0) elastic/package-storage#4888

Closed

andrewkroh mentioned this pull request Jul 26, 2022

[production] Promote many packages from snapshot elastic/package-storage#5004

Merged

girodav mentioned this pull request Jun 22, 2023

[AWS][EC2] Add processors to rename and drop fields #5812

Closed

4 tasks

	Move configurations to support metrics. This change is breaking, as it move
	Move configurations to support metrics. This change is breaking, as it moves

Conversation

endorama commented Feb 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

endorama commented Feb 17, 2022

Uh oh!

endorama commented Mar 2, 2022

Uh oh!

elasticmachine commented Mar 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

endorama commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

endorama commented Jun 13, 2022

Uh oh!

Uh oh!

Uh oh!

endorama commented Jun 17, 2022

Uh oh!

endorama commented Jul 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

endorama commented Jul 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

endorama commented Feb 16, 2022 •

edited

Loading

elasticmachine commented Mar 17, 2022 •

edited

Loading

endorama commented May 11, 2022 •

edited

Loading

elasticmachine commented Jun 9, 2022 •

edited

Loading

endorama commented Jul 12, 2022 •

edited

Loading

endorama commented Jul 13, 2022 •

edited

Loading