[GCS] Adding new integration for Custom GCS Input #4692

P1llus · 2022-11-22T11:25:48Z

What does this PR do?

This adds an integration for the new Google Cloud Storage input that was added to filebeat in 8.5.0

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Custom GCS input package #4330

elasticmachine · 2022-11-22T11:25:51Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

P1llus · 2022-11-22T11:27:15Z

There is currently a few questions up for discussion here:

Will we add system tests, and how we should do it.
Which ECS fields should we use, continuing a discussion around if we should include at least fields related to common metadata processors.
Needs a new icon.

elasticmachine · 2022-11-22T11:30:39Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-07T07:37:36.361+0000
Duration: 14 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-11-22T11:42:23Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚
Classes	100.0% (`0/0`)	💚
Methods	100.0% (`3/3`)	💚
Lines	100.0% (`0/0`)	💚 11.296
Conditionals	100.0% (`0/0`)	💚

P1llus · 2022-11-22T12:02:06Z

Integration has not been fully tested yet, but will be soon (before merging)

P1llus · 2022-11-23T09:38:11Z

Waiting for this to merge to add system tests: elastic/stream#46

kcreddy

Just minor change requested. Rest LGTM 👍🏼

packages/google_cloud_storage/data_stream/generic/manifest.yml

kcreddy · 2022-12-01T10:37:22Z

packages/google_cloud_storage/data_stream/generic/agent/stream/gcs.yml.hbs

+{{#if service_account_file}}
+auth.credentials_file.path: {{service_account_file}}
+{{/if}}
+{{#if buckets}}


Another field parse_json could be added as per the doc

botelastic · 2022-12-31T11:05:02Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

narph · 2023-01-09T08:32:25Z

@P1llus , can we get this in?

ShourieG · 2023-01-09T09:02:00Z

@narph @P1llus I feel if we could wait till this PR: elastic/beats#34155 is merged, it would be awesome. This contains a lot of significant improvements to the GCS input which makes it more stable and behave better at scale. Having the integration live after the changes have gone through would be ideal.

P1llus · 2023-01-16T18:38:52Z

This was waiting for elastic/stream#46 and elastic/elastic-package#1073
The latter was only merged a few days ago, so I will go ahead and finish the first one.

The reason behind it was so that we could implement the dynamic ECS template already from the initial version. It also seems that we should bump the minimum version as per @ShourieG comment, so we can ensure we have all the features needed for the input to work as intended.

P1llus · 2023-01-19T10:10:05Z

All blocking features have now been resolved, will just have to update the PR with the system test and add dynamic ECS template before merging: #5055

ShourieG · 2023-01-23T07:39:19Z

@narph @P1llus PR: elastic/beats#34155 is now merged, hence unblocking this PR.

P1llus · 2023-01-23T11:39:53Z

This new input needs to be added to the filebeat spec file for elastic-agent first: https://github.com/elastic/elastic-agent/blob/main/specs/filebeat.spec.yml

P1llus · 2023-01-23T14:34:37Z

Spec file updated as per: elastic/elastic-agent#2149
Awaiting new snapshot image to be generated.

P1llus · 2023-01-25T15:08:04Z

There is currently no possible way to override the URL endpoint for the GCS client, we would need to add a alternative_host variable, similar to gcp-pubsub, so that we have the possibility to route the client to our local system test.

…kage_gcs

packages/google_cloud_storage/_dev/build/docs/README.md

Co-authored-by: ShourieG <[email protected]>

ShourieG

LGTM

elasticmachine · 2023-02-07T08:19:41Z

Package google_cloud_storage - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=google_cloud_storage

initial commit

8d8dddc

P1llus added Team:Security-External Integrations Integration:google_cloud_storage Custom GCS (Google Cloud Storage) Input labels Nov 22, 2022

adding changelog and CODEOWNERS

3a61af9

P1llus mentioned this pull request Nov 22, 2022

[Azure Blob Storage] Adding integration for Custom Azure Blob Storage Input #4693

Merged

4 tasks

P1llus requested a review from a team November 22, 2022 11:54

kcreddy approved these changes Dec 1, 2022

View reviewed changes

botelastic bot added the Stalled label Dec 31, 2022

botelastic bot removed the Stalled label Jan 9, 2023

P1llus mentioned this pull request Jan 23, 2023

[Enhancement] Adding GCS and ABS to allowed inputs in filebeat spec elastic/elastic-agent#2149

Merged

6 tasks

P1llus added 5 commits February 6, 2023 11:26

Final MVP commit

0e6d8fd

stashing changes

fa05077

Merge branch 'main' into package_gcs

51da75b

Initial MVP commit

2e89141

Merge branch 'package_gcs' of github.com:P1llus/integrations into pac…

31bacfa

…kage_gcs

update tags

9ee8266

ShourieG reviewed Feb 7, 2023

View reviewed changes

packages/google_cloud_storage/_dev/build/docs/README.md Outdated Show resolved Hide resolved

P1llus and others added 2 commits February 7, 2023 08:36

Update packages/google_cloud_storage/_dev/build/docs/README.md

5e5e226

Co-authored-by: ShourieG <[email protected]>

elastic-package format

b2f3e46

ShourieG approved these changes Feb 7, 2023

View reviewed changes

P1llus merged commit 8607186 into elastic:main Feb 7, 2023

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GCS] Adding new integration for Custom GCS Input #4692

[GCS] Adding new integration for Custom GCS Input #4692

P1llus commented Nov 22, 2022 •

edited

Loading

elasticmachine commented Nov 22, 2022

P1llus commented Nov 22, 2022 •

edited

Loading

elasticmachine commented Nov 22, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Nov 22, 2022 •

edited

Loading

P1llus commented Nov 22, 2022

P1llus commented Nov 23, 2022

kcreddy left a comment

kcreddy Dec 1, 2022

botelastic bot commented Dec 31, 2022

narph commented Jan 9, 2023

ShourieG commented Jan 9, 2023

P1llus commented Jan 16, 2023

P1llus commented Jan 19, 2023

ShourieG commented Jan 23, 2023

P1llus commented Jan 23, 2023

P1llus commented Jan 23, 2023

P1llus commented Jan 25, 2023

ShourieG left a comment

elasticmachine commented Feb 7, 2023

[GCS] Adding new integration for Custom GCS Input #4692

[GCS] Adding new integration for Custom GCS Input #4692

Conversation

P1llus commented Nov 22, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented Nov 22, 2022

P1llus commented Nov 22, 2022 • edited Loading

elasticmachine commented Nov 22, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Nov 22, 2022 • edited Loading

🌐 Coverage report

P1llus commented Nov 22, 2022

P1llus commented Nov 23, 2022

kcreddy left a comment

Choose a reason for hiding this comment

kcreddy Dec 1, 2022

Choose a reason for hiding this comment

botelastic bot commented Dec 31, 2022

narph commented Jan 9, 2023

ShourieG commented Jan 9, 2023

P1llus commented Jan 16, 2023

P1llus commented Jan 19, 2023

ShourieG commented Jan 23, 2023

P1llus commented Jan 23, 2023

P1llus commented Jan 23, 2023

P1llus commented Jan 25, 2023

ShourieG left a comment

Choose a reason for hiding this comment

elasticmachine commented Feb 7, 2023

P1llus commented Nov 22, 2022 •

edited

Loading

P1llus commented Nov 22, 2022 •

edited

Loading

elasticmachine commented Nov 22, 2022 •

edited

Loading

elasticmachine commented Nov 22, 2022 •

edited

Loading