Add Fleet API Client #311

taylor-swanson · 2023-04-27T16:21:44Z

Add API client for Fleet
Add code generator for Fleet API types, uses OpenAPI spec
Add related docs and tests
Data sources and resources will come in later PRs

- Add resource for Fleet Agent Policies, Server Hosts, and Outputs - Add data source for Fleet Enrollment Tokens - Add API client for Fleet - Add code generator for Fleet API types, uses OpenAPI spec - Add related docs and tests

…rit kibana values

tobio · 2023-04-27T23:46:40Z

This is awesome! Thanks for the work here. Are we able to split this one up a little to simplify the review process. It feels like it should be relatively easy to have a PR per resource/data source, and maybe an initial one adding the fleet client?

taylor-swanson · 2023-04-28T13:20:19Z

This is awesome! Thanks for the work here. Are we able to split this one up a little to simplify the review process. It feels like it should be relatively easy to have a PR per resource/data source, and maybe an initial one adding the fleet client?

I think that should be doable! Yeah the codegen adds a bit of weight to the review, along with the new fleet client. The output and fleet_server_host resources were also late additions. It's pretty easy to separate these out into separate changes, so I'll go ahead and do that. I also need to figure out the minimum supported version of each data source/resource for the acceptance tests, since not every version of the stack supports every fleet endpoint.

I'll go ahead and remove all of the resources from this PR, but I'll maybe leave the data source in (enrollment tokens). I may need at least one thing in there for codegen to work properly, but I'll see about that.

- Will be re-added in later PRs, this PR will only be for the new fleet client

taylor-swanson · 2023-04-28T13:53:12Z

Okay, removed all of the data source/resources from this PR, so it will just be about the fleet client. I'll add those back in later PRs.

~~Also, noticed buildkite is running but it doesn't seem to have a pipeline file, so it failed.~~ (Never mind, I think that was because I accidentally pushed the branch to upstream and not my fork).

tobio

Couple of comments inline, generally lgtm. Modifying the openapi spec during generation feels pretty dangerous, is there a reason we need that here, shouldn't we be able to publish a working spec?

go.mod

internal/clients/api_client.go

tobio · 2023-04-27T23:35:32Z

internal/clients/fleet/fleetapi/generate.go

This file terrifies me, why do we need to be editing the openapi spec?

This was originally written by @andrewkroh and I continued to use it as it seemed to work fine.

Andrew, did we edit it to work around some issues with the API, and is it even necessary anymore?

Also, I'm looking at using openapitools/openapi-generator-cli to see if that works well with the Fleet API. If it works, I'll use that instead, as I'd rather be consistent with everything else.

Okay, so OpenAPI Generator cannot generate valid code with the Fleet spec:

$ make build-ci go build -o terraform-provider-elasticstack # github.com/elastic/terraform-provider-elasticstack/generated/fleet generated/fleet/model_package_policy.go:39:6: NewPackagePolicy redeclared in this block generated/fleet/model_new_package_policy.go:21:6: other declaration of NewPackagePolicy make: *** [build-ci] Error 1

Also, it added some serious bloat to this change:

411 files changed, 86208 insertions(+), 541 deletions(-)

If we could limit what models are generated, that may help. We are waiting on some changes to be made in the API for package policies, so we don't want them being generated now anyway.

@andrewkroh are you able to help me understand the need for this file?

Andrew, did we edit it to work around some issues with the API, and is it even necessary anymore?

Maybe we can try generating the client without this file and see what the result is?

I wrote this when I was hacking a PoC. In it's current state I never intended it for production. It does a few things. Some of which should be incorporated into the upstream schema.

It filters the generated code to only include the parts that are used. This removes a bunch of unused and deprecated models.

It fixes some issues with the generated code to avoid anonymous nested structs. This makes the generated code easier to use.

It adds some parameters that are missing from the openapi spec, but are used by the real API as observed by inspecting Kibana's requests to the API.

I cleaned this code up, so hopefully less "hacks". It's driven mostly by structs, so a bit safer to deal with now instead of blind maps. That said, I left in the filtering of paths and methods, since that is actually very useful to us and cuts down on the code bloat. I also kept in the filtering of the kbn-xsrf parameters. If I understand it correctly, we don't need this in generated code as we set the header in our Fleet client (correct me if I'm wrong, @andrewkroh).

@tobio, any other comments on this?

I think given the current state of the Fleet OpenAPI spec, the filtering that we are currently doing (especially with my latest round of changes), is fair. To recap, all I am doing now is:

Filtering out unneeded/deprecated API endpoints and methods. As Andrew mentioned, this dramatically cuts down on the amount of code generated.

Removed the kbn-xsrf parameter from generated code. As mentioned, we already set the header in the API client. I have not noticed any issues when removing this code.

Since this is all still considered experimental code, I'd rather get this in and start iterating on it sooner than later. We plan to use this extensively on the Security External Integrations team for our demo environment to continuously test our integrations.

Yep, they're all good points. 👍 on getting this in and making progress.

internal/clients/fleet/fleetapi/generate.go

internal/clients/fleet/debug.go

…ent variables

…nt, fix empty fleet client being generated

tobio

🎉 thanks for all the effort here, looking forward to getting the resources added too.

andrewkroh

Got a chance to look it over today. Nice work.

andrewkroh · 2023-05-11T23:35:59Z

internal/clients/fleet/fleetapi/doc.go

@@ -0,0 +1,5 @@
+package fleetapi
+
+//go:generate go run generate.go -o fleet-filtered.json


Do you think we should use -v v8.7.1 or some release tag instead of generating from main (from default)?

Yeah I think we should once the API goes GA (or whenever we get the package policy changes?). I could quickly test it right now and if there's not any significant differences, we could lock to v8.7.1 or something equivalent

taylor-swanson added 2 commits April 27, 2023 11:20

Add Fleet Support

94b234c

- Add resource for Fleet Agent Policies, Server Hosts, and Outputs - Add data source for Fleet Enrollment Tokens - Add API client for Fleet - Add code generator for Fleet API types, uses OpenAPI spec - Add related docs and tests

Update changelog

baaabd7

taylor-swanson self-assigned this Apr 27, 2023

taylor-swanson added 10 commits April 27, 2023 11:26

make fmt

91097b8

Update docs after fmt

9412214

Remove problematic fleet config check, Make fleet acctest config inhe…

94142f6

…rit kibana values

Fix config issues

fee273d

Regen docs

cdc28ba

Add fleet endpoint to test config

be706e4

Also add user and pass for fleet test

00d9443

Enable fleet in kibana, add version constraints to tests

0235e5f

Fix yml

312f010

Fix wrong password

af40310

Remove fleet data-source/resources

8c092f3

- Will be re-added in later PRs, this PR will only be for the new fleet client

taylor-swanson changed the title ~~Add Fleet Support~~ Add Fleet API Client Apr 28, 2023

taylor-swanson marked this pull request as ready for review April 28, 2023 13:51

Update changelog description

aa8e942

tobio reviewed May 2, 2023

View reviewed changes

taylor-swanson added 8 commits May 2, 2023 10:06

Move oapi-codegen go.mod entry to tools

897def3

Merge branch 'main' into fleet-provider

06b4111

Revert debug.go back to what's in main

6e3cffd

Relocate duplicate print JSON lines to utils package

073d683

Use kibana config as base config for fleet, remove redundant environm…

e14f46a

…ent variables

Expose ELASTICSEARCH_INSECURE env variable in acceptance testing clie…

be1eeaf

…nt, fix empty fleet client being generated

Clean up generate.go

4252ddc

Merge branch 'main' into fleet-provider

e628338

tobio approved these changes May 9, 2023

View reviewed changes

tobio merged commit f291236 into elastic:main May 9, 2023

andrewkroh reviewed May 11, 2023

View reviewed changes

taylor-swanson deleted the fleet-provider branch May 15, 2023 17:18

		@@ -0,0 +1,5 @@
		package fleetapi

		//go:generate go run generate.go -o fleet-filtered.json

Add Fleet API Client #311

Add Fleet API Client #311

Uh oh!

Conversation

taylor-swanson commented Apr 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobio commented Apr 27, 2023

Uh oh!

taylor-swanson commented Apr 28, 2023

Uh oh!

taylor-swanson commented Apr 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tobio left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh May 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taylor-swanson commented Apr 27, 2023 •

edited

Loading

taylor-swanson commented Apr 28, 2023 •

edited

Loading

andrewkroh May 11, 2023 •

edited

Loading