Adding var_groups spec

[seanrathier]

What does this PR do?

Add a new var_groups field to the package manifest schema that:

• Defines mutually exclusive groups of variables
• Controls variable visibility based on the selected option
• Stores the selection in the policy for backend processing
• Uses additionalProperties: true on options to allow feature-specific extensions

Why is it important?

Current Limitation

Fleet's generic UI renders all integration variables in a flat list, regardless of authentication method or configuration context. Users see fields that don't apply to their chosen setup (e.g., showing access_key_id when using Assume Role authentication), creating confusion and a poor user experience.

Specific Use Cases

1. Cloud Connector Integration: The Cloud Connector feature (for agentless AWS/GCP/Azure data collection) is currently only available in CSPM and Cloud Asset Inventory, which use custom UIs. Scaling this to other integrations (AWS GuardDuty, Security Hub, etc.) requires a manifest-driven approach.
2. Authentication Method Selection: AWS integrations support 5+ authentication methods (Direct Keys, Temporary Keys, Assume Role, Shared Credentials, Cloud Connector), each with different required variables.
3. Generic Conditional Groups: Other use cases exist beyond authentication, such as selecting compression methods where each method has different configuration options.

Checklist

• I have added test packages to test/packages that prove my change is effective.
• I have added an entry in spec/changelog.yml.

Related issues

• relates [Change Proposal] Add var_groups to support Cloud Connector integration with Fleet #1054
• relates [Cloud-Security] Adding var-groups to support authentication vars - related to cloud connector feature integrations#16985
• relates [Cloud Security] [Fleet] Add var_group components and validation kibana#249449
• relates https://github.com/elastic/security-team/issues/15346

[Copilot]

Pull request overview

This PR adds a new var_groups schema to the package manifest specification, enabling conditional variable groups that control which authentication variables are visible based on user selection. This addresses the limitation where Fleet's generic UI currently displays all integration variables in a flat list regardless of their context.

Changes:

• Introduces var_groups field with mutually exclusive options for organizing related variables
• Adds validation rules to ensure var_groups reference valid variables and maintain naming uniqueness
• Supports deployment mode filtering and input-level option hiding for advanced use cases

Reviewed changes

Copilot reviewed 30 out of 40 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
spec/integration/manifest.spec.yml	Adds var_groups schema definition with options, deployment mode filtering, and hide_in_var_group_options for inputs
spec/integration/data_stream/manifest.spec.yml	Adds var_groups support at stream level with version patch to remove from pre-3.6.0
spec/changelog.yml	Documents the var_groups enhancement for Cloud Connector integration
code/go/internal/validator/spec.go	Registers ValidateVarGroups semantic validation rule for format_version >= 3.6.0
code/go/internal/validator/semantic/validate_var_groups.go	Implements validation logic for var_groups including duplicate detection and required field checking
code/go/internal/validator/semantic/validate_var_groups_test.go	Adds comprehensive test coverage for var_groups validation scenarios
code/go/pkg/validator/validator_test.go	Registers test packages for good and bad var_groups validation cases
test/packages/good_var_groups/*	Complete test package demonstrating valid var_groups usage with Cloud Connector extensions
test/packages/bad_var_groups_/	Test packages for invalid scenarios (missing vars, duplicate names, required conflicts)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[teresaromero]

LGTM, please update branch with conflicts

[seanrathier]

/test

[mrodm]

👍

[zmoog]

The Azure Logs integration is adding new authentication options (connection string, client credentials, workload identity, and managed identity, etc) so this will be really handy!

[seanrathier]

@zmoog
This is the merged PR that add the VarGroup rendering

• [Cloud Security] [Fleet] Add var_group components and validation kibana#249449

[zmoog]

This is the merged PR that add the VarGroup rendering

Looking forward to the 9.4 release! 🚀

[teresaromero]

@seanrathier could you resolve changelog conflicts? thanks!

[seanrathier]

@teresaromero merged conflicts. Thanks for the heads up.

Are we merging the package spec 3.6.0 soon?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c57ae74

History

• 💚 Build #1761 succeeded 8ce43f7
• 💚 Build #1755 succeeded ea13144
• 💔 Build #1753 failed ea13144
• 💚 Build #1747 succeeded 6f68753
• 💔 Build #1746 failed 2594422
• 💚 Build #1743 succeeded c98a0be

cc @seanrathier