out_gcs: support application default credentials

[balys]

Summary

The native GCS output currently supports an explicit
google_service_credentials file and the legacy
GOOGLE_SERVICE_CREDENTIALS environment variable. It does not check Google's
standard GOOGLE_APPLICATION_CREDENTIALS variable and fails initialization when
no credentials file is configured, preventing the plugin from using an attached
GCE service account or GKE Workload Identity Federation.

This change adds the relevant Application Default Credentials behavior to
out_gcs:

1. explicit google_service_credentials configuration;
2. GOOGLE_APPLICATION_CREDENTIALS;
3. legacy GOOGLE_SERVICE_CREDENTIALS; and
4. the GCE/GKE metadata server when no credentials file is configured.

Metadata tokens are retrieved from the default service-account token endpoint,
parsed through Fluent Bit's existing OAuth2 cache, and refreshed before expiry.
The metadata server URL is configurable for testing and non-default
environments.

Addresses #1032 and #9022.

Related work

PR #11758 adds external_account STS token exchange to out_stackdriver for
workloads outside Google Cloud. This PR is complementary: it adds standard ADC
environment lookup and attached GCE/GKE metadata credentials to out_gcs. It
does not add external_account credential-file support.

Compatibility

Existing explicit google_service_credentials configurations are unchanged.
GOOGLE_SERVICE_CREDENTIALS remains supported for backwards compatibility. If
both environment variables are set, GOOGLE_APPLICATION_CREDENTIALS takes
precedence and a warning is logged.

GOOGLE_APPLICATION_CREDENTIALS uses the service-account JSON format already
supported by out_gcs. Other ADC file types, such as authorized_user and
external_account, are outside this change.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

GKE Workload Identity or an attached GCE service account requires no credential
property:

[INPUT]
    Name  dummy
    Tag   gcs.test

[OUTPUT]
    Name                    gcs
    Match                   *
    bucket                  my-log-bucket
    gcs_key_format          logs/$TAG/%Y/%m/%d/%H/%M/%S-$UUID
    store_dir               /var/lib/fluent-bit/gcs
    upload_timeout          1m
    compression             gzip
    preserve_data_ordering  On

A service-account file can use standard ADC:

export GOOGLE_APPLICATION_CREDENTIALS=/var/run/secrets/google/credentials.json

• Debug log output from testing the change

[ info] [output:gcs:gcs.0] using GCE/GKE metadata server authentication
[ info] [output:gcs:gcs.0] worker #0 started

Focused runtime test:

1/1 Test #8: flb-rt-out_gcs ... Passed
100% tests passed, 0 tests failed out of 1

The runtime suite covers standard ADC precedence, successful metadata token
retrieval and upload, and rejection of an invalid metadata response without an
upload attempt.

• Attached Valgrind output that shows no leaks or memory corruption was found

SUCCESS: All unit tests have passed.
HEAP SUMMARY:
    in use at exit: 0 bytes in 0 blocks
    total heap usage: 36,980 allocs, 36,980 frees, 9,894,221 bytes allocated
All heap blocks were freed -- no leaks are possible
ERROR SUMMARY: 0 errors from 0 contexts

Commands:

cmake --build build --target flb-rt-out_gcs -j8
ctest --test-dir build -R '^flb-rt-out_gcs$' --output-on-failure
valgrind --leak-check=full \
  --show-leak-kinds=definite,indirect \
  --errors-for-leak-kinds=definite,indirect \
  --error-exitcode=99 build/bin/flb-rt-out_gcs

If this is a change to packaging of containers or native binaries then please
confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build. [N/A: no packaging changes]
• Set ok-package-test label to test for all targets (requires maintainer to do). [N/A: no packaging changes]

Documentation

• Documentation required for this feature

A follow-up documentation update should describe the credential precedence and
keyless GCE/GKE configuration. The code config-map description includes the new
metadata_server option.

Backporting

• Backport to latest stable release. [N/A: target master for the next release]

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I
understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Added Google Cloud metadata-server authentication for GCS deployments running on GCE or GKE without service-account credential files.
  • Added the metadata_server configuration option.
  • Added validation for metadata-server token responses.

• Bug Fixes

  • Credential configuration now prefers GOOGLE_APPLICATION_CREDENTIALS over the legacy environment variable and warns when both are set.
  • Retained fallback to service-account credentials when metadata authentication is unavailable or disabled.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1266c29e-8f84-4f6c-a6e2-373c8efc08ed

📥 Commits

Reviewing files that changed from the base of the PR and between ae51533 and 3b74a08.

📒 Files selected for processing (3)

• plugins/out_gcs/gcs.c
• plugins/out_gcs/gcs.h
• tests/runtime/out_gcs.c

---

📝 Walkthrough

Walkthrough

The GCS output plugin adds GCE/GKE metadata-server authentication, credential environment-variable precedence, configurable metadata-server settings, upstream lifecycle handling, and runtime tests for valid and invalid token responses.

Changes

GCS metadata authentication

Layer / File(s)	Summary
Metadata contract and credential initialization plugins/out_gcs/gcs.h, plugins/out_gcs/gcs.c	Adds metadata-server fields and constants. Prefers GOOGLE_APPLICATION_CREDENTIALS, falls back to GOOGLE_SERVICE_CREDENTIALS, and enables metadata authentication when no credential file is configured.
Metadata token flow and upstream lifecycle plugins/out_gcs/gcs.c	Fetches and validates metadata tokens, uses them for OAuth requests and uploads, creates the metadata upstream, and destroys it during cleanup.
Credential and metadata authentication tests tests/runtime/out_gcs.c	Adds coverage for credential precedence, successful metadata authentication, invalid metadata responses, request tracking, and test registration.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GCSPlugin
  participant MetadataServer
  participant GCSUploadEndpoint
  GCSPlugin->>MetadataServer: Request service-account token
  MetadataServer-->>GCSPlugin: Return token and expiry
  GCSPlugin->>GCSUploadEndpoint: Upload with bearer token

Loading

Possibly related PRs

• fluent/fluent-bit#11792: Introduced the out_gcs plugin that this change extends.

Suggested reviewers: edsiper

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the pull request's primary change: Application Default Credentials support for the GCS output plugin.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[edsiper]

@JeffLuoo is this ok from your side ?