out_s3: blob: fix handling of pre-signed URLs #11246

edsiper · 2025-12-02T23:10:57Z

When using pre-signed URLs for blob uploads via authorization_endpoint_url), the plugin didn't extract or use the host from the pre-signed URL. It treated the URL as a URI path, so requests went to the wrong host or failed.

This patch added s3_parse_presigned_url() to parse pre-signed URLs and extract host, URI, and port Updated put_blob_object(), complete_multipart_upload(), and abort_multipart_upload() to:

Extract the host from the pre-signed URL
Temporarily set ctx->s3_client->host to the extracted host
Validate the port matches the configuration
Restore the original host after the request

Now blob uploads using pre-signed URLs now correctly use the host specified in the URL instead of the default S3 client host.

Example test:

service:
  flush: 1
  log_level: debug

pipeline:
  inputs:
    - name: blob
      path: '/tmp/the_project_gutenberg_ebook.txt'
      database_file: blob.db
      upload_success_suffix: emit_log

  outputs:
    - name: s3
      match: '*'
      bucket: fluent-bit-s3-integration-blob
      region: us-east-1
      blob_database_file: s3-blob.db

      # File buffering before upload
      total_file_size: 1M
      upload_timeout: 5s

      # Blob upload settings
      upload_parts_timeout: 10s

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

Bug Fixes
- Improved presigned URL handling with enhanced parsing and validation of host, URI, and port information.
- Strengthened error recovery and resource cleanup for presigned URL operations.
- Added port validation to ensure consistency when using presigned URLs with S3.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

When using pre-signed URLs for blob uploads via authorization_endpoint_url), the plugin didn't extract or use the host from the pre-signed URL. It treated the URL as a URI path, so requests went to the wrong host or failed. This patch added s3_parse_presigned_url() to parse pre-signed URLs and extract host, URI, and port Updated put_blob_object(), complete_multipart_upload(), and abort_multipart_upload() to: - Extract the host from the pre-signed URL - Temporarily set ctx->s3_client->host to the extracted host - Validate the port matches the configuration - Restore the original host after the request Now blob uploads using pre-signed URLs now correctly use the host specified in the URL instead of the default S3 client host. Signed-off-by: Eduardo Silva <[email protected]>

coderabbitai · 2025-12-02T23:11:16Z

Walkthrough

A new presigned URL parsing helper is introduced to the S3 output plugin, enabling dynamic endpoint resolution. The function is integrated into blob upload and multipart operations, replacing the original host with the presigned endpoint during transfers and restoring it afterward. Error handling is centralized via cleanup labels to ensure consistent resource deallocation.

Changes

Cohort / File(s)	Summary
Presigned URL parsing infrastructure `plugins/out_s3/s3.h`, `plugins/out_s3/s3.c`	Added new `s3_parse_presigned_url()` helper function that parses presigned URLs to extract host, URI, and port. Function validates inputs, applies scheme-based default ports, and returns allocated strings.
Put blob object integration `plugins/out_s3/s3.c`	Modified `put_blob_object()` to use presigned URL parsing, temporarily swapping the S3 client host with the presigned host, validating port consistency, and restoring original state. Converted early returns to centralized cleanup paths.
Multipart operations integration `plugins/out_s3/s3_multipart.c`	Extended presigned URL handling to multipart upload operations with host swapping and validation. Refactored error paths to use single cleanup flow, added result variable for standardized return handling, and ensured resource cleanup across all error conditions.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Resource cleanup consistency: Verify that all error paths properly restore original host, deallocate presigned host, URI, and other resources without leaks.
Port validation logic: Ensure port consistency checks between presigned and client configuration are correct and handle edge cases (mismatches, default ports, scheme-based defaults).
State management: Review host-swapping mechanism in both blob and multipart operations; ensure race conditions are not introduced and original state is restored even on partial failures.
Error propagation: Confirm that goto cleanup paths are correctly placed and all control flow branches (success/failure) reach appropriate cleanup points.

Suggested labels

docs-required, backport to v4.1.x

Poem

🐰 A presigned hop, a URL parse,
Endpoints swapped with graceful care,
Host and port in balance dance,
S3 buckets everywhere!
Cleanup labels guard the way,
Resources freed, no leaks to fray.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'out_s3: blob: fix handling of pre-signed URLs' accurately and clearly summarizes the main change: fixing pre-signed URL handling in the S3 output plugin for blob operations.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch s3-blob-fix

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

plugins/out_s3/s3.c (1)
2590-2606: Consider extracting common presigned URL host-swap pattern.

The presigned URL handling pattern (parse → validate port → swap host → cleanup) is repeated across five functions (put_blob_object, complete_multipart_upload, abort_multipart_upload, create_multipart_upload, upload_part).

A helper struct + setup/teardown functions could reduce duplication:
struct presigned_context {
    flb_sds_t host;
    flb_sds_t uri;
    const char *original_host;
    int port;
};

int presigned_setup(struct flb_s3 *ctx, const char *url, struct presigned_context *pc);
void presigned_cleanup(struct flb_s3 *ctx, struct presigned_context *pc);
This is optional and can be deferred if the current approach is preferred for clarity.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a8f1e4b and 1b478a2.

📒 Files selected for processing (3)

plugins/out_s3/s3.c (7 hunks)
plugins/out_s3/s3.h (1 hunks)
plugins/out_s3/s3_multipart.c (15 hunks)

🧰 Additional context used

🧬 Code graph analysis (3)

plugins/out_s3/s3.h (1)

plugins/out_s3/s3.c (1)

s3_parse_presigned_url (2208-2284)

plugins/out_s3/s3_multipart.c (1)

plugins/out_s3/s3.c (1)

s3_parse_presigned_url (2208-2284)

plugins/out_s3/s3.c (3)

src/flb_utils.c (1)

flb_utils_url_split (1441-1534)

src/flb_sds.c (2)

flb_sds_create (78-90)

flb_sds_destroy (389-399)

include/fluent-bit/flb_mem.h (1)

flb_free (126-128)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (31)

GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
GitHub Check: PR - fuzzing test
GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
GitHub Check: pr-compile-without-cxx (3.31.6)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-centos-7
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)

🔇 Additional comments (8)

plugins/out_s3/s3.h (1)

221-224: LGTM!

The function declaration is well-structured with appropriate parameters for parsing presigned URLs. The signature matches the implementation in s3.c and follows the existing coding conventions in this header.

plugins/out_s3/s3_multipart.c (4)

405-516: LGTM - Cleanup flow is well-structured.

The presigned URL handling and cleanup logic is correctly implemented:

Resources (body, presigned_host, uri) are properly freed in all paths

original_host restoration is guarded by NULL check

Early returns before resource allocation don't leak memory

518-615: LGTM!

The abort_multipart_upload function follows the same consistent pattern as complete_multipart_upload. Resource cleanup is properly handled via the abort_cleanup label.

617-723: LGTM!

The create_multipart_upload function correctly places the cleanup section before response handling. This ensures that the host is restored and resources are freed regardless of the response outcome, with subsequent returns exiting cleanly.

769-910: LGTM!

The upload_part function follows the established pattern correctly. Headers are freed immediately after the request (lines 847-850), and the cleanup section properly handles host restoration and URI cleanup.

plugins/out_s3/s3.c (3)

2208-2284: LGTM - Robust URL parsing with proper cleanup.

The implementation correctly:

Validates required parameters upfront

Handles port resolution from explicit port or scheme defaults

Creates owned flb_sds_t copies for output

Frees all temporary allocations in both success and error paths

2583-2607: LGTM - Presigned URL handling is correctly integrated.

The presigned path properly:

Fetches the full presigned URL and immediately parses it

Destroys the temporary presigned_full after parsing to avoid leaks

Validates port consistency before swapping the host

Cleans up presigned_host and uri on port validation failure

2660-2670: LGTM - Cleanup section is well-designed.

The cleanup label consolidates all resource deallocation and state restoration, ensuring consistent behavior across all exit paths (success, failure, and early error conditions).

edsiper · 2025-12-02T23:15:39Z

Ping @singholt @sparrc (pls if you can take a look at these changes)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-12-02T23:17:47Z

plugins/out_s3/s3_multipart.c

+        original_host = ctx->s3_client->host;
+        ctx->s3_client->host = presigned_host;


Pre-signed host never used for network connection

The new pre-signed flow swaps ctx->s3_client->host to the parsed host before issuing the request (plugins/out_s3/s3_multipart.c:440-441), but the HTTP connection still comes from ctx->s3_client->upstream, which is created once with the configured endpoint (plugins/out_s3/s3.c:984-989) and reused in request_do (src/aws/flb_aws_util.c:342-376). When the pre-signed URL points at a different host—the case this change is meant to fix—the TCP/TLS connection (and SNI) stay bound to the original endpoint, so the request is still sent to the wrong server even though the Host header is rewritten. The upstream needs to be rebuilt or the connection opened against the pre-signed host for these URLs to work.

Useful? React with 👍 / 👎.

edsiper requested a review from a team as a code owner December 2, 2025 23:10

github-actions bot added the docs-required label Dec 2, 2025

edsiper temporarily deployed to pr December 2, 2025 23:11 — with GitHub Actions Inactive

edsiper assigned edsiper and unassigned edsiper Dec 2, 2025

coderabbitai bot reviewed Dec 2, 2025

View reviewed changes

edsiper added this to the Fluent Bit v4.2.1 milestone Dec 2, 2025

chatgpt-codex-connector bot reviewed Dec 2, 2025

View reviewed changes

edsiper temporarily deployed to pr December 2, 2025 23:29 — with GitHub Actions Inactive

edsiper temporarily deployed to pr December 2, 2025 23:30 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

out_s3: blob: fix handling of pre-signed URLs #11246

out_s3: blob: fix handling of pre-signed URLs #11246

Uh oh!

edsiper commented Dec 2, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Dec 2, 2025 •

edited

Loading

Uh oh!

coderabbitai bot left a comment

Uh oh!

edsiper commented Dec 2, 2025 •

edited

Loading

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Dec 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		original_host = ctx->s3_client->host;
		ctx->s3_client->host = presigned_host;

out_s3: blob: fix handling of pre-signed URLs #11246

Are you sure you want to change the base?

out_s3: blob: fix handling of pre-signed URLs #11246

Uh oh!

Conversation

edsiper commented Dec 2, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Poem

Pre-merge checks and finishing touches

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

edsiper commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edsiper commented Dec 2, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Dec 2, 2025 •

edited

Loading

edsiper commented Dec 2, 2025 •

edited

Loading