Skip to content

Enable HTTP debugging for AWS client requests #11015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
+17 −0
Open

Enable HTTP debugging for AWS client requests #11015

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3863eea
Enable HTTP debugging for AWS client requests
runderwo Oct 13, 2025
7ac03a2
Apply code quality changes
runderwo Oct 14, 2025
62f98ed
Enable HTTP debug in Firehose client for illustration
runderwo Oct 14, 2025
e27feac
Code quality fixes
runderwo Oct 14, 2025
909b105
Code quality fixes
runderwo Oct 14, 2025
4daa282
Remove fixme
runderwo Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions include/fluent-bit/flb_aws_util.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@ struct flb_aws_client {

/* Send all log messages as debug; used in AWS Cred Providers on init */
int debug_only;

/* Callbacks context */
struct flb_callback *http_cb_ctx;
};

/* frees dynamic_headers */
Expand Down
5 changes: 5 additions & 0 deletions plugins/out_kinesis_firehose/firehose.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
#include <fluent-bit/flb_aws_util.h>
#include <fluent-bit/flb_mem.h>
#include <fluent-bit/flb_http_client.h>
#include <fluent-bit/flb_http_client_debug.h>
#include <fluent-bit/flb_utils.h>

#include <fluent-bit/aws/flb_aws_compress.h>
Expand Down Expand Up @@ -288,6 +289,10 @@ static int cb_firehose_init(struct flb_output_instance *ins,
ctx->firehose_client->proxy = NULL;
ctx->firehose_client->static_headers = &content_type_header;
ctx->firehose_client->static_headers_len = 1;
if (flb_http_client_debug_setup(ctx->firehose_client->http_cb_ctx, &ins->properties) < 0) {
Comment on lines 291 to +292
Copy link
Contributor

@cosmo0920 cosmo0920 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flb_http_client_debug_setup is not always existing in Fluent Bit core.
This is because only if enabled for -DFLB_HTTP_CLIENT_DEBUG=ON on configure.
So, we need to wrap up with

#ifdef FLB_HAVE_HTTP_CLIENT_DEBUG
    if (flb_http_client_debug_setup(ctx->firehose_client->http_cb_ctx, &ins->properties) < 0) {
        flb_plg_error(ctx->ins, "AWS HTTP client debug initialization error");
        goto error;
    }
#endif

here.

flb_plg_error(ctx->ins, "AWS HTTP client debug initialization error");
goto error;
}

struct flb_upstream *upstream = flb_upstream_create(config, ctx->endpoint,
ctx->port, FLB_IO_TLS,
Expand Down
9 changes: 9 additions & 0 deletions src/aws/flb_aws_util.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,12 @@ struct flb_aws_client *flb_aws_client_create()
client->client_vtable = &client_vtable;
client->retry_requests = FLB_FALSE;
client->debug_only = FLB_FALSE;
client->http_cb_ctx = flb_callback_create("aws client"); // FIXME: what name?
if (!client->http_cb_ctx) {
flb_errno();
flb_free(client);
return NULL;
}
return client;
}

Expand All @@ -291,6 +297,7 @@ void flb_aws_client_destroy(struct flb_aws_client *aws_client)
if (aws_client->extra_user_agent) {
flb_sds_destroy(aws_client->extra_user_agent);
}
flb_callback_destroy(aws_client->http_cb_ctx);
flb_free(aws_client);
Comment on lines 297 to 301
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Guard destroying AWS client HTTP callback context

When tearing down an AWS client the code now calls flb_callback_destroy(aws_client->http_cb_ctx) unconditionally. Several call sites (e.g. the credential provider unit tests’ custom client generators) allocate struct flb_aws_client via flb_calloc and never populate http_cb_ctx, yet still invoke flb_aws_client_destroy. Because flb_callback_destroy dereferences its argument, passing a NULL pointer will segfault during test runs or in any external code that injects a lightweight mock client. A null check before destroying the callback context avoids the crash while preserving existing behaviour.

Useful? React with 👍 / 👎.

Copy link
Contributor

@cosmo0920 cosmo0920 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@runderwo We need to wrap up with existence checking due to there's a possibility to be absent for registering aws_client->http_cb_ctx callbacks. This is because this type of register couldn't be caught up on dry_run or unit testing. So, we need to use a guard like a pessimistic way.

}
}
Expand Down Expand Up @@ -386,6 +393,8 @@ struct flb_http_client *request_do(struct flb_aws_client *aws_client,
goto error;
}

c->cb_ctx = aws_client->http_cb_ctx;

/* Increase the maximum HTTP response buffer size to fit large responses from AWS services */
ret = flb_http_buffer_size(c, FLB_MAX_AWS_RESP_BUFFER_SIZE);
if (ret != 0) {
Expand Down
Loading