Skip to content

fix(stability): use /api/v2/health for fluent-bit container health check - #167

Closed
TheanLim wants to merge 1 commit into
aws:mainfrom
TheanLim:fix/stability-healthcheck-api-v2
Closed

fix(stability): use /api/v2/health for fluent-bit container health check#167
TheanLim wants to merge 1 commit into
aws:mainfrom
TheanLim:fix/stability-healthcheck-api-v2

Conversation

@TheanLim

@TheanLim TheanLim commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What

Change the fluent-bit container health check in the two stability templates used
by ecs-firelens-stability-tests from /api/v1/health to /api/v2/health:

  • apps/firelens-stability/templates/golden-path-mountebank-fargate-v01-11-2023/task-definition.json
  • apps/firelens-stability/templates/s3-fargate-v04-05-2023/task-definition.json

Why

The health check added in #166 uses /api/v1/health. On Fluent Bit 5.0.x
this returns HTTP 404 with the stability configs (which set HTTP_Server On
but not Health_Check On), so all 285 tasks report UNHEALTHY when qualifying
any 5.0.x image.

Root cause (source-level; not called out in upstream release notes):

  1. /api/v1/health is registered only when Health_Check On — in both 4.x
    and 5.x (src/http_server/api/v1/register.c, guarded by
    hs->config->health_check == FLB_TRUE). This gating is not new.
  2. What changed in 5.x is the HTTP server rewrite
    (http_server: unify Fluent Bit HTTP listeners on flb_http_server fluent/fluent-bit#11538, "http_server: unify Fluent Bit HTTP listeners on
    flb_http_server", merged 2026-03-12): the root / route is now exact-match
    (flb_hs_register_endpoint(hs, "/", FLB_HS_ROUTE_EXACT, cb_root) in
    src/http_server/flb_hs.c), so any unregistered path returns 404. In
    <= 4.x the Monkey-based server registered / as a catch-all
    (mk_vhost_handler(hs->ctx, vid, "/", cb_root, hs)), so unregistered paths —
    including /api/v1/health when Health_Check was off — fell through to the
    root handler and returned HTTP 200 (build-info). That is why the check
    silently "passed" on older images.
  3. /api/v2/health is registered unconditionally in 5.x
    (src/http_server/api/v2/register.c), independent of the Health_Check
    toggle, so it returns HTTP 200 on all lines — which is why it is the fix.

This wasn't caught earlier because:

  • The health check was only added in Add ECS health check to Fluent Bit container #166 (2026-06-11); older 5.0.5/5.0.7 runs
    predated it.
  • Add ECS health check to Fluent Bit container #166 was validated via FireLens using the stable tag, which currently
    resolves to aws-for-fluent-bit:2.34.3 = Fluent Bit 1.9.10. On 1.9.10,
    /api/v1/health returns 200 (no dedicated health handler; any path 200s), so
    the check passed there. The bug only surfaces on the 5.0.x line that
    qualification actually builds and runs.

/api/v2/health returns HTTP 200 on every release line regardless of the
Health_Check toggle, so it is a safe, minimal fix.

Compatibility (reproduced against real images, HTTP_Server On, no Health_Check)

Fluent Bit /api/v1/health /api/v2/health
1.9.10 200 200
4.2.2 200 200
5.0.5 404 200
5.0.7 404 200
5.0.9 404 200

Testing

End-to-end on ECS Fargate with the ecs-firelens-stability-tests collection
(285 tasks), using the same init-debug-5.0.9 image for both runs — the only
difference is the health-check endpoint:

  • Before (/api/v1/health): 285 running, 285 UNHEALTHY
  • After (/api/v2/health): 285 running, 285/285 HEALTHY

Also confirmed /api/v2/health → 200 on FLB 1.9.10 and 5.0.7 (the two lines we
build and release), so no regression for 2.x qualifications.

Notes

  • Preserves Add ECS health check to Fluent Bit container #166's intent: a deadlocked HTTP server (fluent-bit#11769) still
    fails curl -f on /api/v2/health (same server thread).
  • No config churn required (the alternative fix of adding Health_Check On to
    every fluent-bit config was considered and rejected as more invasive, and it
    changes health semantics to output-error thresholds).

PR aws#166 added a container health check to the two ecs-firelens-stability-tests
templates using `curl -f http://127.0.0.1:2020/api/v1/health`. On Fluent Bit
5.0.x, `/api/v1/health` is gated behind the `Health_Check` feature and returns
HTTP 404 when it is not enabled (the default). The stability test [SERVICE]
config sets `HTTP_Server On` but not `Health_Check On`, so every task reports
UNHEALTHY during 5.0.x qualification.

`/api/v2/health` returns HTTP 200 on all release lines regardless of the
`Health_Check` toggle, so switching to it fixes the 5.0.x line while keeping
the 1.9.10 (2.x) line green. The original intent of aws#166 -- surfacing an HTTP
server deadlock (fluent-bit#11769) -- is preserved: both endpoints are served
by the same HTTP server thread, so a hung server fails `curl -f` either way.
@TheanLim

Copy link
Copy Markdown
Contributor Author

Closing for now while we decide between two fixes: (A) switch to /api/v2/health (minimal; real health on 5.0.x, liveness-only on 1.9.10 where v2 doesn't exist and 200s via the catch-all root), vs (B) keep /api/v1/health and enable Health_Check On in the stability configs (real health on both 1.9.10 and 5.0.x, but changes semantics to output error/retry thresholds). Will reopen or refile after deciding. Branch is preserved.

@TheanLim TheanLim closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant