fix(stability): use /api/v2/health for fluent-bit container health check - #167
Closed
TheanLim wants to merge 1 commit into
Closed
fix(stability): use /api/v2/health for fluent-bit container health check#167TheanLim wants to merge 1 commit into
TheanLim wants to merge 1 commit into
Conversation
PR aws#166 added a container health check to the two ecs-firelens-stability-tests templates using `curl -f http://127.0.0.1:2020/api/v1/health`. On Fluent Bit 5.0.x, `/api/v1/health` is gated behind the `Health_Check` feature and returns HTTP 404 when it is not enabled (the default). The stability test [SERVICE] config sets `HTTP_Server On` but not `Health_Check On`, so every task reports UNHEALTHY during 5.0.x qualification. `/api/v2/health` returns HTTP 200 on all release lines regardless of the `Health_Check` toggle, so switching to it fixes the 5.0.x line while keeping the 1.9.10 (2.x) line green. The original intent of aws#166 -- surfacing an HTTP server deadlock (fluent-bit#11769) -- is preserved: both endpoints are served by the same HTTP server thread, so a hung server fails `curl -f` either way.
Contributor
Author
|
Closing for now while we decide between two fixes: (A) switch to /api/v2/health (minimal; real health on 5.0.x, liveness-only on 1.9.10 where v2 doesn't exist and 200s via the catch-all root), vs (B) keep /api/v1/health and enable Health_Check On in the stability configs (real health on both 1.9.10 and 5.0.x, but changes semantics to output error/retry thresholds). Will reopen or refile after deciding. Branch is preserved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Change the fluent-bit container health check in the two stability templates used
by
ecs-firelens-stability-testsfrom/api/v1/healthto/api/v2/health:apps/firelens-stability/templates/golden-path-mountebank-fargate-v01-11-2023/task-definition.jsonapps/firelens-stability/templates/s3-fargate-v04-05-2023/task-definition.jsonWhy
The health check added in #166 uses
/api/v1/health. On Fluent Bit 5.0.xthis returns HTTP 404 with the stability configs (which set
HTTP_Server Onbut not
Health_Check On), so all 285 tasks report UNHEALTHY when qualifyingany 5.0.x image.
Root cause (source-level; not called out in upstream release notes):
/api/v1/healthis registered only whenHealth_Check On— in both 4.xand 5.x (
src/http_server/api/v1/register.c, guarded byhs->config->health_check == FLB_TRUE). This gating is not new.(http_server: unify Fluent Bit HTTP listeners on flb_http_server fluent/fluent-bit#11538, "http_server: unify Fluent Bit HTTP listeners on
flb_http_server", merged 2026-03-12): the root
/route is now exact-match(
flb_hs_register_endpoint(hs, "/", FLB_HS_ROUTE_EXACT, cb_root)insrc/http_server/flb_hs.c), so any unregistered path returns 404. In<= 4.x the Monkey-based server registered
/as a catch-all(
mk_vhost_handler(hs->ctx, vid, "/", cb_root, hs)), so unregistered paths —including
/api/v1/healthwhenHealth_Checkwas off — fell through to theroot handler and returned HTTP 200 (build-info). That is why the check
silently "passed" on older images.
/api/v2/healthis registered unconditionally in 5.x(
src/http_server/api/v2/register.c), independent of theHealth_Checktoggle, so it returns HTTP 200 on all lines — which is why it is the fix.
This wasn't caught earlier because:
predated it.
Add ECS health check to Fluent Bit container #166 was validated via FireLens using the
stabletag, which currentlyresolves to
aws-for-fluent-bit:2.34.3= Fluent Bit 1.9.10. On 1.9.10,/api/v1/healthreturns 200 (no dedicated health handler; any path 200s), sothe check passed there. The bug only surfaces on the 5.0.x line that
qualification actually builds and runs.
/api/v2/healthreturns HTTP 200 on every release line regardless of theHealth_Checktoggle, so it is a safe, minimal fix.Compatibility (reproduced against real images,
HTTP_Server On, noHealth_Check)/api/v1/health/api/v2/healthTesting
End-to-end on ECS Fargate with the
ecs-firelens-stability-testscollection(285 tasks), using the same
init-debug-5.0.9image for both runs — the onlydifference is the health-check endpoint:
/api/v1/health): 285 running, 285 UNHEALTHY/api/v2/health): 285 running, 285/285 HEALTHYAlso confirmed
/api/v2/health→ 200 on FLB 1.9.10 and 5.0.7 (the two lines webuild and release), so no regression for 2.x qualifications.
Notes
fails
curl -fon/api/v2/health(same server thread).Health_Check Ontoevery fluent-bit config was considered and rejected as more invasive, and it
changes health semantics to output-error thresholds).