Skip to content

Add ECS health check to Fluent Bit container - #166

Merged
prateekchaudhry merged 2 commits into
aws:mainfrom
prateekchaudhry:add-fluent-bit-healthcheck
Jun 11, 2026
Merged

Add ECS health check to Fluent Bit container#166
prateekchaudhry merged 2 commits into
aws:mainfrom
prateekchaudhry:add-fluent-bit-healthcheck

Conversation

@prateekchaudhry

Copy link
Copy Markdown
Contributor

Summary

Add ECS container health check to the fluent-bit container in the task definition templates used by the ecs-firelens-stability-tests qualification collection. We have seen issues related to http server being deadlocked - fluent/fluent-bit#11769 . While this change does not test or repro the exact same issue, it is a basic sanity check being added.

Changes

Added to golden-path-mountebank-fargate-v01-11-2023/task-definition.json and s3-fargate-v04-05-2023/task-definition.json:

"healthCheck": {
  "command": ["CMD-SHELL", "curl -f http://127.0.0.1:2020/api/v1/health || exit 1"],
  "interval": 30,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 30
}

These templates are referenced by ecs-firelens-stability-tests collection -

  • golden-path/case-config.json - golden-path-mountebank-fargate-v01-11-2023
  • s3-stability/case-config.json - s3-fargate-v04-05-2023
  • s3-sensitivity/case-config.json - s3-fargate-v04-05-2023

Testing

  • Validated end-to-end on ECS Fargate with FireLens init image + S3 config (HTTP_Server On) + health check - containers report HEALTHY

Add container health check to the fluent-bit container in the two task
definition templates used by the ecs-firelens-stability-tests
qualification collection:
- golden-path-mountebank-fargate-v01-11-2023
- s3-fargate-v04-05-2023

The health check curls the Fluent Bit HTTP server health endpoint every
30 seconds. If the HTTP server deadlocks (as seen in fluent-bit#11769),
ECS will mark the container UNHEALTHY after 3 consecutive failures
(90s), making the issue visible in the CloudWatch monitoring dashboard.

These are the only 2 templates used by the ecs-firelens-stability-tests
collection which runs ~285 tasks during qualification.
singholt
singholt previously approved these changes Jun 11, 2026
danehlim
danehlim previously approved these changes Jun 11, 2026
Only specify startPeriod (30s) since interval, timeout, and retries
match ECS defaults. startPeriod is needed to allow the init process
to download S3 configs before health checks begin.
@prateekchaudhry
prateekchaudhry dismissed stale reviews from danehlim and singholt via 13accad June 11, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants