Skip to content

chore(aws): Increase healthcheck timeout to 15 mins#8

Merged
Antonio-RiveroMartnez merged 2 commits into
mainfrom
health_check
Feb 6, 2026
Merged

chore(aws): Increase healthcheck timeout to 15 mins#8
Antonio-RiveroMartnez merged 2 commits into
mainfrom
health_check

Conversation

@Antonio-RiveroMartnez
Copy link
Copy Markdown
Contributor

@Antonio-RiveroMartnez Antonio-RiveroMartnez commented Feb 5, 2026

Problem

Showtime deployments are failing health checks even though the container is running correctly. The health check times out before the container becomes ready.

Root cause: Superset's Parquet modernization (PR #36538) merged in January 2026 changed how example data is loaded. Instead of using a pre-built DuckDB file, 23 datasets (~400k rows) are now loaded at runtime from Parquet files. Making the whole startup process take 10+ minutes, but showtime's health check timeout was only 10 minutes.

Timeline of a failed deployment
Time Event
T+0 Container starts, ECS marks service "stable"
T+0 Superset begins loading examples (blocking)
T+10min Showtime health check times out (20 attempts exhausted)
T+10+min Examples finish loading, gunicorn starts
T+10+min Container now healthy (too late)

This PR Increases max_attempts from 20 to 30 (10 min → 15 min timeout) and adds some unit tests.

Another approach we could take but it implies more complexity and connecting new aws clients, permissions etc:

  1. ECS tasks write logs to CloudWatch (typically /ecs/superset-ci/ or similar)
  2. Watch for gunicorn startup message like "Listening at: http://0.0.0.0:8088" or "Booting
    worker with pid"
  3. Only then start HTTP health checks

For example This PR: apache/superset#37694 is marked as fail but the env is actually up and working: http://34.219.183.81:8080/

  PR #37694 (SHA: b427b82)                                                                        
      ↓                                                                                           
  ECS Service: pr-37694-b427b82-service                                                           
      ↓                                                                                           
  ECS Task: de146f58caf14cafbae0b62fa8f8855c                                                      
      ↓                                                                                           
  Network Interface: eni-035b411082b95b518                                                        
      ↓                                                                                           
  Public IP: 34.219.183.81                                                                        
      ↓
  Health: OK ✅

@Antonio-RiveroMartnez Antonio-RiveroMartnez changed the title chore(aws): Increase healthcheck timeout to 20 mins chore(aws): Increase healthcheck timeout to 15 mins Feb 5, 2026
@Antonio-RiveroMartnez Antonio-RiveroMartnez merged commit 0773247 into main Feb 6, 2026
2 checks passed
@Antonio-RiveroMartnez Antonio-RiveroMartnez deleted the health_check branch February 6, 2026 14:05
@sadpandajoe sadpandajoe mentioned this pull request Apr 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants