[feature request] Add ability to check consul service health

[peimanja]

Not all of our ASG are behind ALBs, and EC2 health check not really showing us much in terms of health of the application.

It would be neat to add an option to specify the Consul service of your application running on that ASG and use that to determine if Bouncer should kill the old instances or not.

[holtwilkins]

Hey @peimanja , can you explain a bit more how you're using bouncer? Note that bouncer currently only uses ASGs launch configs / launch templates to determine if the AMI in question is out of date, not whether it's healthy. Are you currently using bouncer to automatically recycle nodes which are unhealthy?

[peimanja]

Hey @peimanja , can you explain a bit more how you're using bouncer? Note that bouncer currently only uses ASGs launch configs / launch templates to determine if the AMI in question is out of date, not whether it's healthy. Are you currently using bouncer to automatically recycle nodes which are unhealthy?

Sorry I should have been more clear. We use bouncer to perform rolling updates in case of launch templates change. I was wondering if in Canary mode we could use Consul health checks before terminating the old instances.

for example:
We have an ASG with 3 instances running and each instance has a service registered with Consul. AMI gets updated and Bouncer starts spinning up three new instances. Before terminating the old instances, it would check if six healthy Consul service is running.

[holtwilkins]

Ah, I follow. So, I think the confusion now is around EC2 health checks & ALBs. An ALB shouldn't be required from a health perspective, instead you can add an EC2_INSTANCE_LAUNCHING lifecycle hook as required to your instance.

It will be a little different on your instance's side, but https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroupLifecycle.html is the AWS page laying out how to use lifecycle hooks. If you add the launching hooks to your instances, they'll sit in Pending:Wait until the instance (or something) responds to the hook saying that instance is online. Given your use-case I think there are two main choices for you

• The instances themselves could answer the hook. As part of their userdata script & IAM profile, they will have to ability to inspect their own services or Consul healthchecks, and only respond with CONTINUE to the hook once they're considered "good". At this point, they will be placed InService in the ASG, and bouncer will see them as such as well because of this (only at this point will they be put into service in ALBs, if any, as well).
• Some other service is watching Consul to answer on nodes' behalf. The nodes could alternatively join Consul as they do without code changes to them, and instead you could have an external watchdog job that will answer hooks with CONTINUE on nodes' behalf once they become green in Consul.

So yeah, bouncer is really assuming that you're using launching lifecycle hooks at least, so that it doesn't start tearing-down old nodes before your new ones are up.

Make sense?

[peimanja]

@holtwilkins That's exactly what I was looking for. Thanks!