Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

如何自定义停机逻辑,实现优雅停机 #3665

Closed
yoboygo opened this issue Mar 27, 2024 · 11 comments
Closed

如何自定义停机逻辑,实现优雅停机 #3665

yoboygo opened this issue Mar 27, 2024 · 11 comments
Labels
area/kubernetes Tagged as kubernetes-related issue/pr area/nacos spring cloud alibaba nacos kind/discussion Mark as discussion issues/pr stale wait-for-feedback

Comments

@yoboygo
Copy link

yoboygo commented Mar 27, 2024

环境 k8s
spring-cloud-alibaba=2021.0.4.0
nacos=2.2.0
您好,我现在想实现一个优雅停机的逻辑,具体如下:

  1. 在执行停机逻辑之前,先将服务权重设置为 0
  2. 等待Client端的缓存刷新
  3. 等待已经hold的请求结束
  4. 执行停机逻辑,从nacos摘除

现在有问题的是第1步,我的实现逻辑是通过sh脚本调用nacos的API,但是NacosAPI偶发不可用(删除/data/protocol数据可恢复),导致这个方案显的不够完美,跟踪代码发现停机逻辑是在静态代码块中注册的退出钩子,但又不知改如何修改这个逻辑,您能给我点建议吗?谢谢!

@ruansheng8
Copy link
Collaborator

可以通过继承 ShutdownEndpoint ,并在优雅关机逻辑中调用 NacosAutoServiceRegistration#destroy() 来注销实例

@ruansheng8
Copy link
Collaborator

ruansheng8 commented Mar 28, 2024

上面的逻辑中,设置权重为0实际上不需要,因为这会导致客户端刷新2次服务列表:权重变更时会收到实例变化通知,实例摘除时会再收到一次,在实际应用中,直接注销实例,等注销成功后再延迟几秒关机,就能实现应用上的无感发布了。

同时还需要使用 Nacos 的 Subscriber<InstancesChangeEvent> 来订阅实例变更通知,保证实时刷新 OpenFeignSpring Cloud Gateway 等相关框架的内部例缓存列表

@yoboygo
Copy link
Author

yoboygo commented Mar 28, 2024

感谢您的回复!请问ShutdownEndpoint在哪里,没找到 T_T,还有就是我看com.alibaba.nacos.common.http.HttpClientBeanHolder#shutdown()似乎是关闭HTTP请求的,这个在静态代码块里注册到了退出的钩子里,钩子又是顺序启动,并发执行的,实现了这个ShutdownEndpoint是否能阻止这个钩子执行?不然已经hold的HTTP请求就会被强制关闭

@ruansheng8
Copy link
Collaborator

ruansheng8 commented Mar 28, 2024

感谢您的回复!请问ShutdownEndpoint在哪里,没找到 T_T,还有就是我看com.alibaba.nacos.common.http.HttpClientBeanHolder#shutdown()似乎是关闭HTTP请求的,这个在静态代码块里注册到了退出的钩子里,钩子又是顺序启动,并发执行的,实现了这个ShutdownEndpoint是否能阻止这个钩子执行?不然已经hold的HTTP请求就会被强制关闭

ShutdownEndpoint 是 Spring Boot Actuator 中的优雅关机端点,只需要在这个端点的逻辑中调用 NacosAutoServiceRegistration#destroy() 来销毁当前实例即可

@yoboygo
Copy link
Author

yoboygo commented Mar 28, 2024

哦哦哦 我明白您的意思了!Pod在接收到停机信号之后,不能直接把停机信号转给JVM进程,而是要通过ShutdownEndpoint实现停机!

@ruansheng8
Copy link
Collaborator

哦哦哦 我明白您的意思了!Pod在接收到停机信号之后,不能直接把停机信号转给JVM进程,而是要通过ShutdownEndpoint实现停机!

是的,因为这个过程中,需要处理已经进来的请求,如果直接停止,那么已经进来的请求未处理完成就被强制中断,在关机前还需要调用所有 DisposableBean 对象的 destroy() 方法来释放资源(这个步骤actuator默认的ShutdownEndpoint 已经实现好了),

@yuluo-yx yuluo-yx added kind/discussion Mark as discussion issues/pr area/nacos spring cloud alibaba nacos area/kubernetes Tagged as kubernetes-related issue/pr labels Mar 28, 2024
@yoboygo
Copy link
Author

yoboygo commented Mar 28, 2024

好的,非常感谢!按照这个思路我再调整下我的方案,非常感谢!

@1095071913
Copy link

其实可以通过容错机制解决,当消费者缓存的Nacos注册表还没同步好,会调用到已经关闭的pod,这时候会返回调用失败,那么就容错轮询到下一个节点处理这一次请求

@ruansheng8
Copy link
Collaborator

其实可以通过容错机制解决,当消费者缓存的Nacos注册表还没同步好,会调用到已经关闭的pod,这时候会返回调用失败,那么就容错轮询到下一个节点处理这一次请求

如果对于可靠性要求极高的情况下,通常需要同时配合其他机制来进一步保证稳定性。

例如:当前服务实例下线时,基于消息订阅等相关方式将当前下线的实例通知给其他服务,然后其他服务将“下线的服务实例”缓存到当前服务的“黑名单”中,然后在负载均衡选择实例的时候过滤掉“黑名单”中的实例,将过滤后的实例列表继续传给下一层负载均衡策略(灰度、容错等相关策略)

Copy link

github-actions bot commented May 1, 2024

This issue has been open 30 days with no activity. This will be closed in 7 days.

@github-actions github-actions bot added the stale label May 1, 2024
Copy link

github-actions bot commented May 9, 2024

This issue has been automatically marked as stale because it hasn't had any recent activity.If you think this should still be open, or the problem still persists, just pop a reply in the comments and one of the maintainers will (try!) to follow up. Thank you for your interest and contribution to the Sping Cloud Alibaba Community.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubernetes Tagged as kubernetes-related issue/pr area/nacos spring cloud alibaba nacos kind/discussion Mark as discussion issues/pr stale wait-for-feedback
Projects
None yet
Development

No branches or pull requests

4 participants