Request context canceled during graceful shutdown

[LinEvil]

What happened:

1. Send a http request, it is slow and takes 5s to return.
2. Terminate the program by Ctrl+C before the request returns.
3. Graceful shutdown and the context of the request is canceled.
4. Throwing context canceled error and exit.

Logs:

INFO msg=[HTTP] server listening on: 127.0.0.1:8000
DEBUG msg=I'm going to sleep for 5 seconds.
INFO msg=[HTTP] server stopping
DEBUG msg=Time's up. Now to check my context.
DEBUG msg=context canceled
INFO msg=watcher's ctx cancel : context canceled

What you expected to happen:

The context of the request should not be canceled, and any request before executing a graceful shutdown should be processed normally unless the request time reaches http.timeout or shutdown time reaches stopTimeout.

How to reproduce it (as minimally and precisely as possible):

# configs/config.yaml
server:
  http:
    addr: 127.0.0.1:8000
    timeout: 8s

// internal/service/greeter.go
func (s *GreeterService) SayHello(ctx context.Context, in *v1.HelloRequest) (*v1.HelloReply, error) {
	s.log.WithContext(ctx).Infof("SayHello Received: %v", in.GetName())
	s.log.WithContext(ctx).Debug("I'm going to sleep for 5 seconds.")
	time.Sleep(5 * time.Second)	// press Ctrl+C
	s.log.WithContext(ctx).Debug("Time's up. Now to check my context.")
	if ctx.Err() == context.Canceled {
		s.log.WithContext(ctx).Debug("context canceled")
		return nil, ctx.Err()
	}
	return &v1.HelloReply{Message: "Hello " + in.GetName()}, nil
}

Anything else we need to know?:

The issue is similar to #1856, which was fixed in v2.2.1 (#1873 #1895).
My case works fine between v2.2.1 and v2.5.2

Since v2.5.3, the feature #2403 changed srv.Start(context), which may cause the current issue.
I tried rolling back that line and the problem was fixed, but I'm not sure if this is right.

Environment:

• Kratos version (use kratos -v): v2.7.1
• Go version (use go version): go1.21.4
• OS (e.g: cat /etc/os-release): Windows
• Others:

[dosubot]

🤖

Hello @LinEvil,

I'm Dosu, an AI bot here to assist you with Kratos. While we wait for a human maintainer, I'll do my best to help you solve bugs, answer questions, and guide you on your journey as a contributor. Let's tackle this together!

Thank you for the detailed report. I understand that you're experiencing an issue where the context of a request is being canceled during a graceful shutdown, even though you expect the request to be processed normally unless it reaches the http.timeout or stopTimeout.

You've correctly identified that this issue is similar to two previously solved issues in the Kratos repository: graceful shutdown err and [Question] 服务优雅退出. Both of these issues were related to the context being canceled during a graceful shutdown, and they were resolved by modifying the way the context is managed during the shutdown process.

You've also pointed out that the issue seems to have reappeared in version v2.5.3, after the introduction of the feature #2403. This feature changed the srv.Start(context) function, which is responsible for starting the gRPC server and setting the base context for the server.

Based on your description, it seems that the change to the srv.Start(context) function may be causing the context of the request to be canceled prematurely during a graceful shutdown. You mentioned that rolling back that line fixed the problem, which suggests that this could indeed be the cause of the issue.

However, without more information, it's difficult to say for certain what the best solution would be. It's possible that the change to the srv.Start(context) function was necessary for other reasons, and that the issue needs to be addressed in a different way.

I would recommend creating a new issue in the Kratos repository to report this problem. Be sure to include all the details you've provided here, as well as any additional information you think might be relevant. The maintainers of the Kratos repository will be able to help you find the best solution.

I hope this helps, and please let me know if you have any other questions.

Sources

• graceful shutdown err
• [Question] 服务优雅退出
• transport/grpc/server.go

[shenqidebaozi]

If you already have a solution, you can directly submit a PR

[LinEvil]

Need review
#3097