trigger TTL job manually may hang for a long time

[lcwangchao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

Some TTL integration test hangs and we found the goroutine is blocked with the below stack:

github.com/pingcap/tidb/ttl/client.(*etcdClient).waitCmdResponse(0xc003aacba0, {0x60cc570?, 0xc008c97f80?}, {0xc008b64f60, 0x24}, {0x4f65ca0, 0xc008ca67e0})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ttl/client/command.go:170 +0x1f4
github.com/pingcap/tidb/ttl/client.(*etcdClient).Command(0x60cc570?, {0x60cc570, 0xc008c97f80}, {0x57dfddd, 0xf}, {0x4f4faa0, 0xc00ccb5ba0}, {0x4f4fae0, 0xc00d414a98})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ttl/client/command.go:210 +0x10c
github.com/pingcap/tidb/ttl/client.TriggerNewTTLJob({0x60cc570, 0xc008c97f80}, {0x60cd728, 0xc003aacba0}, {0xc008b64f17, 0x7}, {0xc008b64f1f, 0x2})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ttl/client/command.go:106 +0x115
github.com/pingcap/tidb/server/handler/ttlhandler.TTLJobTriggerHandler.ServeHTTP({{0x60fcb40?, 0xc000036b40?}}, {0x60ca650, 0xc008cda000}, 0xc008cce300)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/handler/ttlhandler/ttl.go:62 +0x336
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004e9800, {0x60ca650, 0xc008cda000}, 0xc008cce100)
	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x1cf
net/http.(*ServeMux).ServeHTTP(0xc00ce8fa78?, {0x60ca650, 0xc008cda000}, 0xc008cce100)
	/usr/local/go/src/net/http/server.go:2500 +0x149
github.com/pingcap/tidb/server/internal/util.CorsHandler.ServeHTTP({{0x60a3dc0?, 0xc0014ad840?}, 0xc0002b3600?}, {0x60ca650?, 0xc008cda000?}, 0x52dae20?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/util/util.go:219 +0x21b
net/http.serverHandler.ServeHTTP({0xc007b4eab0?}, {0x60ca650, 0xc008cda000}, 0xc008cce100)
	/usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc007b32900, {0x60cc570, 0xc002a38330})
	/usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x5ed

It seems we use NewTimer by mistake (we should use NewTicker) here:

tidb/ttl/client/command.go

Line 164 in c512ede

ticker := time.NewTimer(time.Second)

The bug exists in several versions but in the earlier versions, this will not be triggered because the trigger action will be returned fast even max tasks limit is reached. But in the current version, trigger action will wait for the task limit condition.

This bug only affects integration tests.