Skip to content

Healthcheck: new simpler implementation #6155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sougou merged 39 commits into from
May 27, 2020
Merged

Healthcheck: new simpler implementation #6155

Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
8dd5132
healthcheck: rename healthcheck and associated structs to legacy
deepthi Mar 31, 2020
f543f21
healthcheck: move more code to legacy, and create stubs for new healt…
deepthi Apr 11, 2020
27ec840
healthcheck: remove unused structs, compactify new healthcheck functi…
deepthi Apr 16, 2020
b29b745
healthcheck: move tabletStats into separate file, implement cell map …
deepthi Apr 21, 2020
9e3adee
healthcheck: implement topo watch, tablet_stats_cache test, stub new …
deepthi Apr 22, 2020
d4cb214
healthcheck: remove unnecessary flag, fix some TODOs
deepthi Apr 23, 2020
21b9f74
healthcheck: simplify healthcheck struct by deleting tablet_stats_cac…
deepthi Apr 28, 2020
8e2b518
healthcheck: if cells_to_watch is empty, operate on just localCell
deepthi Apr 28, 2020
5808934
healthcheck: delete commented code
deepthi Apr 28, 2020
3d95319
healthcheck: use tabletHealth.mu to serialize access to members. hand…
deepthi Apr 28, 2020
84f21ab
healthcheck: move shuffleTablets back to gateway
deepthi Apr 28, 2020
30c6f47
healthcheck: Move topoWatch back into TopologyWatcher and use TabletR…
deepthi Apr 29, 2020
b0ce1c5
healthcheck: remove references to TabletToMapKey in new topo watcher
deepthi Apr 29, 2020
1abc4f4
healthcheck: notify buffer when failover ends, rename entries -> heal…
deepthi Apr 29, 2020
8a020f3
healthcheck: rename old tests to legacy
deepthi Apr 30, 2020
543dd23
healthcheck: move loadTablets from TopologyWatcher to HealthCheckImpl,
deepthi May 2, 2020
1c511af
healthcheck: fold healthCheckConn into TabletHealth, fix CacheStatus,…
deepthi May 5, 2020
fe9bdd4
healthcheck: rename TabletHealth->tabletHealthCheck, create a simpler…
deepthi May 6, 2020
741c012
healthcheck: HealthCheck cache in vtgate debug/status page
deepthi May 6, 2020
94e597f
healthcheck: refactor stream into smaller funcs, use channel for unit…
deepthi May 7, 2020
4b85e4b
healthcheck: ScatterConn should return the correct cache status based…
deepthi May 7, 2020
b2f8c89
healthcheck: topology_watcher unit tests
deepthi May 7, 2020
8946b06
healthcheck: fix stats and healthy, endtoend test
deepthi May 8, 2020
2b95b25
healthcheck: endtoend test for scatter_conn and buffer
deepthi May 12, 2020
3c89878
healthcheck: lock hc.mu before accessing members
deepthi May 12, 2020
08250b8
healthcheck: unit tests for GetHealthyTabletStats
deepthi May 13, 2020
00d137e
healthcheck: update protobuf generated sources, fix shadowing bug whi…
deepthi May 13, 2020
c4fab25
healthcheck: cell alias test, add new tests to config.json
deepthi May 13, 2020
97a77a4
healthcheck: cell alias unit test
deepthi May 14, 2020
e0fb77e
healthcheck: remove HealthCheck interface, rename HealthCheckImpl -> …
deepthi May 20, 2020
08deeac
healthcheck: use channel to notify buffer of new master
deepthi May 22, 2020
edd4066
healthcheck: close subscriber goroutine when healthcheck is closed
deepthi May 22, 2020
45b9b1d
healthcheck: fix endtoend setup to use correct gateway
deepthi May 23, 2020
3f172ca
addressed review comments
systay May 25, 2020
cb276c9
healthcheck: remove mu from tabletHealthCheck, replace more uses of t…
deepthi May 25, 2020
f56e82b
healthcheck: make new tabletgateway the default, deprecate discoveryg…
deepthi May 25, 2020
a49e079
healthcheck: fix vtexplain test to use legacy gateway
deepthi May 25, 2020
d9e420b
healthcheck: fix race condition by using thread-safe healthData inste…
deepthi May 25, 2020
3343c21
healthcheck: legacy_replication_lag_algorithm is used by new healthch…
deepthi May 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions go/cmd/vtcombo/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ var (

ts *topo.Server
resilientServer *srvtopo.ResilientServer
healthCheck discovery.HealthCheck
healthCheck discovery.LegacyHealthCheck
)

func init() {
Expand Down Expand Up @@ -118,7 +118,7 @@ func main() {

// vtgate configuration and init
resilientServer = srvtopo.NewResilientServer(ts, "ResilientSrvTopoServer")
healthCheck = discovery.NewHealthCheck(1*time.Millisecond /*retryDelay*/, 1*time.Hour /*healthCheckTimeout*/)
healthCheck := discovery.NewLegacyHealthCheck(1*time.Millisecond /*retryDelay*/, 1*time.Hour /*healthCheckTimeout*/)
tabletTypesToWait := []topodatapb.TabletType{
topodatapb.TabletType_MASTER,
topodatapb.TabletType_REPLICA,
Expand All @@ -128,7 +128,7 @@ func main() {
vtgate.QueryLogHandler = "/debug/vtgate/querylog"
vtgate.QueryLogzHandler = "/debug/vtgate/querylogz"
vtgate.QueryzHandler = "/debug/vtgate/queryz"
vtg := vtgate.Init(context.Background(), healthCheck, resilientServer, tpb.Cells[0], 2 /*retryCount*/, tabletTypesToWait)
vtg := vtgate.LegacyInit(context.Background(), healthCheck, resilientServer, tpb.Cells[0], 2 /*retryCount*/, tabletTypesToWait)

// vtctld configuration and init
vtctld.InitVtctld(ts)
Expand Down
2 changes: 1 addition & 1 deletion go/cmd/vtcombo/status.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ func addStatusParts(vtg *vtgate.VTGate) {
servenv.AddStatusPart("Gateway Status", vtgate.StatusTemplate, func() interface{} {
return vtg.GetGatewayCacheStatus()
})
servenv.AddStatusPart("Health Check Cache", discovery.HealthCheckTemplate, func() interface{} {
servenv.AddStatusPart("Health Check Cache", discovery.LegacyHealthCheckTemplate, func() interface{} {
return healthCheck.CacheStatus()
})
}
12 changes: 9 additions & 3 deletions go/cmd/vtgate/status.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,13 @@ func addStatusParts(vtg *vtgate.VTGate) {
servenv.AddStatusPart("Gateway Status", vtgate.StatusTemplate, func() interface{} {
return vtg.GetGatewayCacheStatus()
})
servenv.AddStatusPart("Health Check Cache", discovery.HealthCheckTemplate, func() interface{} {
return healthCheck.CacheStatus()
})
if *vtgate.GatewayImplementation == vtgate.GatewayImplementationDiscovery {
servenv.AddStatusPart("Health Check Cache", discovery.LegacyHealthCheckTemplate, func() interface{} {
return legacyHealthCheck.CacheStatus()
})
} else {
servenv.AddStatusPart("Health Check Cache", discovery.HealthCheckTemplate, func() interface{} {
return vtg.Gateway().HealthCheck().CacheStatus()
})
}
}
30 changes: 20 additions & 10 deletions go/cmd/vtgate/vtgate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,12 @@ import (
)

var (
cell = flag.String("cell", "test_nj", "cell to use")
retryCount = flag.Int("retry-count", 2, "retry count")
healthCheckRetryDelay = flag.Duration("healthcheck_retry_delay", 2*time.Millisecond, "health check retry delay")
healthCheckTimeout = flag.Duration("healthcheck_timeout", time.Minute, "the health check timeout period")
tabletTypesToWait = flag.String("tablet_types_to_wait", "", "wait till connected for specified tablet types during Gateway initialization")
cell = flag.String("cell", "test_nj", "cell to use")
tabletTypesToWait = flag.String("tablet_types_to_wait", "", "wait till connected for specified tablet types during Gateway initialization")
)

var resilientServer *srvtopo.ResilientServer
var healthCheck discovery.HealthCheck
var legacyHealthCheck discovery.LegacyHealthCheck

func init() {
rand.Seed(time.Now().UnixNano())
Expand All @@ -63,9 +60,6 @@ func main() {

resilientServer = srvtopo.NewResilientServer(ts, "ResilientSrvTopoServer")

healthCheck = discovery.NewHealthCheck(*healthCheckRetryDelay, *healthCheckTimeout)
healthCheck.RegisterStats()

tabletTypes := make([]topodatapb.TabletType, 0, 1)
if len(*tabletTypesToWait) != 0 {
for _, ttStr := range strings.Split(*tabletTypesToWait, ",") {
Expand All @@ -78,12 +72,28 @@ func main() {
}
}

vtg := vtgate.Init(context.Background(), healthCheck, resilientServer, *cell, *retryCount, tabletTypes)
var vtg *vtgate.VTGate
if *vtgate.GatewayImplementation == vtgate.GatewayImplementationDiscovery {
// default value
legacyHealthCheck = discovery.NewLegacyHealthCheck(*vtgate.HealthCheckRetryDelay, *vtgate.HealthCheckTimeout)
legacyHealthCheck.RegisterStats()

vtg = vtgate.LegacyInit(context.Background(), legacyHealthCheck, resilientServer, *cell, *vtgate.RetryCount, tabletTypes)
} else {
// use new Init otherwise
vtg = vtgate.Init(context.Background(), resilientServer, *cell, tabletTypes)
}

servenv.OnRun(func() {
// Flags are parsed now. Parse the template using the actual flag value and overwrite the current template.
discovery.ParseTabletURLTemplateFromFlag()
addStatusParts(vtg)
})
servenv.OnClose(func() {
_ = vtg.Gateway().Close(context.Background())
if legacyHealthCheck != nil {
_ = legacyHealthCheck.Close()
}
Comment on lines +93 to +96

@harshit-gangal harshit-gangal May 25, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can change the function call
"_ = vtg.Gateway().Close(context.Background())" -> "vtg.Gateway().Close(context.Background())"
"_ = legacyHealthCheck.Close()" -> "legacyHealthCheck.Close()"

@deepthi deepthi May 25, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignoring the return value produces warnings.

@harshit-gangal harshit-gangal May 26, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is still ignored :)

@systay systay May 26, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally think it's clearer using the underscore to mark that we are aware of the returned value, but we are ignoring it. Makes it clear that it's not by mistake

})
servenv.RunDefault()
}
Loading