Skip to content

Latest commit

 

History

History
65 lines (47 loc) · 3.31 KB

graceful-shutdown.adoc

File metadata and controls

65 lines (47 loc) · 3.31 KB

Graceful shutdown

You can configure the graceful shutdown grace period as described in concepts:operations/graceful_shutdown.adoc.

Masters

As a default, masters have 20 minutes to shut down gracefully.

The HBase master process receives a SIGTERM signal when Kubernetes wants to terminate the Pod. After the graceful shutdown timeout runs out, and the process is still running, Kubernetes issues a SIGKILL signal.

This is equivalent to executing the bin/hbase-daemon.sh stop master command, which internally executes kill <master-pid> (code), waits for a configurable period of time (defaults to 20 minutes), and finally executes kill -9 <master-pid> to SIGKILL the master (code).

However, there is no message in the log acknowledging the graceful shutdown.

RegionServers

By default, RegionServers have 60 minutes to shut down gracefully.

They use the same mechanism described above. In contrast to the Master servers, they will, however, acknowledge the graceful shutdown with a message in the logs:

2023-10-11 12:38:05,059 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5875de6a
2023-10-11 12:38:05,060 INFO  [shutdown-hook-0] regionserver.HRegionServer: ***** STOPPING region server 'test-hbase-regionserver-default-0.test-hbase-regionserver-default.kuttl-test-topical-parakeet.svc.cluster.local,16020,1697027870348' *****

The operator allows for finer control over the shutdown process of region servers. For each region server pod, the region mover tool may be invoked before terminating the region server’s pod. The affected regions are transferred to other pods thus ensuring that the data is not lost.

Here is a an example:

spec:
  regionServers:
    config:
      regionMover
        runBeforeShutdown: true, (1)
        maxThreads: 5, (2)
        ack: false, (3)
        extraOpts: ["--designatedFile", "/path/to/designatedFile"] (4)

<1>: Run the region mover tool before shutting down the region server. Default is false. <2>: Maximum number of threads to use for moving regions. Default is 1. <3>: Enable or disable region confirmation on the present and target servers. Default is true. <4>: Extra options to pass to the region mover tool.

Note
 There is no need to explicitly specify a timeout for the region movement. The operator will compute an appropriate timeout that cannot exceed the gracefulShutdownTimeout for region servers.

RestServers

As a default, RestServers have 5 minutes to shut down gracefully.

They use the same mechanism described above. In contrast to the Master servers, they will, however, acknowledge the graceful shutdown with a message in the logs:

2023-10-11 12:40:42,309 INFO  [JettyShutdownThread] server.AbstractConnector: Stopped ServerConnector@62dae540{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-10-11 12:40:42,309 INFO  [JettyShutdownThread] server.session: node0 Stopped scavenging
2023-10-11 12:40:42,316 INFO  [main] RESTServer: ***** STOPPING service 'RESTServer' *****