Skip to content

HBASE-27226 Document native TLS support in Netty RPC #4717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bbeaudreault merged 5 commits into from
Sep 8, 2022
Merged

HBASE-27226 Document native TLS support in Netty RPC #4717

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
27088e2
HBASE-27226. Add generic TLS config + self-signed certs
anmolnar Aug 22, 2022
d53c05d
HBASE-27226. Add upgrade process and additional props
anmolnar Aug 22, 2022
994af41
HBASE-27226. Amend upgrade steps + fix some typos
anmolnar Aug 23, 2022
33c1516
HBASE-27226. Add warning block for replication, fix some other review…
anmolnar Aug 25, 2022
51785a3
HBASE-27226. Add Hadoop credential api support and filetype autodetec…
anmolnar Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
222 changes: 222 additions & 0 deletions src/main/asciidoc/_chapters/security.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -675,6 +675,228 @@ For more information about ACLs, please see the <<hbase.accesscontrol.configurat
It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPNEGO HTTP authentication.
This is future work.

== Transport Level Security (TLS) in HBase RPC communication

Since version `2.6.0` HBase supports TLS encryption in server-client and Master-RegionServer communication.
link:https://en.wikipedia.org/wiki/Transport_Layer_Security/[Transport Layer Security (TLS)] is a standard
cryptographic protocol designed to provide communications security over a computer network. HBase TLS implementation
works exactly how secure websites are accessed via *https* prefix in a web browser: once established all communication
on the channel will be securely hidden from malicious access.

The encryption works at the transport level which means it's independent of the configured authentication method. Secure
client access mentioned in the previous section requires Kerberos to be configured and used in HBase authentication, while
TLS can be configured with any other SASL mechanism or even with simple client access methods, effectively preventing
attackers from eavesdropping the communication. No Kerberos KDC or other complicated infrastructure required.

HBase TLS is based on the Netty library therefore it only works with Netty client and server RPC implementation. Netty's
powerful SSL implementation is a great foundation for highly secure and performant communication providing the latest and
greatest cryptographic solution at all times.

Since Region Servers effectively work as clients from Master's perspective, TLS supports encrypted communication
between cluster members too.

=== Server side configuration

We need to setup Java key store for the server. Key store is the list of private keys that a server can use to configure TLS
encryption. See link:https://en.wikipedia.org/wiki/Transport_Layer_Security/[TLS wikipedia page]
for further details of the protocol. Add the following configuration to `hbase-site.xml` on Master, Region Servers and HBase
clients:

[source,xml]
----
<property>
<name>hbase.server.netty.tls.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.rpc.tls.keystore.type</name>
<value>JKS</value>
</property>
<property>
<name>hbase.rpc.tls.keystore.location</name>
<value>/path/to/keystore.jks</value>
</property>
<property>
<name>hbase.rpc.tls.keystore.password</name>
<value>keyStor3pa$$w0rd</value>
Copy link
Contributor

@bbeaudreault bbeaudreault Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth calling out that the users should make sure their hbase-site.xml has appropriate permissions since we're now putting a password in it. We could leave that out, but just thinking you went into such detail on the other stuff below that maybe it makes sense to mention.

Copy link
Contributor

@ss77892 ss77892 Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say that we should avoid putting passwords in the configuration file at all costs. Possible the hdfs approach when you have the default password/password received through the environment variable/a special file that contains the password would be more suitable.

</property>
----

The supported store types are based on the registered security providers. If not specified, JKS will be used by default.

=== Client side configuration

We need to configure trust store for the client. Trust store contains the list of certificates that the client should trust
when doing the handshake with the server. Add the following to `hbase-site.xml`.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.rpc.tls.truststore.type</name>
<value>JKS</value>
</property>
<property>
<name>hbase.rpc.tls.truststore.location</name>
<value>/path/to/truststore.jks</value>
</property>
<property>
<name>hbase.rpc.tls.truststore.password</name>
<value>trustStor3pa$$w0rd</value>
</property>
----

However specifying a trust store is not always required. Standard JDK implementations are shipped with a standard list
of trusted certificates (the certificates of Certificate Authorities) and if your private key is provided by one of them,
you don't need to configure your clients to trust it. Similarly to an internet browser, you don't need to setup the
certificates of every single website you're planning to visit. Later in this documentation we'll walk through the steps of
creating self-signed certificates which requires a trust store setup.

You can check the list of public certificate authorities shipped with your JDK implementation:

----
keytool -keystore $JAVA_HOME/jre/lib/security/cacerts -list
----

Password is empty by default.

=== Creating self-signed certificates

While obtaining globally trusted certificates from Certificate Authorities is convenient is perfect valid to generate
our own private/public keypairs and set them up specifically for the HBase cluster. Especially if we don't want to enable
public access to the cluster, paying money for a certificate doesn't make sense.

Follow the following steps to generate self-signed certificates.

. Create SSL key store JKS to store local credentials

Please note that the alias (-alias) and the distinguished name (-dname) must match the hostname of the machine that is
associated with, otherwise hostname verification won't work.

----
keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keysize 2048 -dname "cn=$(hostname -f)" -keypass password -keystore keystore.jks -storepass password
----

At the end of this operation you'll have as many key store files as many servers you have in your cluster. Each cluster member
will have its own key store.

[start=2]
. Extract the signed public key (certificate) from each key store

----
keytool -exportcert -alias $(hostname -f) -keystore keystore.jks -file $(hostname -f).cer -rfc
----

[start=3]
. Create SSL trust store JKS containing certificates for the clients

The same truststore (storing all accepted certs) should be shared on participants of the cluster. You need to use different
aliases to store multiple certificates in the same truststore. Name of the aliases doesn't matter.

----
keytool -importcert -alias [host1..3] -file [host1..3].cer -keystore truststore.jks -storepass password
----

=== Upgrading existing non-TLS cluster with no downtime

Here are the steps needed to upgrade an already running HBase cluster to TLS without downtime by taking advantage of
port unification functionality. There's a property on server side called `hbase.server.netty.tls.supportplaintext`
which makes possible to accept TLS and plaintext connections on the same socket port.

. Create the necessary key stores and trust stores for all server participants as described in the previous section.

. Enable secure communication on the Master node with `supportplaintext=True`. Restart the Master.
Copy link
Contributor

@bbeaudreault bbeaudreault Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think there needs to be more steps here, because the server side actually is impacted by both hbase.client.netty.tls.enabled AND hbase.server.netty.tls.enabled. If in this step you enable both, the HMaster will not be able to communicate with RegionServers (who will be enabled below). If you only enable server side, when you get to the "disable plaintext" step below you'll start to see failures because regionservers will be trying to communicate with the HMaster over plaintext.

So really the user needs to fully restart the cluster a few times:

  1. Enable hbase.server.netty.tls.enabled only, with supportPlaintext=true. Servers will accept TLS, but not send TLS..
  2. Additionally enable hbase.client.netty.tls.enabled on servers, keeping supportPlaintext=true. Servers will now accept and send TLS.
  3. Remove supportPlaintext=true. Servers will reject requests if not TLS.

Clients can be updated to use hbase.client.netty.tls.enabled either after step 1 or 2, but before 3.

Copy link
Contributor

@bbeaudreault bbeaudreault Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably also worth calling out somewhere that once hbase.client.netty.tls.enabled is enabled on the server side, the cluster will only be able to communicate with other clusters which have TLS enabled. For example, this would impact intra-cluster replication.

Copy link
Contributor

@Apache9 Apache9 Aug 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to implement some fallback logic at client side?

Copy link
Contributor

@Apache9 Apache9 Aug 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theoretically at client side it is possible. In the NettyRpcConnection implementation, before the ssl handshake finishes, we will not send any data out. So if we want to fallback to non ssl communication, we could just remove the ssl handler and set up the connection without encryption. But probably at server side we will just close the connection...

Copy link
Contributor

@bbeaudreault bbeaudreault Aug 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea @Apache9. I logged an issue for that: https://issues.apache.org/jira/browse/HBASE-27318


. Enable secure communication on the Region Servers with `supportplaintext=True`. Restart RSs in rolling restart fashion.

. Enable secure communication on the clients.

. Disable plaintext communication on the Master by removing `supportplaintext` property. Restart Master.

. Disable plaintext communication on the Region Servers by removing `supportplaintext` property. Restart RSs in rolling
restart fashion.

=== Additional configuration

==== Enabled protocols

Comma-separated list of TLS protocol versions to enable. Default is empty.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.enabledProtocols</name>
<value>TLSv1.2,TLSv1.3</value>
</property>
----

==== Default protocol

Set the default TLS protocol version to use. Default is TLSv1.2. Use this protocol if enabled protocols is not defined.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.protocol</name>
<value>TLSv1.2</value>
</property>
----

==== Enabled cipher suites

List of enabled cipher suites in TLS protocol. Useful when you want to disable certain cipher suites due to recent
security concerns. Default value is a mix of CBC and GCM ciphers. Due to performance reasons we prefer CBC ciphers for
Java 8 and GCM ciphers for Java 9+.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.ciphersuites</name>
<value>TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256</value>
</property>
----

==== Certificate Revocation Checking

There's a built-in mechanism in JDK's TrustManager which automatically checks certificates for revocation. See
link:https://docs.oracle.com/cd/E19263-01/817-5215/ssl.html#wp19807[Managing Server Certificates]. Disabled by default.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.clr</name>
<value>false</value>
</property>
----

==== Online Certificate Status Protocol

Enables link:https://en.wikipedia.org/wiki/OCSP_stapling[OCSP] stapling. Please note that not all `SSLProvider`
implementations support OCSP stapling and an exception will be thrown upon. Disabled by default.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.ocsp</name>
<value>false</value>
</property>
----

==== Client handshake timeout

Set the TLS client handshake timeout is milliseconds. Default is 5 seconds.

[source,xml]
----
<property>
<name>hbase.client.netty.tls.handshaketimeout</name>
<value>5000</value>
</property>
----

== Securing Access to HDFS and ZooKeeper
Secure HBase requires secure ZooKeeper and HDFS so that users cannot access and/or modify the metadata and data from under HBase. HBase uses HDFS (or configured file system) to keep its data files as well as write ahead logs (WALs) and other data. HBase uses ZooKeeper to store some metadata for operations (master address, table locks, recovery state, etc).

Expand Down