Share TLS config between RabbitMQ plugins by default

[ferozjilla]

Is your feature request related to a problem? Please describe.

TLS for AMQP can be enabled by following - https://www.rabbitmq.com/kubernetes/operator/using-operator.html#tls-conf.

TLS for other RabbitMQ plugins - the management plugin for example, must be done through a few CR changes. For example, by configuring the TLS properties for the plugin through additionalConfig and opening the TLS port through the statefulSetOverride.

The extra work for enabling TLS for plugins may not be justified given that in most cases:

1. When a user enables TLS, they wish to enable TLS for all plugins and not just AMQP
2. The same TLS credentials are used across plugins anyway

This most cases belief came up in our public slack channel and was confirmed by a member of the core team, and a platform architect.

Given this, we can extend the AMQP TLS setting to all plugins.

Scenario: TLS is configured for all enabled plugins
Given I deploy a RabbitMQ cluster with `spec.tls` set, and the plugins - management and stomp
When I try with contact RabbitMQ over the default TLS ports for each of the plugins - amqp, management, stomp
Then the connection succeeds and uses TLS

Scenario: Adding a plugin when TLS is enabled
Given I have a RabbitMQ cluster with TLS enabled
And I update the cluster by adding the MQTT plugin in the `additionalPlugins` section
When I try to connect to RabbitMQ using MQTTS over the default TLS port for MQTT
Then the connection succeeds and uses TLS

Scenario: Overriding the default TLS settings
Given I have a RabbitMQ cluster with TLS enabled and the management plugin
And I update the cluster to change the TLS port for management to 15678
By adding `management.ssl.port = 15678` to the `additionalConfig`
By opening port 15678 using the `statefulSetOverride`
When I try to connect over the new port
Then the connection succeeds and uses TLS

Scenario: Disable non-ssl listeners
Given I have a RabbitMQ cluster with TLS enabled and the management plugin
When I try to connect to the non-TLS management port (15672)
Then I see that the connection fails

Additional Context

RabbitMQ ships with some tier-1 plugins. We will need to decide which of these plugins to auto-enable TLS for. For example, enabling TLS for the management plugin involves much the same work as enabling TLS for AMQP. However, for the shovel and federation plugins, since another cluster is involved, this may be trickier.

The changes for AMQP TLS can be found in it's PR.

[ferozjilla]

TLS for Tier-1 plugins

Going through the list of tier-1 plugins that ship with RabbitMQ - https://www.rabbitmq.com/plugins.html#tier1-plugins

LDAP

What is it?
The LDAP plugin defers authentication + authorisation to an LDAP v3 implementation: commonly OpenLDAP and Microsoft Active Directory.

TLS
TLS is configurable and requires adding a property to rabbitmq.conf and advanced.config - https://www.rabbitmq.com/ldap.html#ldap-tls

AMQP 1.0

What is it?

AMQP 0-9-1 and 1.0 are very much different protocols and thus 1.0 is treated as a separate protocol supported by RabbitMQ, not a revision of the original protocol that will eventually supersede it.

TLS
Shares the same TLS settings as AMQP 0.9.1

Web MQTT

What is it?

The Web MQTT plugin makes it possible to use MQTT over a WebSocket connection.
The goal of this plugin is to enable MQTT messaging in Web applications.

TLS
Configurable by adding properties in rabbitmq.conf: https://www.rabbitmq.com/web-mqtt.html#tls

Web STOMP

What is it?

The Web STOMP plugin makes it possible to use STOMP over a WebSocket connection.
The goal of this plugin is to enable STOMP messaging in Web applications.

TLS
Configurable by adding properties in rabbitmq.conf: https://www.rabbitmq.com/web-stomp.html#tls

RabbitMQ Trust Store

What is it?

This plugin provides support for TLS (x509) certificate whitelisting. All plugins which use the global TLS options will be configured with the same whitelist.

TLS
The ceretificates to whitelist are supplied by "providers", which can be a filesystem directory, or over http(s). The http provider is securable by TLS but it would not make sense for us to set this default since that would assume we use the http provider in the first place (the other provider may be being used instead).

No default TLS configuration even if plugin is enabled.

STOMP

What is it?

STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol.

STOMP provides an interoperable wire format so that STOMP clients can communicate with any STOMP message broker to provide easy and widespread messaging interoperability among many languages, platforms and brokers.

TLS
Configurable: https://www.rabbitmq.com/stomp.html#tls
non-TLS listeners can also be disabled

MQTT

What is it?

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth. MQTT today is used in a wide variety of industries, such as automotive, manufacturing, telecommunications, oil and gas, etc.

TLS
Configurable: https://www.rabbitmq.com/mqtt.html#tls

Shovel

What is it?

a core RabbitMQ plugin that unidirectionally moves messages from a source to a destination.

A shovel behaves like a well-written client application, which connects to its source and destination, consumes and republishes messages, and uses acknowledgements on both ends to cope with failures.

TLS
Configurable: https://www.rabbitmq.com/shovel.html#tls

Since shovel uses AMQP 0.9.1 or AMQP 1.0 underneath, configuring TLS involves configuring AMQP TLS on the broker and configuring the Erlang client to use TLS.

Looks a bit more involved, so may be considered in a separate issue, but will skip for now.

Federation

What is it?

The federation plugin allows you to make exchanges and queues federated. A federated exchange or queue can receive messages from one or more upstreams (remote exchanges and queues on other brokers).

The high-level goal of the Federation plugin is to transmit messages between brokers without requiring clustering.

TLS
Similar to Shovel: https://www.rabbitmq.com/federation.html#tls-connections

Consistent Hash Exchange

What is it?

This plugin adds a consistent-hash exchange type to RabbitMQ. This exchange type uses consistent hashing (intro blog posts: one, two, three) to distribute messages between the bound queues

TLS
Not supported, just an internal exchange type.

RabbitMQ Auth Backend HTTP

What is it?

This plugin provides the ability for your RabbitMQ server to perform authentication (determining who can log in) and authorisation (determining what permissions they have) by making requests to an HTTP server.

TLS
Configurable - https://github.com/rabbitmq/rabbitmq-auth-backend-http/blob/v3.7.x/README.md#using-tlshttps. Although not clear if just through rabbitmq.conf or additionally through advanced.conf.

[ferozjilla]

Based on the learnings above:

List of plugins to share TLS config with -

• mqtt
• stomp
• amqp_1_0

Shovel and Federation are similar in that they require the Erlang client to use TLS in addition to configuring AMQP 0.9.1/AMQP 1.0 tls in the upstream RabbitMQ broker.
Decision: They are more involved and may be done together if/when there is a request for them.

The TLS config may be shared for the Web Stomp, Web MQTT, Auth Backend HTTP and the LDAP plugins too. For these plugins, the CA cert property is required too. This is not the case for the other plugins and we at the moment do not assert that the CA cert property be provided in the TLS secret.
Decision: Thus, this would also need a little more work. We could document a reference example using these, and defer implementing until a user need comes up.

TLS not applicable for Consistent Hash Exchange and RabbitMQ Trust Store

[ferozjilla]

RabbitMQ Cluster Operator TLS Configuration

At the moment, TLS can be configured in a RabbitMQ Cluster using the following options:

spec:
  tls:
    secretName: rabbitmq-server-certs
    caSecretName: rabbitmq-ca-cert
    caCertName: ca.crt

What should the contents of these secrets be? What do these secrets translate to in terms of RabbitMQ properties? Not sure, let's read the docs. Okay, secretName must contain a tls.crt and a tls.key that represent the RabbitMQ server key and cert, and the caSecretName must contain a ca cert with key caCertName (the key in the secret itself is caCertName).

Conclusion 1: The property names are not self documenting and require going through the documentation to understand.

That answers question 1. What about question 2 - what do these secrets translate to? The docs tell me that filling in caCertName and caSecretName gives me mutual TLS, but is that enforced? Is fail_if_no_peer_cert set to true in rabbitmq.conf? Are non-tls listeners disabled or can I still connect via non-tls ports too?

To answer this, I can either just try it and verify, or read through the source code. The documentation does not cover this at the moment.

Conclusion 2: It is not clear what the properties translate to in terms of rabbitmq.conf properties. It is also not easy to find this out.

The answer from the source code is that the properties translate to the following in rabbitmq.conf:

ssl_options.certfile   = /etc/rabbitmq-tls/tls.crt
ssl_options.keyfile    = /etc/rabbitmq-tls/tls.key
listeners.ssl.default  = 5671

management.ssl.certfile   = /etc/rabbitmq-tls/tls.crt
management.ssl.keyfile    = /etc/rabbitmq-tls/tls.key
management.ssl.port       = 15671

ssl_options.cacertfile = /etc/rabbitmq-tls/ca.certificate
ssl_options.verify     = verify_peer
management.ssl.cacertfile = /etc/rabbitmq-tls/ca.certificate

The three sections separated by a line space set up amqp tls, management tls, and amqp and management mtls respectively.

Neither fail_if_no_peer_cert nor disable_tcp_listeners is configured. This means mutual TLS is not enforced and RabbitMQ will continue listening over non-TLS ports.

There is no configuraion option in spec.TLS to configure either of the above properties and it must be done using the Additional Config.

Conclusion 3: Custom Resource level toggle is missing for fail_if_no_peer_cert and disable_tcp_listeners.

Perhaps this is okay, given that there is still a way to configure this.

Document How spec.TLS Translates to RabbitMQ TLS Config

We should capture the TLS properties in rabbitmq.conf created as a result of setting TLS properties in the RabbitMQ Cluster custom resource.

Rename Properties

To aid with conclusions 1 and 2, we can rename the existing properties to:

spec:
  tls:
    # provide cert and key inline
    cert:
    key:
    # alternatively, provide a cert key secret
    certKeySecret:
    
    # provide ca cert inline
    caCert:
    # alternatively, provide ca secret
    caCertSecret:
    caCertNameInSecret: 

The inline options are a nice to have, but not necessary. So far, we have just renamed secretName -> certKeySecret and caCertName -> caCertNameInSecret. certKeySecret has a better mapping to RabbitMQ configuration, and caCertKeyInSecret is perhaps more understandable.

Add New Top Level Properties

As for conclusion 3, we can extend the options to top level configuration:

spec:
  tls:
    certKeySecret:
    caCertSecret:
    caCertKeyInSecret: 

    # or enforceTLS?
    failIfNoPeerCert:
    
    disableNonTLSListeners:  

This can equally be done without the renaming too.

Output

• Create a story to document the effect TLS properties have to rabbitmq.conf
• Rename tls properties after agreeing with the team
• Add fail_if_no_peer_cert and disable_tcp_listeners properties to spec.TLS after agreeing with the team

[ansd]

Decision:

1. Remove caCertName field
2. Add web stomp and web mqtt, management to the shared tls config.

[ablease]

Today we removed the CaCertName property and completed the config map changes for web-stomp and web-mqtt.

Next step tomorrow is service and statefulset changes for web-stomp and web-mqtt.

cc @ferozjilla