secure-inter-node.html.md.erb

---
breadcrumb: VMware RabbitMQ for Tanzu Application Service Documentation
title: Securing Inter-node Traffic with TLS
owner: London Services
---

This topic describes how to configure on-demand <%= vars.product_full %> to secure
communication between nodes in a RabbitMQ cluster.

##<a id="overview"></a> Overview

In <%= vars.product_short %>, nodes in a cluster communicate with other nodes
through Erlang distribution links.
RabbitMQ CLI tools, such as <code>rabbitmqctl</code>, also establish connections
to nodes in the cluster through distribution links.

For on-demand service instances, you can secure communication between nodes in
the cluster with <%= vars.product_short %> using mutual TLS to provide
authentication and encryption for the traffic.

For more information about clustering with TLS in RabbitMQ, see the
<a href="https://www.rabbitmq.com/clustering-ssl.html">RabbitMQ documentation</a>.

<p class="note">
  <strong>Note:</strong> Inter-node communication is independent of communication
  between your apps and RabbitMQ. You can secure inter-node traffic without modifying
  your apps to communicate over TLS.
</p>

To configure inter-node TLS for on-demand service instances:

1. [Configure Inter-node TLS](#configuration)
1. [Migrate to a Secure Inter-node Cluster](#migration)

##<a id="configuration"></a> Configure Inter-node TLS

To configure inter-node mutual TLS in the <%= vars.product_short %> tile:

1. Click the <strong>Security for On-Demand Plans</strong> tab.
  <%= image_tag("images/configure-oauth.png",
    :alt => "Screenshot of the Security for On-Demand Plans pane.
    The pane contains the **TLS Options** group of radio buttons,
    a checkbox to **Secure Inter-node communications with TLS on new Instances**,
    the **RabbitMQ TLS versions** group of checkboxes,
    the **OAuth Options** group of radio buttons, and a Save button.
    The example shows the **Not Configured** radio button selected for TLS Options,
    the Secure Inter-node communications with TLS checkbox unticked,
    TLS v1.3 and TLS v1.2 checkboxes selected for RabbitMQ TLS versions,
    and the **Optional** radio button selected for OAuth.") %>
  <a href="./images/configure-oauth.png" target="_blank" aria-hidden="true">Click here to view a larger version of the image above</a>

1. Select the **Secure Inter-node communications with TLS on new Instances**
checkbox.

1. Click <strong>Save</strong>.

1. Go back to **<%= vars.ops_manager %> Installation Dashboard** &gt; **Review Pending Changes**.

1. Click **Apply Changes** to apply the changes to the <%= vars.product_short %> tile.

After applying changes, inter-node communication in new service instances is
secured with mutual TLS.

<p class="note warning">
  <strong>Warning:</strong> This option only affects service instances created
  after clicking <strong>Apply Changes</strong>.
  It has no effect on existing service instances.
  To migrate your app traffic to a cluster using inter-node TLS, see
  <a href="#migration">Migrate to a Secure Inter-node Cluster</a> below.
</p>

##<a id="migration"></a> Migrate to a Secure Inter-node Cluster

You cannot enable inter-node TLS communication on existing service instances.
To use this feature, you must manually migrate apps to a new service instance which has
**Secure Inter-node communications with TLS** enabled.
This migration can involve less downtime if you use a blue-green deployment.

You must migrate because, when you click **Apply changes**, the rolling upgrade process that updates the cluster
cannot update existing instances to use inter-node TLS communication.
The rolling upgrade would fail because an Erlang distribution cannot contain a
mix of nodes that communicate over TLS and nodes that do not.

<p class="note">
  <strong>Note:</strong> This migration procedure assumes that you are familiar with RabbitMQ concepts,
  such as dynamic shovels and how to configure them.
  If you need help with this procedure, contact VMware Support.
  <br><br>
  You must manually create RabbitMQ objects.
  This might not be feasible if you have more than 100 queues.
  VMware Support can provide a script to perform some of these steps.
</p>

To migrate apps to a secure inter-node cluster using a blue-green deployment:

1. In the subnet of your original service instance, create a new service instance
with the same plan and configuration as the original service instance by running:

    ```
    cf create-service p.rabbitmq PLAN-NAME GREEN-INSTANCE-NAME -c PARAMETERS-AS-JSON
    ```
    Where:

    - `PLAN-NAME` is plan you used for the original service instance
    - `GREEN-INSTANCE-NAME` is the name of the new service instance
    - `PARAMETERS-AS-JSON` is the configuration you applied to the original service instance

    For example, if your original instance has TLS enabled:

    <pre class="terminal">
    $ cf create-service p.rabbitmq my-rabbitmq-plan rabbitmq-green -c '{"tls": true}'
    </pre>

    <p class="note">
      <strong>Note:</strong> In this procedure, this new service instance and its
      corresponding cluster are referred to as green. The original service instance
      and cluster are referred to as blue.
    </p>

1. After the green service instance is deployed, run these commands to create admin credentials for both the blue
and green service instances if they do not already have credentials:

    ```
    cf create-service-key BLUE-INSTANCE-NAME admin-key -c '{"tags": "administrator"}'
    ```
    ```
    cf create-service-key GREEN-INSTANCE-NAME admin-key -c '{"tags": "administrator"}'
    ```
    Where:

    - `BLUE-INSTANCE-NAME` is the name of your original service instance
    - `GREEN-INSTANCE-NAME` is the name of the new service instance

1. Retrieve the service key for the blue and green instances by running these commands:

    ```
    cf service-key BLUE-INSTANCE-NAME admin-key
    ```
    ```
    cf service-key GREEN-INSTANCE-NAME admin-key
    ```
    From the output, record:

    - The URL and admin credentials for the Management UI
    - The AMQP or AMQPS URI listed in the service-key.
    You will need this to set up shovels between the two clusters.
    This is stored under the key `uri`.

1. Log in to the Management UI on both the blue and green clusters.

1. Stop the apps bound to the blue service instance so that no new messages are
sent and no new topology objects, such as queues or policies, are created.
Do so by running:

    ```
    cf stop APP-NAME
    ```
    Where `APP-NAME` is the name of your messaging app.

1. In the **Overview** pane of the Management UI for the blue (original) cluster,
navigate to **Export Definitions** and click **Download broker definitions**.
This saves a JSON file to your local machine that contains the topology metadata you need to re-create
the cluster on another service instance.
  <p class="note">
    <strong>Note:</strong> If you do not see the <strong>Export definitions</strong> section of
    the <strong>Overview</strong> pane, it is likely that the service-key you created did not
    contain the tag to enable admin permissions.
    Re-check the command you ran to create the key.
  </p>

1. Modify the definitions file you downloaded in the previous step to remove
anything you do not want on your new cluster.
For example, the new cluster will create its own users, so you must remove any
user credentials before importing the file onto the new cluster.

1. In the **Overview** pane of the Management UI for the green (new) cluster,
navigate to **Import Definitions** and add the JSON file you downloaded from
the **blue** cluster and click **Upload broker definitions**.
This creates the topology required on the green cluster.
  <p class="note">
    <strong>Note:</strong> If you do not see the <strong>Import definitions</strong> section of
    the <strong>Overview</strong> pane, it is likely that the service-key you created did not
    contain the tag to enable admin permissions.
    Re-check the command you ran to create the key.
  </p>

1. Configure a shovel on the green cluster to drain any messages from the blue cluster.
Create a shovel for each queue that you want to drain to the green cluster.<br><br>
If you have many queues, VMware recommends that you create these in batches and wait for them to complete before you create
new shovels. This is to avoid overloading the system.

    1. In the Management UI of the green cluster, navigate to **Admin** and go to
    **Shovel Management** in the sidebar.
    1. Click **Add a new shovel**.
    1. Under **Source**:
        - Set **URI** to the URI you retrieved from the service key for the blue cluster.
        For example: <code>amqp://user:password@server-name</code>
        <br><br>
        If you are using TLS for AMQP traffic, add the following query parameter onto the end of the URI:

            ```
            ?cacertfile=/var/vcap/jobs/rabbitmq-server/etc/cacert.pem&certfile=/var/vcap/jobs/rabbitmq-server/etc/cert.pem&keyfile=/var/vcap/jobs/rabbitmq-server/etc/key.pem&verify=verify\_peer
            ```
            For example, your shovel's source URI will look similar to:

            <pre class="terminal">amqps://user:password@server-name?cacertfile=/var/vcap/jobs/rabbitmq-server/etc/cacert.pem&certfile=/var/vcap/jobs/rabbitmq-server/etc/cert.pem&keyfile=/var/vcap/jobs/rabbitmq-server/etc/key.pem&verify=verify\_peer</pre>

        - Set **Queue** to the name of a queue that you want to drain from the blue cluster.
        - Set **Auto-delete** to **After initial length transferred**.

1. Wait for all of the shovels to disappear from the Management UI of the green cluster.
This indicates that all messages have been drained from the blue cluster.

1. Unbind your apps from the blue cluster and bind them to the green cluster:
    - If your apps are running on the same foundation, re-bind your apps by running these commands:

        ```
        cf unbind-service APP-NAME BLUE-INSTANCE-NAME
        ```
        ```
        cf bind-service APP-NAME GREEN-INSTANCE-NAME
        ```
    - If your apps are running off-foundation and communicate with RabbitMQ through
    the Service Gateway:
        1. Create a new service key on the green cluster by running:

            ```
            cf create-service-key GREEN-INSTANCE-NAME messaging-app-key
            ```
        1. Supply the service key to your app so that it can communicate with the new cluster.

1. Restart your apps. If you know which of your apps are consumers and which are producers,
VMware recommends that you restart the consumers first to avoid building up a backlog on the green cluster.

    ```
    cf start APP-NAME
    ```

1. Confirm in the Management UI for the blue cluster that the blue cluster has
no messages in queues and no throughput.

1. Confirm in the Management UI for the green cluster that the green cluster has
the expected throughput.

1. After confirming the green cluster is as expected, delete the blue cluster by running:

    ```
    cf delete-service BLUE-INSTANCE-NAME
    ```