HDDS-6214. [Multi-Tenant] Fix KMS Encryption/Decryption

[smengcl]

https://issues.apache.org/jira/browse/HDDS-6214

Goal: Pass the correct user principal from the client to KMS to get the correct DEK.

Before this patch, in multi-tenancy, accessId is passed to KMS as the user name rather than the actual user principal (which is not desired).

Testing

• Tested manually, see the instructions and results below.

Manual testing with Ranger KMS

Deployment and Configuration

1. Deploy an Ozone + Ranger KMS cluster with security enabled
2. Log in to Ranger Web UI as keyadmin user, edit the policy (or add a new policy) to grant systest user Create, Get Metadata, Generate EEK and Decrypt EEK permission for the purpose of this test. Save and log out.
3. IMPORTANT. Make sure to add the following config to Ranger KMS's kms-site.xml (or core-site.xml, as long as the KMS would pick it up) to allow s3g (assuming this is the login user of S3 Gateway) to impersonate other users when its talking to the KMS:

<property>
  <name>hadoop.kms.proxyuser.s3g.users</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.kms.proxyuser.s3g.hosts</name>
  <value>*</value>
</property>

In a production cluster, the value for hadoop.kms.proxyuser.s3g.hosts shall be limited to the S3G host only for better security. i.e. https://github.com/apache/ozone/blob/cef8bf5/hadoop-hdds/docs/content/security/SecuringTDE.md?plain=1#L109

hadoop.kms.proxyuser.s3g.users could also be limited to the users that would use S3 Gateway.

Be sure to restart the Ranger KMS after making the config change to take effect.

Commands

Set up a test tenant

kinit as an Ozone admin (om user in this case)

$ kinit -kt /cdep/keytabs/om.keytab om

$ ozone tenant create tenant1 --om-service-id=ozone1
22/02/01 23:00:00 INFO rpc.RpcClient: Creating Tenant: 'tenant1', with new volume: 'tenant1'
Created tenant 'tenant1'.

$ ozone tenant user assign systest --tenant=tenant1 --om-service-id=ozone1
Assigned 'systest' to 'tenant1' with accessId 'tenant1$systest'.
export AWS_ACCESS_KEY_ID='tenant1$systest'
export AWS_SECRET_ACCESS_KEY='<RETURNED_GENERATED_SHA256_ACCESS_KEY>'

Create an encrypted bucket

kinit as a regular user (systest user in this case)

$ kinit -kt /cdep/keytabs/systest.keytab systest

$ hadoop key create encKey
...
encKey has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@434a63ab has been updated.

$ ozone sh bucket create -k encKey --layout=FILE_SYSTEM_OPTIMIZED /tenant1/encrypted-bucket1
22/02/01 23:29:36 INFO rpc.RpcClient: Creating Bucket: tenant1/encrypted-bucket1, with systest as owner and Versioning false and Storage Type set to DISK and Encryption set to true

Note that we can only create encrypted buckets via OzoneShell for now, and --layout=FILE_SYSTEM_OPTIMIZED is required here if we want to use HCFS (ofs / o3fs) to access the bucket (because the default layout is OBJECT_STORE for now, and it will break HCFS access).

Run ozone sh bucket info to confirm the DEKs (data encryption keys actually used to encrypt each file, which also differs for every file) in the bucket will be encrypted by the key name in encryptionKeyName field:

$ ozone sh bucket info /tenant1/encrypted-bucket1
{
  "metadata" : { },
  "volumeName" : "tenant1",
  "name" : "encrypted-bucket1",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 805306368,
  "usedNamespace" : 1,
  "creationTime" : "2022-02-01T23:29:36.926Z",
  "modificationTime" : "2022-02-01T23:29:36.926Z",
  "encryptionKeyName" : "encKey",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "bucketLayout" : "FILE_SYSTEM_OPTIMIZED",
  "owner" : "systest",
  "link" : false
}

Ozone Shell access

$ vim file1.txt
# this is test file1

$ ozone sh key put /tenant1/encrypted-bucket1/file1.txt file1.txt
...

We can run ozone sh key info to confirm that the key is encrypted. It should print the IV and EDEK (encryptedDataEncryptionKey) of this key.

$ ozone sh key info /tenant1/encrypted-bucket1/file1.txt
{
  "volumeName" : "tenant1",
  "bucketName" : "encrypted-bucket1",
  "name" : "file1.txt",
  "dataSize" : 15,
  "creationTime" : "2022-02-01T23:34:22.062Z",
  "modificationTime" : "2022-02-01T23:34:27.373Z",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "replicationType" : "RATIS"
  },
  "ozoneKeyLocations" : [ {
    "containerID" : 4,
    "localID" : 109611004723200011,
    "length" : 15,
    "offset" : 0,
    "keyOffset" : 0
  } ],
  "metadata" : { },
  "fileEncryptionInfo" : {
    "cipherSuite" : "AES_CTR_NOPADDING",
    "iv" : "WWdijCnKdz9vu/PPuFAR4A==",
    "keyName" : "encKey",
    "ezKeyVersionName" : "orMslQ2AuxFLRCV0kAgDFv51HWyZm5KHanH4U4sW5Du",
    "encryptedDataEncryptionKey" : "RUxfXOmy/LAr0auDFZurbw==",
    "cryptoProtocolVersion" : "ENCRYPTION_ZONES"
  },
  "replicationType" : "RATIS",
  "replicationFactor" : 3
}

File retrieval via Ozone Shell should work, as expected:

$ ozone sh key cat /tenant1/encrypted-bucket1/file1.txt
...
this is test file1

S3 API access via S3 Gateway

Here we use AWS CLI to perform the testing, other S3 API compatible clients should also work seamlessly.

Install AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

$ aws --version
aws-cli/2.4.15 Python/3.8.8 Linux/3.10.0-1062.el7.x86_64 exe/x86_64.centos.7 prompt/off

Configure Access Key and Secret

$ aws configure
AWS Access Key ID [None]: tenant1$systest
AWS Secret Access Key [None]: <RETURNED_GENERATED_SHA256_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

Testing S3API

Optionally, we are defining an alias first to reduce the clutter in the AWS CLI command. If the S3 Gateway endpoint enables HTTPS and its certificate is not signed by a trusted CA, you will need to specify the CA bundle to pass the certiticate check.

$ alias aws3='aws s3api --ca-bundle /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem --endpoint https://s3g-host1.com:9879'

Reading the file1 we uploaded earlier via Ozone Shell:

$ aws3 list-buckets
{
    "Buckets": [
        {
            "Name": "encrypted-bucket1",
            "CreationDate": "2022-02-01T23:29:36.926000+00:00"
        }
    ]
}

$ aws3 list-objects --bucket encrypted-bucket1
{
    "Contents": [
        {
            "Key": "file1.txt",
            "LastModified": "2022-02-01T23:34:27.373000+00:00",
            "ETag": "2022-02-01T23:34:27.373Z",
            "Size": 15,
            "StorageClass": "STANDARD"
        }
    ]
}

$ aws3 get-object --bucket encrypted-bucket1 --key file1.txt file1-get.txt
{
    "AcceptRanges": "bytes",
    "LastModified": "2022-02-01T23:34:27+00:00",
    "ContentLength": 15,
    "CacheControl": "no-cache",
    "ContentType": "application/octet-stream",
    "Expires": "2022-02-01T23:59:46+00:00",
    "Metadata": {}
}

$ cat file1-get.txt
this is test file1

Put a new file via S3 Gateway, then get it back:

$ vim file2.txt
# this is test file42

$ aws3 put-object --bucket encrypted-bucket1 --key file2.txt --body file2.txt

$ aws3 list-objects --bucket encrypted-bucket1
{
    "Contents": [
        {
            "Key": "file1.txt",
            "LastModified": "2022-02-01T23:34:27.373000+00:00",
            "ETag": "2022-02-01T23:34:27.373Z",
            "Size": 15,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "file2.txt",
            "LastModified": "2022-02-02T00:01:27.363000+00:00",
            "ETag": "2022-02-02T00:01:27.363Z",
            "Size": 20,
            "StorageClass": "STANDARD"
        }
    ]
}

$ aws3 get-object --bucket encrypted-bucket1 --key file2.txt file2-get.txt
{
    "AcceptRanges": "bytes",
    "LastModified": "2022-02-02T00:01:27+00:00",
    "ContentLength": 20,
    "CacheControl": "no-cache",
    "ContentType": "application/octet-stream",
    "Expires": "2022-02-02T00:02:53+00:00",
    "Metadata": {}
}

$ cat file2-get.txt
this is test file42

This concludes the multi-tenancy manual testing with KMS encryption on the bucket via Ozone Shell and S3 API.

[prashantpogde]

Minor comments and nitpicks.

[prashantpogde]

Thanks @smengcl. Looks good to me.. Can you please also update the PR with the manual testing steps with KMS setup? That would be helpful

[smengcl]

Thanks @smengcl. Looks good to me.. Can you please also update the PR with the manual testing steps with KMS setup? That would be helpful

Done. One should be able to follow the steps above to make encrypted buckets work with multi-tenancy.

[errose28]

Thanks for working on this @smengcl. Some comments inline.

[smengcl]

Also triggered a CI in my fork on a branch that has HDDS-6239 applied:
https://github.com/smengcl/hadoop-ozone/commits/HDDS-6214-CI

[smengcl]

Renamed S3VolumeInfo (along with related method names) to S3VolumeContext per @avijayanhwx 's request.

Triggered proper CI here: https://github.com/smengcl/hadoop-ozone/commits/HDDS-6214-CI-2

[smengcl]

CI passed on my fork.

[errose28]

Thanks for the updates @smengcl. Looks like the name of the response proto got updated but the corresponding request proto did not. Other than that the rest of the changes look good.

[smengcl]

CI 3: https://github.com/smengcl/hadoop-ozone/commits/HDDS-6214-CI-3

[errose28]

Thanks for the quick updates @smengcl. LGTM +1.

[smengcl]

Thanks @errose28 and @prashantpogde for reviewing this.