HDDS-6275. [Multi-Tenant] Add feature documentation and CLI quick start guide

hadoop-hdds/docs/content/feature/Nonrolling-Upgrade.md

@@ -1,5 +1,6 @@
 ---
 title: Non-Rolling Upgrades and Downgrades
+weight: 9
 menu:
    main:
       parent: Features

hadoop-hdds/docs/content/feature/S3-Multi-Tenancy-Access-Control.md

@@ -0,0 +1,68 @@
+---
+title: "Access Control"
+weight: 13
+menu:
+   main:
+      parent: "S3 Multi-Tenancy"
+summary: Access Control with Ranger in Ozone Multi-Tenancy
+hideFromSectionPage: true
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+### Ranger Policies
+
+When a tenant is created, Ozone will create a set of Ranger policies on the tenant's volume which allow the following:
+
+1. All users are able to create new buckets;
+2. Only the bucket owner (i.e. the user that creates the bucket) and tenant admins can access the bucket content.
+    - Note: For Ozone admins, typically there would be other Ranger policies that grants them full access to the cluster, if this is case they should be able to access the buckets as well. Though it is still possible to create new Ranger policies to explicitly deny them access to buckets.
+
+Ranger admin is responsible for manually adding new policies to grant or deny any other access patterns. For example:
+- Allow all users in a tenant read-only access to a bucket.
+  - Corresponding Ranger policy Allow Condition: `Roles = tenantName-UserRole, Permissions = READ,LIST`
+
+It is recommended to add new policies instead of editing the default tenant policies created by Ozone. **DO NOT** remove the **Policy Label** on those default tenant policies, or else the Ozone Manager might fail to sync with Ranger for those policies.
+
+### Ranger Roles
+
+These new Ranger policies would have the corresponding **Ranger roles** added in their **Allow Conditions**.
+
+Namely, `tenantName-UserRole` and `tenantName-AdminRole` Ranger roles are created when a tenant is created by an Ozone administrator under the CLI.
+
+`tenantName-UserRole` contains a list of all user names that are assigned to this tenant.
+
+`tenantName-AdminRole` contains a list of all tenant admins that are assigned to this tenant.
+
+We leverage Ranger roles mainly for the advantage of easier user management in a tenant:
+1. When new users are assigned to a tenant, Ozone Manager simply adds the new user to `tenantName-UserRole` Ranger role.
+2. When new tenant admins are assigned, Ozone Manager simply adds the user name to `tenantName-AdminRole` Ranger role. Delegated tenant admins will have the "Role Admin" checkbox checked, while non-delegated tenant admins won't.
+    - Role admins in a Ranger role has the permission to edit that Ranger role.
+3. And because `tenantName-AdminRole` is the "Role Admin" of `tenantName-UserRole`, whichever user in the `tenantName-AdminRole` automatically has the permission to add new users to the tenant, meaning all tenant admins (whether delegated or not) has the permission to assign and revoke users in this tenant.
+
+- **DO NOT** manually edit any Ranger roles created by Ozone. Any changes to them will be overwritten by the Ozone Manager's Ranger sync thread. Changes in tenant membership should be done using [Multi-Tenancy CLI commands]({{< ref "feature/S3-Tenant-Commands.md" >}}).
+
+
+### Ranger Sync
+
+A Ranger Sync thread has been implemented to keep the Ranger policy and role states in-sync with Ozone Manager database in case of Ozone Manager crashes during tenant administrative operations.
+
+The Ranger Sync thread does the following:
+1. Cleans up any default tenant policies if a tenant is already deleted.
+2. Checks if default tenant roles are out-of-sync (could be caused by OM crash during user assign/revoke operation). Overwrites them if this is the case.
+3. Performs all Ranger update (write) operations queued by Ozone tenant commands from the last sync, if any.
+   - This implies there will be a delay before Ranger policies and roles are updated for any tenant write operations (tenant create/delete, tenant user assign/revoke/assignadmin/revokeadmin, etc.). 

hadoop-hdds/docs/content/feature/S3-Multi-Tenancy-Setup.md

@@ -0,0 +1,108 @@
+---
+title: "Setup"
+weight: 11
+menu:
+   main:
+      parent: "S3 Multi-Tenancy"
+summary: Preparing Ozone clusters to enable Multi-Tenancy feature
+hideFromSectionPage: true
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+Steps to enable S3 Multi-Tenancy feature in Ozone clusters.
+
+
+## Setting up S3 Multi-Tenancy for production
+
+### Secure the cluster
+
+Follow [this guide]({{< ref "security/SecureOzone.md" >}}) to Kerberize (secure) the cluster if the cluster is not Kerberized yet.
+
+### Set up S3 Gateway
+
+Follow [this guide]({{< ref "interface/S3.md" >}}) the cluster to set up at least one S3 Gateway if the cluster doesn't have a S3 Gateway yet.
+
+### Set up Apache Ranger
+
+First make sure ACL is enabled, and `RangerOzoneAuthorizer` is the effective ACL authorizer implementation in-use for Ozone.
+If that is not the case, [follow this]({{< ref "security/SecurityWithRanger.md" >}}). 
+
+Then simply add the following configs to `ozone-site.xml`:
+
+```xml
+<property>
+	<name>ozone.om.ranger.https-address</name>
+	<value>https://RANGER_HOSTNAME:6182</value>
+</property>
+<property>
+	<name>ozone.om.ranger.https.admin.api.user</name>
+	<value>RANGER_ADMIN_USERNAME</value>
+</property>
+<property>
+	<name>ozone.om.ranger.https.admin.api.passwd</name>
+	<value>RANGER_ADMIN_PASSWORD</value>
+</property>
+```
+
+Finally restart all OzoneManagers to apply the new configs.
+
+Now you can follow the [Multi-Tenancy CLI command]({{< ref "feature/S3-Tenant-Commands.md" >}}) guide to try the commands. 
+
+
+## Try in a Docker Compose cluster (For developers)
+
+Developers are encouraged to try out the CLI commands inside the `./compose/ozonesecure/docker-compose.yaml` cluster environment that ships with Ozone.
+
+The Docker Compose cluster has Kerberos and security pre-configured.
+But note it differs from an actual production cluster that Ranger has been replaced with a mock server. And OzoneManager does **not** use Ranger for ACL.
+
+Because the mock server does not mock all Ranger endpoints, some operations that works for a real Ranger deployment will not work by default. e.g. assigning users to a tenant other than `tenantone`.
+But one can add new custom endpoints in `./compose/ozonesecure/mockserverInitialization.json` as needed.
+
+To launch the Docker Compose cluster locally, from Ozone distribution root:
+
+```shell
+cd compose/ozonesecure
+docker-compose up -d --scale datanode=3
+docker-compose exec scm bash
+```
+
+It might be necessary to run the following command first before testing the tenant commands in the `compose/ozonesecure` Docker environment
+in order to workaround a Docker-specific DNS issue when first contacting Ranger.
+
+```shell
+bash-4.2$ curl -k https://ranger:6182/
+{}
+```
+
+Then all subsequent requests to Ranger (mock server) should work as expected.
+
+Otherwise you might see such DNS error:
+
+```shell
+bash-4.2$ ozone tenant create tenantone
+2022-02-16 00:00:00,000 [main] INFO rpc.RpcClient: Creating Tenant: 'tenantone', with new volume: 'tenantone'
+INTERNAL_ERROR No subject alternative DNS name matching ranger found.
+```
+
+
+Operations requiring Ozone cluster administrator privilege are run as `om/om` user:
+
+```shell
+kinit -kt /etc/security/keytabs/om.keytab om/[email protected]
+```

hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md

@@ -0,0 +1,75 @@
+---
+title: "S3 Multi-Tenancy"
+weight: 10
+menu:
+   main:
+      parent: Features
+summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API.
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are
+confined to a **single** designated S3 volume (that is volume `s3v`, by default).
+
+Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created.
+Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control.
+
+The concept **tenant** is introduced to Ozone by multi-tenancy.
+Each tenant has its own designated volume.
+Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair
+generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI.
+
+> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants).
+
+> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs).
+
+## Basics
+
+1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI.
+2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. 
+   - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID.
+   - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger.
+   - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager.
+   - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant).
+   - One user can be assigned to multiple tenants. The user will have a different key pair to access each tenant. For instance, `testuser` could use `tenantone$testuser` to access `tenantone` buckets, and use `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway.
+     - A bucket link can be set up if cross-tenant (cross-volume) access is desired. See **Creating bucket links** section in [Tenant commands]({{< ref "feature/S3-Tenant-Commands.md" >}}).
+3. The Ozone cluster admin can then assign tenant admin roles to that user.
+4. Tenant admin are able to assign new users to the tenant.
+   - They can even assign new tenant admins in their tenant, if they are delegated tenant admins, which is the default. See **Assign a user as a tenant admin** section in [Tenant commands]({{< ref "feature/S3-Tenant-Commands.md" >}}).
+   - Note that tenant admins still need to use Ozone tenant CLI to assign new users to the tenant.
+     - Once tenant admins get the Kerberos TGT (via `kinit`), they can run `user assign` command to assign new users. Ozone Manager will recognize that they are the tenant admins and allow the user to do so in their tenants.
+5. After that, users can use any S3-compatible client (awscli, Python boto3 library, etc.) to access the buckets in the tenant volume via S3 Gateway using the generated key pairs.
+
+
+## Access Control
+
+Ozone multi-tenancy relies on [Apache Ranger]({{< ref "security/SecurityWithRanger.md" >}}) to enforce access control to resources.
+
+See [Access Control]({{< ref "feature/S3-Multi-Tenancy-Access-Control.md" >}}) for more information.
+
+## Setup
+
+See [Multi-Tenancy Setup]({{< ref "S3-Multi-Tenancy-Setup.md" >}}).
+
+## Usage
+
+See [Tenant Subcommands]({{< ref "feature/S3-Tenant-Commands.md" >}}).
+
+## References
+
+ * For developers: check out the upstream jira [HDDS-4944](https://issues.apache.org/jira/browse/HDDS-4944) and the attached design docs.