HDDS-4392. [DOC] Add Recon architecture to docs

hadoop-hdds/docs/content/concept/OzoneManager.md

@@ -97,7 +97,7 @@ the data from the data node.
 
 For a detailed view of Ozone Manager this section gives a quick overview about the provided network services and the stored persisted data.
 
-**Network services provided by Ozone Manager:**
+### Network services provided by Ozone Manager:
 
 Ozone provides a network service for the client and for administration commands. The main service calls
 
@@ -115,7 +115,7 @@ Ozone provides a network service for the client and for administration commands.
    * ServiceList (used for service discovery)
    * DBUpdates (used by [Recon]({{< ref "feature/Recon.md" >}}) downloads snapshots)
 
-**Persisted state**
+### Persisted state
 
 The following data is persisted in Ozone Manager side in a specific RocksDB directory:
 

hadoop-hdds/docs/content/concept/Recon.md

@@ -0,0 +1,163 @@
+---
+title: "Recon"
+date: "2020-10-27"
+weight: 8
+menu: 
+  main:
+     parent: Architecture
+summary: Recon serves as a management and monitoring console for Ozone.
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+Recon serves as a management and monitoring console for Ozone. It gives a 
+bird's-eye view of Ozone and helps users troubleshoot any issues by presenting
+the current state of the cluster through REST based APIs and rich web UI.
+
+
+## High Level Design
+
+{{<figure src="/concept/ReconHighLevelDesign.png" width="800px">}}
+
+<br/>
+
+On a high level, Recon collects and aggregates metadata from Ozone Manager (OM), 
+Storage Container Manager (SCM) and Datanodes (DN) and acts as a central 
+management and monitoring console. Ozone administrators can use Recon to query 
+the current state of the system without overloading OM or SCM. 
+
+Recon maintains multiple databases to enable batch processing, faster querying 
+and to persist aggregate information. It maintains a local copy of OM db and 
+SCM db along with a SQL database for persisting aggregate information.
+
+Recon also integrates with Prometheus to provide a HTTP endpoint to query Prometheus 
+for Ozone metrics and also to display a few crucial point in time metrics in 
+the web UI.
+
+## Recon and Ozone Manager
+
+{{<figure src="/concept/ReconOmDesign.png" width="800px">}}
+
+<br/>
+
+Recon gets a full snapshot of OM rocks db initially from the leader OM's HTTP 
+endpoint, untars the file and initializes RocksDB for querying locally. The
+database is kept in sync by periodically requesting delta updates from the leader
+OM via RPC calls from the last applied sequence id. If for any reason, the delta 
+updates could not be retrieved or applied to the local db, a full snapshot is 
+requested again to keep the local db in sync with OM db. Due to this, Recon can 
+show stale information since the local db will not always be in sync.
+
+The db updates retrieved from OM is then converted into a batch of events for 
+further processing by OM db tasks via [Recon Task Framework](#task-framework). 
+
+
+## Recon and Storage Container Manager
+
+{{<figure src="/concept/ReconScmDesign.png" width="800px">}}
+
+<br/>
+
+Recon also acts as a passive SCM for datanodes. When Recon is configured in the
+cluster, all the datanodes register with Recon and send heartbeats, container 
+reports, incremental container reports etc. to Recon similar to SCM. Recon uses
+all the information it gets from datanodes to construct its own copy of SCM rocks db 
+locally. Recon never sends any command to datanodes in response and just acts as
+a passive SCM for faster lookup of SCM metadata.
+
+## <a name="task-framework"></a> Task Framework
+
+Recon has its own Task framework to enable batch processing of data obtained 
+from OM and SCM. A task can listen to and act upon db events such as `PUT`, `DELETE`,
+`UPDATE`, etc. on either OM db or SCM db. Based on this, a task either implements 
+`org.apache.hadoop.ozone.recon.tasks.ReconOmTask` or extends 
+`org.apache.hadoop.ozone.recon.scm.ReconScmTask`.   
+
+An example `ReconOmTask` is `ContainerKeyMapperTask` that persists the container -> key 
+mapping in RocksDB. This is useful to understand which keys were part of the container 
+when the container is reported missing or is in a bad health state. Another example is 
+`FileSizeCountTask` which keeps track of count of files within a given file size range in 
+a SQL database. These tasks have implementations for two scenarios:
+
+ - Full snapshot (reprocess())
+ - Delta updates (process())
+
+When a full snapshot of OM db is obtained from the leader OM, the reprocess() 
+is called on all the registered OM tasks. On subsequent delta updates, process()
+is called on these OM tasks. 
+
+An example `ReconScmTask` is `ContainerHealthTask` that runs in configurable 
+intervals to scan the list of all the containers and to persist the state of 
+unhealthy containers (`MISSING`, `MIS_REPLICATED`, `UNDER_REPLICATED`, `OVER_REPLICATED`)
+in a SQL table. This information is used to determine if there are any missing 
+containers in the cluster.
+
+## Recon and Prometheus
+
+Recon can integrate with any Prometheus instance configured to collected metrics 
+and can display useful information in Recon UI in Datanodes and Pipelines pages.
+Recon also exposes a proxy endpoint ([/metrics]({{< ref "interface/ReconApi.md#metrics" >}}))
+to query Prometheus. This integration can be enabled by setting this configuration `ozone.recon.prometheus.http.endpoint` 
+to the Prometheus endpoint like `ozone.recon.prometheus.http.endpoint=localhost:9090`.
+
+## API Reference
+
+[Link to complete API Reference]({{< ref "interface/ReconApi.md" >}})
+
+## Persisted state
+
+ * A local copy of [OM database]({{< ref "concept/OzoneManager.md#persisted-state" >}})
+ * A local copy of [SCM database]({{< ref "concept/StorageContainerManager.md#persisted-state" >}})
+ * The following data is persisted in Recon in the specified RocksDB directory:
+     * ContainerKey table
+       * Stores the mapping (container, key) -> count
+     * ContainerKeyCount table
+       * Stores containerID -> no. of keys count within the container
+
+ * The following data is stored in the configured SQL database (default is Derby):
+     * GlobalStats table
+       * A Key -> Value table to store aggregate information like total
+       number of volumes / buckets / keys present in the cluster
+     * FileCountBySize table
+       * Keeps track of the number of files present within a file size range in the cluster
+     * ReconTaskStatus table
+       * Keeps track of the status and last run timestamp of the registered OM and SCM 
+       db tasks in the [Recon Task Framework](#task-framework)
+     * ContainerHistory table
+       * Stores ContainerReplica -> Datanode mapping with last known timestamp. This 
+       is used to determine the last known datanodes when a container is reported missing 
+     * UnhealthyContainers table
+       * Keeps track of all the Unhealthy Containers (MISSING, UNDER_REPLICATED,
+       OVER_REPLICATED, MIS_REPLICATED) in the cluster at any given time
+
+
+## Notable configurations
+
+key | default | <div style="width: 300px;">description</div>
+----|---------|------------
+ozone.recon.http-address | 0.0.0.0:9888 | The address and the base port where the Recon web UI will listen on.
+ozone.recon.address | 0.0.0.0:9891 | RPC address of the Recon.
+ozone.recon.db.dir | none | Directory where the Recon Server stores its metadata.
+ozone.recon.om.db.dir | none | Directory where the Recon Server stores its OM snapshot DB.
+ozone.recon.om.snapshot<br>.task.interval.delay | 10m | Interval in MINUTES by Recon to request OM DB Snapshot / delta updates.
+ozone.recon.task<br>.missingcontainer.interval | 300s | Time interval of the periodic check for Unhealthy Containers in the cluster.
+ozone.recon.sql.db.jooq.dialect | DERBY | Please refer to [SQL Dialect](https://www.jooq.org/javadoc/latest/org.jooq/org/jooq/SQLDialect.html) to specify a different dialect.
+ozone.recon.sql.db.jdbc.url | jdbc:derby:${ozone.recon.db.dir}<br>/ozone_recon_derby.db | Recon SQL database jdbc url.
+ozone.recon.sql.db.username | none | Recon SQL database username.
+ozone.recon.sql.db.password | none | Recon SQL database password.
+ozone.recon.sql.db.driver | org.apache.derby.jdbc<br>.EmbeddedDriver | Recon SQL database jdbc driver.
+

hadoop-hdds/docs/content/concept/StorageContainerManager.md

@@ -56,7 +56,7 @@ token infrastructure depends on this certificate infrastructure.
 
 For a detailed view of Storage Container Manager this section gives a quick overview about the provided network services and the stored persisted data.
 
-**Network services provided by Storage Container Manager:**
+### Network services provided by Storage Container Manager:
 
  * Pipelines: List/Delete/Activate/Deactivate
    * pipelines are set of datanodes to form replication groups
@@ -74,8 +74,7 @@ For a detailed view of Storage Container Manager this section gives a quick over
 
 Note: client doesn't connect directly to the SCM
 
-**Persisted state**
-
+### Persisted state
 
 The following data is persisted in Storage Container Manager side in a specific RocksDB directory
 

hadoop-hdds/docs/content/feature/Recon.md

@@ -1,5 +1,5 @@
 ---
-title: "Recon"
+title: "Recon Server"
 weight: 7
 menu:
    main:
@@ -23,25 +23,19 @@ summary: Recon is the Web UI and analysis service for Ozone
   limitations under the License.
 -->
 
-Recon is the Web UI and analytics service for Ozone. It's an optional component, but strongly recommended as it can add additional visibility.
+Recon serves as a management and monitoring console for Ozone. 
+It's an optional component, but it is strongly recommended to add it to the cluster
+since Recon can help with troubleshooting the cluster at critical times. 
+Refer to [Recon Architecture]({{< ref "concept/Recon.md" >}}) for detailed architecture overview and 
+[Recon API]({{< ref "interface/ReconApi.md" >}}) documentation
+for HTTP API reference.
 
-Recon collects all the data from an Ozone cluster and **store** them in a SQL database for further analyses.
-
- 1. Ozone Manager data is downloaded in the background by an async process. A RocksDB snapshots are created on OM side periodically, and the incremental data is copied to Recon and processed.
- 2. Datanodes can send Heartbeats not just to SCM but Recon. Recon can be a read-only listener of the Heartbeats and updates the local database based on the received information.
-
-Once Recon is configured, we are ready to start the service.
+Recon is a service that brings its own HTTP web server and can be started by
+the following command.
 
 {{< highlight bash >}}
 ozone --daemon start recon
 {{< /highlight >}}
 
-## Notable configurations
 
-key | default | description
-----|---------|------------
-ozone.recon.http-address | 0.0.0.0:9888 | The address and the base port where the Recon web UI will listen on.
-ozone.recon.address | 0.0.0.0:9891 | RPC address of the Recon.
-ozone.recon.db.dir | none | Directory where the Recon Server stores its metadata.
-ozone.recon.om.db.dir | none | Directory where the Recon Server stores its OM snapshot DB.
-ozone.recon.om.snapshot.task.interval.delay | 10m | Interval in MINUTES by Recon to request OM DB Snapshot.
+