HBASE-26265 Update ref guide to mention the new store file tracker im…

src/main/asciidoc/_chapters/store_file_tracking.adoc

@@ -0,0 +1,175 @@
+////
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+////
+
+[[storefiletracking]]
+= Store File Tracking
+:doctype: book
+:numbered:
+:toc: left
+:icons: font
+:experimental:
+
+== Overview
+
+This feature introduces an abstraction layer to track store files still used/needed by store
+engines, allowing for plugging different approaches of identifying store
+files required by the given store.
+
+Historically, HBase internals have relied on creating hfiles on temporary directories first, renaming
+those files to the actual store directory at operation commit time. That's a simple and convenient
+way to separate transient from already finalised files that are ready to serve client reads with data.
+This approach works well with strong consistent file systems, but with the popularity of less consistent
+file systems, mainly Object Store file systems, dependency on rename operations starts to introduce
+performance penalties. Amazon S3 Object Store, in particular, has been the most affected deployment,
+due to the its lack of atomic renames, requiring an additional locking layer implemented by HBOSS,
+to guarantee consistency and integrity of operations.
+
+With *Store File Tracking*, decision on where to originally create new hfiles and how to proceed upon
+commit is delegated to the specific Store File Tracking implementation.
+It can be set at individual Table or Column Family configurations, as well as in processes
+*hbase-site.xml* configuration file.
+
+NOTE: When specified in *hbase_site.xml*, this configuration is also saved into tables configuration
+at table creation time. This is to avoid dangerous configuration mismatches between processes, which
+could potentially lead to data loss.
+
+== Available Implementations
+
+Store File Tracking initial version provides three builtin implementations:
+
+* DEFAULT
+* FILE
+* MIGRATION
+
+### DEFAULT
+
+As per the name, this is the Store File Tracking implementation used by default when now explicit
+configuration has been defined. The DEFAULT tracker implements the standard approach using temporary
+directories and renames.
+
+### FILE
+
+A file tracker implementation that creates new files straight in the store directory, avoiding the
+need for rename operations. It keeps a list of committed hfiles in memory, backed by meta files, in
+each store directory. Whenever a new hfile is committed, the list of _tracked files_ in the given
+store is updated and a new meta file is written with this list contents, discarding the previous
+meta file now containing an out dated list.
+
+### MIGRATION
+
+A special implementation to be used when swapping between Store File Tracking implementations on
+pre-existing tables that already contain data, and therefore, files being tracked under an specific
+logic.
+
+== Usage
+
+For fresh deployments that don't yet contain any user data, *FILE* implementation can be just set as
+value for *hbase.store.file-tracker.impl* property in global *hbase-site.xml* configuration, prior
+to the first hbase start. Omitting this property sets the *DEFAULT* implementation.
+
+### Switching implementations globally
+
+For running clusters with tables already containing data, Store File Tracking implementation can
+only be changed with the *MIGRATION* implementation, so that the _new tracker_ can safely build its
+list of tracked files based on the list of the _current tracker_. Additional to the
+*hbase.store.file-tracker.impl* property, *MIGRATION* requires the
+*hbase.store.file-tracker.migration.src.impl* and *hbase.store.file-tracker.migration.dst.impl*,
+where the _current_ and _new_ tracker should be specified. For example, to set *MIGRATION* from
+*DEFAULT* to *FILE*, the following should be set in the global config:
+
+----
+<property>
+  <name>hbase.store.file-tracker.impl</name>
+  <value>MIGRATION</value>
+</property>
+<property>
+  <name>hbase.store.file-tracker.migration.src.impl</name>
+  <value>DEFAULT</value>
+</property>
+<property>
+  <name>hbase.store.file-tracker.migration.dst.impl</name>
+  <value>FILE</value>
+</property>
+----
+
+A cluster restart would be needed to effectivelly apply the above configuration.
+
+NOTE: When MIGRATION is defined globally, new tables creation is not allowed.
+
+Once cluster has completely started and all regions have already become online, *MIGRATION* tracker
+can be disabled and the _new_ implementation should be the one set in *hbase.store.file-tracker.impl*.
+On the above example, the new configuration would be:
+
+----
+<property>
+  <name>hbase.store.file-tracker.impl</name>
+  <value>FILE</value>
+</property>
+----
+
+Restart the cluster again to complete migration and allow new tables creation to be executed again.
+
+### Configuring for Table or Column Family
+
+The previous example conveniently allows to set Store File Tracker desired configuration on a single
+place for all cluster tables. That may not be always desired, either because clusters restarts can be
+discouraging, or maybe because some user domain might be more critical to experiment a new feature.
+Whatever the reason, Store File Tracking can be set at Table or Column Family level configuration.
+For example, to specify *FILE* implementation in the table configuration at table creation time,
+the following should be applied:
+
+----
+create 'my-table', 'f1', 'f2', {CONFIGURATION => {'hbase.store.file-tracker.impl' => 'FILE'}}
+----
+
+To define *FILE* for an specific Column Family:
+
+----
+create 'my-table', {NAME=> '1', CONFIGURATION => {'hbase.store.file-tracker.impl' => 'FILE'}}
+----
+
+### Switching trackers at Table or Column Family
+
+Similarly to when switching implementations at global configuration, when switching _trackers_ for
+individual tables or column families, the *MIGRATION* tracker is also required. For example, to
+switch _tracker_ from *DEFAULT* to *FILE* in a table configuration:
+
+----
+alter 'my-table', CONFIGURATION => {'hbase.store.file-tracker.impl' => 'MIGRATION',
+'hbase.store.file-tracker.migration.src.impl' => 'DEFAULT',
+'hbase.store.file-tracker.migration.dst.impl' => 'FILE'}
+----
+
+To apply similar switch at column family level configuration:
+
+----
+alter 'my-table', {NAME => 'f1', CONFIGURATION => {'hbase.store.file-tracker.impl' => 'MIGRATION',
+'hbase.store.file-tracker.migration.src.impl' => 'DEFAULT',
+'hbase.store.file-tracker.migration.dst.impl' => 'FILE'}}
+----
+
+Once all table regions have been onlined again, don't forget to disable MIGRATION, by now setting
+*hbase.store.file-tracker.migration.dst.impl* value as the *hbase.store.file-tracker.impl*. In the above
+example, that would be as follows:
+
+----
+alter 'my-table', CONFIGURATION => {'hbase.store.file-tracker.impl' => 'FILE'}
+----

src/main/asciidoc/book.adoc

@@ -89,6 +89,7 @@ include::_chapters/zookeeper.adoc[]
 include::_chapters/community.adoc[]
 include::_chapters/hbtop.adoc[]
 include::_chapters/tracing.adoc[]
+include::_chapters/store_file_tracking.adoc[]
 
 = Appendix