Add TLDR example to USAGE.md and full example in usage-example/

USAGE.md

@@ -15,6 +15,7 @@ relevant artifacts for their unique set of data sources.
 
 ## Table of Contents
 
+- [TLDR Example](#tldr-example)
 - [Terminology](#terminology)
 - [Setup and Install](#setup-and-install)
   * [Prerequisites](#prerequisites)
@@ -33,6 +34,34 @@ relevant artifacts for their unique set of data sources.
     + [Strict Mode](#strict-mode)
     + [Intermediate-Only](#intermediate-only)
 
+## TLDR Example
+
+Before diving into the details, here's a complete example that:
+
+* takes ECS 1.6 fields
+* selects only the subset of fields relevant to the project's use case
+* includes custom fields relevant to the project
+* outputs the resulting artifacts to a project directory
+* replace the ECS project's sample template settings and
+  mapping settings with ones appropriate to the project
+
+```bash
+python scripts/generator.py --ref v1.6.0 \
+  --subset            ../my-project/fields/subset.yml \
+  --include           ../my-project/fields/custom/ \
+  --out               ../my-project/ \
+  --template-settings ../my-project/fields/template-settings.json \
+  --mapping-settings  ../my-project/fields/mapping-settings.json
+```
+
+The generated Elasticsearch template would be output at
+
+`my-project/generated/elasticsearch/7/template.json`
+
+If this sounds interesting, read on to learn all about each of these settings.
+
+See [usage-example/](usage-example/) for a complete example with source files.
+
 ## Terminology
 
 | Term | Definition |
@@ -80,6 +109,9 @@ $ make ve
 
 All necessary Python dependencies will also be installed with `pip`.
 
+You can use the Python and dependencies from this isolated virtual environment
+by using `build/ve/bin/python` instead of `python` in the examples shown here.
+
 #### Option 2: Install dependencies via pip
 
 Install dependencies using `pip` (An active `virutalenv` is recommended):
@@ -255,40 +287,40 @@ The `--template-settings` argument defines [index level settings](https://www.el
 
 ```json
 {
-    "index_patterns": ["ecs-*"],
-    "order": 1,
-    "settings": {
-        "index": {
-            "mapping": {
-                "total_fields": {
-                    "limit": 10000
-                }
-            },
-            "refresh_interval": "10s"
+  "index_patterns": ["mylog-*"],
+  "order": 1,
+  "settings": {
+    "index": {
+      "mapping": {
+        "total_fields": {
+          "limit": 10000
         }
-    },
-    "mappings": {}
+      },
+      "refresh_interval": "1s"
+    }
+  },
+  "mappings": {}
 }
 ```
 
 `--mapping-settings` works in the same way except now with the [mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) settings for the index. This is an example `mapping.json` file:
 
 ```json
 {
-    "_meta": {
-        "version": "1.5.0"
-    },
+  "_meta": {
+    "version": "1.5.0"
+  },
     "date_detection": false,
     "dynamic_templates": [
-        {
-            "strings_as_keyword": {
-                "mapping": {
-                    "ignore_above": 1024,
-                    "type": "keyword"
-                },
-                "match_mapping_type": "string"
-            }
+      {
+        "strings_as_keyword": {
+          "mapping": {
+            "ignore_above": 1024,
+            "type": "keyword"
+          },
+          "match_mapping_type": "string"
         }
+      }
     ],
     "properties": {}
 }

scripts/generator.py

@@ -35,7 +35,6 @@ def main():
         default_dirs = True
 
     ecs_helpers.make_dirs(out_dir)
-    ecs_helpers.make_dirs(docs_dir)
 
     # To debug issues in the gradual building up of the nested structure, insert
     # statements like this after any step of interest.
@@ -58,6 +57,7 @@ def main():
     if args.include or args.subset:
         exit()
 
+    ecs_helpers.make_dirs(docs_dir)
     asciidoc_fields.generate(nested, ecs_version, docs_dir)
 
 

usage-example/README.md

@@ -0,0 +1,19 @@
+# Concrete usage example
+
+This directory contains a full example of using the ECS tools to manage your
+project's index template (or your Beats field definitions) with the ECS tools.
+
+The `fields` directory contains files managed by the sample project, whereas
+the `generated` directory contains the files generated by running the following
+command from the root of the ECS repository:
+
+```bash
+python scripts/generator.py --ref v1.6.0 \
+  --subset            usage-example/fields/subset.yml \
+  --include           usage-example/fields/custom/ \
+  --out               usage-example/ \
+  --template-settings usage-example/fields/template-settings.json \
+  --mapping-settings  usage-example/fields/mapping-settings.json
+```
+
+Refer back to [USAGE.md](../USAGE.md) for the documentation on each of these flags.

usage-example/fields/custom/acme.yml

@@ -0,0 +1,11 @@
+- name: acme
+  title: ACME
+  description: >
+    Acme Inc. custom fields
+  type: group
+  fields:
+    - name: account.id
+      type: keyword
+      level: custom
+      description: >
+        Customer account for this activity.

usage-example/fields/mapping-settings.json

@@ -0,0 +1,20 @@
+{
+  "_meta": {
+    "version": "1.6.0"
+  },
+  "date_detection": false,
+  "dynamic_templates": [
+    {
+      "strings_as_keyword": {
+        "mapping": {
+          "type": "keyword",
+          "ignore_above": 1024,
+          "fields": {
+            "text": { "type": "text", "norms" : false }
+          }
+        },
+        "match_mapping_type": "string"
+      }
+    }
+  ]
+}

usage-example/fields/subset.yml

@@ -0,0 +1,45 @@
+---
+name: web_logs
+fields:
+  # Project's custom fields
+  acme:
+    fields: "*"
+
+  # ECS basic fields
+  base:
+    fields: "*"
+  ecs:
+    fields: "*"
+  event:
+    fields: "*"
+
+  # web log specific fields
+  user_agent:
+    fields: "*"
+  url:
+    fields: "*"
+  http:
+    fields: "*"
+  user: # mapping url.username to user.name
+    fields:
+      "name": {}
+
+  # Network fields to capture IPs, geo and stuff
+  network:
+    fields: "*"
+  related:
+    fields:
+      ip: {}
+      user: {}
+  source:
+    fields: "*"
+  destination:
+    fields: "*"
+  client:
+    fields: "*"
+  server:
+    fields: "*"
+
+  # pipeline meta-data
+  agent:
+    fields: "*"

usage-example/fields/template-settings.json

@@ -0,0 +1,16 @@
+{
+  "index_patterns": ["acme-weblogs-*"],
+  "order": 1,
+  "settings": {
+    "index": {
+      "codec" : "best_compression",
+      "mapping": {
+        "total_fields": {
+          "limit": 1000
+        }
+      },
+      "refresh_interval": "2s"
+    }
+  }
+}
+