diff --git a/USAGE.md b/USAGE.md index aad055a92b..a1a59ddbc7 100644 --- a/USAGE.md +++ b/USAGE.md @@ -1,163 +1,103 @@ # ECS Tooling Usage -In addition to the published schema and artifacts, the ECS repo also contains tools to generate artifacts based on the current published and custom schemas. +In addition to the published schema and artifacts, the ECS repo contains tools to generate artifacts based on ECS schemas and your custom field definitions. -You may be asking if ECS is a specification for storing event data, where does the ECS tooling fit into the picture? As users implement ECS into their Elastic stack, common questions arise: +## Why Use ECS Tooling? -* ECS has too many fields. Users don't want to generate mappings for fields they don't plan on using soon. -* Users want to adopt ECS but also want to painlessly maintain their own custom field mappings alongside ECS. +* **Subset Generation**: ECS has ~850 fields. Generate mappings for only the fields you need. +* **Custom Fields**: Painlessly maintain your own custom field mappings alongside ECS. +* **Multiple Formats**: Generate Elasticsearch templates, Beats configs, CSV exports, and documentation. -Users can use the ECS tools to tackle both problems. What artifacts are relevant will also vary based on need. Many users will find the Elasticsearch templates most useful, but Beats -contributors will instead find the Beats-formatted YAML field definition files valuable. By maintaining only their customizations and use the tools provided by ECS, they can generate -relevant artifacts for their unique set of data sources. +**For detailed developer documentation**, see [scripts/docs/README.md](scripts/docs/README.md). **NOTE** - These tools and their functionality are considered experimental. ## Table of Contents -- [TLDR Example](#tldr-example) -- [Terminology](#terminology) +- [Quick Start Example](#quick-start-example) - [Setup and Install](#setup-and-install) - * [Prerequisites](#prerequisites) - + [Clone from GitHub](#clone-from-github) - + [Install dependencies](#install-dependencies) -- [Usage](#usage) - * [Getting Started - Generating Artifacts](#getting-started---generating-artifacts) - * [Generator Options](#generator-options) - + [Out](#out) - + [Include](#include) - + [Exclude](#exclude) - + [Subset](#subset) - + [Ref](#ref) - + [Mapping & Template Settings](#mapping--template-settings) - + [Strict Mode](#strict-mode) - + [Intermediate-Only](#intermediate-only) - + [Force-docs](#force-docs) - -## TLDR Example - -Before diving into the details, here's a complete example that: - -* takes ECS 1.6 fields -* selects only the subset of fields relevant to the project's use case -* includes custom fields relevant to the project -* outputs the resulting artifacts to a project directory -* replace the ECS project's sample template settings and - mapping settings with ones appropriate to the project +- [Basic Usage](#basic-usage) +- [Key Generator Options](#key-generator-options) + * [Include Custom Fields](#include-custom-fields) + * [Subset - Use Only Needed Fields](#subset---use-only-needed-fields) + * [Ref - Target Specific ECS Version](#ref---target-specific-ecs-version) + * [Other Options](#other-options) +- [Additional Resources](#additional-resources) + +## Quick Start Example + +Here's a complete example that generates artifacts with: +* ECS 9.1 fields as the base +* A subset of only needed fields +* Custom fields added on top +* Custom template settings ```bash -python scripts/generator.py --ref v1.6.0 \ - --semconv-version $(cat otel-semconv-version) \ - --subset ../my-project/fields/subset.yml \ - --include ../my-project/fields/custom/ \ - --out ../my-project/ \ - --template-settings-legacy ../my-project/fields/template-settings-legacy.json \ - --template-settings ../my-project/fields/template-settings.json \ - --mapping-settings ../my-project/fields/mapping-settings.json +python scripts/generator.py \ + --ref v9.1.0 \ + --semconv-version v1.38.0 \ + --subset ../my-project/fields/subset.yml \ + --include ../my-project/fields/custom/ \ + --out ../my-project/ ``` -The generated Elasticsearch template would be output at +This generates: +* `my-project/generated/elasticsearch/composable/` - Modern Elasticsearch templates +* `my-project/generated/elasticsearch/legacy/` - Legacy templates +* `my-project/generated/beats/` - Beats field definitions +* `my-project/generated/csv/` - CSV field reference -`my-project/generated/elasticsearch/legacy/template.json` - -If this sounds interesting, read on to learn all about each of these settings. - -## Terminology - -| Term | Definition | -| ---- | ---------- | -| ECS | Elastic Common Schema. For the purposes of this guide, ECS may refer to either the schema itself or the repo/tooling used to maintain the schema | -| artifacts | Various kinds of files or programs that can be generated based on ECS | -| field set | Groups of related fields in ECS | -| schema | Another term for a group of related fields in ECS. Used interchangeably with field set | -| schema definition | The markup used to define a schema in ECS | -| attributes | The properties of a field or field set that are used to define that field or field set in a schema definition | +**Note**: The `--semconv-version` flag is required. Use the version from the `otel-semconv-version` file or a specific version like `v1.38.0`. ## Setup and Install -### Prerequisites - -* [Python 3.8+](https://www.python.org/) -* [make](https://www.gnu.org/software/make/) -* [pip](https://pypi.org/project/pip/) -* [git](https://git-scm.com/) - -#### Clone from GitHub - -The recommended way to download the ECS repo is `git clone`: - -``` -$ git clone https://github.com/elastic/ecs -$ cd ecs -``` - -Prior to installing dependencies or running the tools, it's recommended to check out the `git` branch for the ECS version being targeted. - -**Example**: For ECS `1.5.0`: - -``` -$ git checkout v1.5.0 -``` +**Requirements**: Python 3.8+, git -#### Install dependencies +**Clone and setup**: -Install dependencies using `pip` (An active `virtualenv` is recommended): - -``` -$ pip install -r scripts/requirements.txt +```bash +git clone https://github.com/elastic/ecs +cd ecs +git checkout v9.1.0 # Optional: target specific version +pip install -r scripts/requirements.txt # virtualenv recommended ``` -## Usage - -### Getting Started - Generating Artifacts +## Basic Usage -Using the defaults, the [generator](scripts/generator.py) script generates the artifacts based on the [current](schemas) ECS schema. +Generate artifacts from the current ECS schema: +```bash +make generate +# or +python scripts/generator.py --semconv-version v1.38.0 ``` -$ python scripts/generator.py -Loading schemas from local files -Running generator. ECS version 1.5.0 -``` - -**Points to note on the defaults**: - -* Artifacts are created in the [`generated`](generated) directory and the entire schema is included -* Documentation updates will be written to the appropriate file under the `docs` directory. More specifics on generated doc files is covered in the [contributor's file](https://github.com/elastic/ecs/blob/main/CONTRIBUTING.md#generated-documentation-files) -* Each run of the script will rewrite the entirety of the `generated` directory -* The script will need to be executed from the top-level of the ECS repo -* The `version` displayed when running `generator.py` is based on the current value of the [version](version) file in the top-level of the repo - -The generator's defaults are how the ECS team maintains the official artifacts published in the repo. For your own use cases, you may wish to add your own fields or remove others that are unused. The following section details the available options for controlling the output of those artifacts. -### Generator Options - -#### Out - -Generate the ECS artifacts in a different output directory. If the specified directory doesn't exist, it will be created: - -``` -$ python scripts/generator.py --out ../myproject/ecs/out/ -``` +**Key points**: +* Artifacts are created in the `generated/` directory +* Documentation is written to `docs/reference/` +* Each run rewrites the entire `generated/` directory +* Must be run from the ECS repo root +* The `--semconv-version` flag is **required** for OTel integration validation -Inside the directory passed in as the target dir to the `--out` flag, two directories, `generated` and `docs`, will be created. `docs` will contain three asciidoc files based on the contents of the provided schema. `generated` will contain the various artifacts laid out as in the published repo (`beats`, `csv`, `ecs`, `elasticsearch`). +**For complete documentation on how the generator works**, see: +* [scripts/docs/README.md](scripts/docs/README.md) - Complete developer documentation +* [scripts/docs/schema-pipeline.md](scripts/docs/schema-pipeline.md) - Pipeline details +* [scripts/generator.py](scripts/generator.py) - Comprehensive inline documentation -> Note: When running using either the `--subset` or `--include` options, the asciidoc files will _not_ be generated. +## Key Generator Options -#### Include +### Include Custom Fields -Use the `--include` flag to generate ECS artifacts based on the current ECS schema field definitions plus provided custom fields: +Add custom fields to ECS schemas: -``` -$ python scripts/generator.py --include ../myproject/ecs/custom-fields/ -$ python scripts/generator.py --include ../myproject/ecs/custom-fields/ ../myproject/ecs/more-custom-fields/ -$ python scripts/generator.py --include ../myproject/ecs/custom-fields/myprefix*.yml -$ python scripts/generator.py --include ../myproject/ecs/custom-fields/[some]*[re].yml -$ python scripts/generator.py --include ../myproject/ecs/custom-fields/myfile1.yml ../myproject/ecs/custom-fields/myfile2.yml +```bash +python scripts/generator.py \ + --semconv-version v1.38.0 \ + --include ../myproject/custom-fields/ \ + --out ../myproject/out/ ``` -The `--include` flag expects one or more directories or subsets of schema YAML files using the same [file format](https://github.com/elastic/ecs/tree/master/schemas#fields-supported-in-schemasyml) as the ECS schema files. This is useful for maintaining custom field definitions that are _outside_ of the ECS schema, but allows for merging the custom fields with the official ECS fields for your deployment. - -For example, if we defined the following schema definition in a file named `myproject/ecs/custom-fields/widget.yml`: +**Custom field format** - Use the same YAML format as ECS schemas: ```yaml --- @@ -165,191 +105,141 @@ For example, if we defined the following schema definition in a file named `mypr title: Widgets group: 2 short: Fields describing widgets - description: > - The widget fields describe a widget and all its widget-related details. + description: Widget-related fields type: group fields: - - name: id level: extended type: keyword short: Unique identifier of the widget - description: > - Unique identifier of the widget. -``` - -Multiple directory targets can also be provided: - -``` -$ python scripts/generator.py \ - --include ../myproject/custom-fields-A/ ../myproject/custom-fields-B \ - --out ../myproject/out/ -``` - -Generate artifacts using `--include` to load our custom definitions in addition to `--out` to place them in the desired output directory: - -``` -$ python scripts/generator.py --include ../myproject/custom-fields/ --out ../myproject/out/ -Loading schemas from local files -Running generator. ECS version 1.5.0 -Loading user defined schemas: ['../myproject/custom-fields/'] -``` - -We see the artifacts were generated successfully: - -``` -$ ls -lah ../myproject/out/ -total 0 -drwxr-xr-x 2 user ecs 64B Jul 8 13:12 docs -drwxr-xr-x 6 user ecs 192B Jul 8 13:12 generated -``` - -And looking at a specific artifact, `../myprojects/out/generated/elasticsearch/legacy/template.json`, we see our custom fields are included: - -```json -... - "widgets": { - "properties": { - "id": { - "ignore_above": 1024, - "type": "keyword" - } - } - } -... -``` - -Include can be used together with the `--ref` flag to merge custom fields into a targeted ECS version. See [`Ref`](#ref). - -> NOTE: The `--include` mechanism will not validate custom YAML files prior to merging. This allows for modifying existing ECS fields in a custom schema without having to redefine all the mandatory field attributes. - -#### Exclude - -Use the `--exclude` flag to generate ephemeral ECS artifacts based on the current ECS schema field definitions minus fields considered for removal, e.g. to assess impact of removing these. Warning! This is not the recommended route to remove a field permanently as it is not intended to be invoked during the build process. Definitive field removal should be implemented using a custom [Subset](#subset) or via the [RFC process](https://github.com/elastic/ecs/tree/main/rfcs/README.md). Example: - -``` -$ python scripts/generator.py --exclude ../myproject/ecs/custom-fields/ -$ python scripts/generator.py --exclude ../myproject/ecs/custom-fields/ ../myproject/ecs/more-custom-fields/ -$ python scripts/generator.py --exclude ../myproject/ecs/custom-fields/myprefix*.yml -$ python scripts/generator.py --exclude ../myproject/ecs/custom-fields/[some]*[re].yml -$ python scripts/generator.py --exclude ../myproject/ecs/custom-fields/myfile1.yml ../myproject/ecs/custom-fields/myfile2.yml -``` - -The `--exclude` flag expects one or more directories or subsets of schema YAML files using the same [file format](https://github.com/elastic/ecs/tree/master/schemas#fields-supported-in-schemasyml) as the ECS schema files. You can also use a subset, provided that relevant `name` and `fields` fields are preserved. - -``` ---- -- name: log - fields: - - name: original -``` - -The root Field Set `name` must always be present and specified with no dots `.`. Subfields may be specified using dot notation, for example: - -``` ---- -- name: log - fields: - - name: syslog.severity.name + description: Unique identifier of the widget. ``` -Generate artifacts using `--exclude` to load our custom definitions in addition to `--out` to place them in the desired output directory: +**Supports**: Directories, multiple paths, wildcards (`*.yml`), combining with `--ref` -``` -$ python scripts/generator.py --exclude ../myproject/exclude-set.yml/ --out ../myproject/out/ -Loading schemas from local files -Running generator. ECS version 1.11.0 -``` +**See also**: [Schema format documentation](https://github.com/elastic/ecs/tree/main/schemas#fields-supported-in-schemasyml) -#### Subset +### Subset - Use Only Needed Fields -If your indices will never populate particular ECS fields, there's no need to include those field definitions in your index mappings, with the exception of the `base` fieldset, which must exist and which must contain at least the `@timestamp` field. The `--subset` argument allows for passing a subset definition YAML file which indicates which field sets or specific fields to include in the generated artifacts. +Generate artifacts with only the fields you need (reduces mapping size): -``` -$ python scripts/generator.py --subset ../myproject/ecs/subset-fields/ -$ python scripts/generator.py --subset ../myproject/ecs/subset-fields/ ../myproject/ecs/more-subset-fields/ -$ python scripts/generator.py --subset ../myproject/ecs/custom-fields/subset.yml -$ python scripts/generator.py --subset ../myproject/ecs/custom-fields/[some]*[re].yml -$ python scripts/generator.py --subset ../myproject/ecs/custom-fields/myfile1.yml ../myproject/ecs/custom-fields/myfile2.yml +```bash +python scripts/generator.py \ + --semconv-version v1.38.0 \ + --subset ../myproject/subset.yml ``` -Example subset file: +**Example subset file**: ```yaml --- -name: malware_event +name: web_logs fields: base: fields: "@timestamp": {} - agent: - fields: "*" - dll: - fields: "*" - ecs: - fields: "*" - process: + http: + fields: "*" # All http fields + url: + fields: "*" # All url fields + user_agent: fields: - same_as_process: - docs_only: True + original: {} # Specific fields only ``` -The subset file has a defined format, starting with the two top-level required fields: +**Subset format**: +* `name`: Subset name (used for output directory) +* `fields`: Declares which fieldsets/fields to include + * `fields: "*"` - Include all fields in fieldset + * `field_name: {}` - Include specific field + * `docs_only: true` - Include in docs only, not artifacts -* `name`: The name of the subset. Also used to name the directory holding the generated subset intermediate files (e.g. `/generated/ecs/subset/`) -* `fields` Contains the subset field filters +**Tips**: +* Combine with `--include` for custom fields (they must be listed in subset) +* Always include `base` fieldset with at least `@timestamp` -The `fields` object declares which fields to include: +**For detailed subset documentation with examples**, see [scripts/docs/schema-pipeline.md](scripts/docs/schema-pipeline.md#subset-filtering) -* The targeted field sets are declared underneath `fields` by their top-level name (e.g. `base`, `agent`, etc.) -* Underneath each field set, all sub-fields can be captured using a wildcard syntax: `fields: "*"` -* Individual leafs fields can also be targeted: `@timestamp: {}` -* For special cases, the `docs_only: True` attribute will add a field into `./docs` but not add into any other generated - artifact. The only current use case for this feature is to document fields only populated in reused field sets. +### Ref - Target Specific ECS Version -Reviewing the above example, the generator using subset will output artifacts containing: +Generate artifacts from a specific ECS version: -* The `@timestamp` field from the `base` field set -* All `agent.*` fields, `dll.*`, and `ecs.*` fields - -It's also possible to combine `--include` and `--subset` together! Do note that your subset YAML filter file will need to list any custom fields being passed with `--include`. Otherwise, `--subset` will filter those fields out. - -#### Ref +```bash +python scripts/generator.py \ + --semconv-version v1.38.0 \ + --ref v9.0.0 +``` -The `--ref` argument allows for passing a specific `git` tag (e.g. `v1.5.0`) or commit hash (`1454f8b`) that will be used to build ECS artifacts. +**Combines with other options**: -``` -$ python scripts/generator.py --ref v1.5.0 +```bash +# Generate from ECS v9.0.0 + experimental + custom fields +python scripts/generator.py \ + --semconv-version v1.38.0 \ + --ref v9.0.0 \ + --include experimental/schemas ../myproject/fields/custom ``` -The `--ref` argument loads field definitions from the specified git reference (branch, tag, etc.) from directories [`./schemas`](./schemas) and [`./experimental/schemas`](./experimental/schemas) (when specified via `--include`). +Loads schemas from git history (tags, branches, commits). Requires git. -Here's another example loading both ECS fields and [experimental](experimental/README.md) changes *from branch "1.7"*, then adds custom fields on top. +### Other Options +**`--out `** - Output to custom directory +```bash +python scripts/generator.py --semconv-version v1.38.0 --out ../myproject/ ``` -$ python scripts/generator.py --ref 1.7 --include experimental/schemas ../myproject/fields/custom --out ../myproject/out -``` - -The command above will produce artifacts based on: -* main ECS field definitions as of branch 1.7 -* experimental ECS changes as of branch 1.7 -* custom fields in `../myproject/fields/custom` as they are on the filesystem +**`--exclude `** - Remove specific fields (for testing deprecation impact) +```bash +python scripts/generator.py --semconv-version v1.38.0 --exclude deprecated-fields.yml +``` -> Note: `--ref` does have a dependency on `git` being installed and all expected commits/tags fetched from the ECS upstream repo. This will unlikely be an issue unless you downloaded the ECS as a zip archive from GitHub vs. cloning it. +**`--strict`** - Enable strict validation (required for CI/CD) +```bash +python scripts/generator.py --semconv-version v1.38.0 --strict +``` -#### Mapping & Template Settings +Strict mode requires the following conditions, else the script exits on an exception: -The `--template-settings-legacy` / `--template-settings` and `--mapping-settings` arguments allow overriding the default template and mapping settings, respectively, in the generated Elasticsearch template artifacts. Both artifacts expect a JSON file which contains custom settings defined. +* Short descriptions must be less than or equal to 120 characters. +* Example values containing arrays or objects must be quoted to avoid unexpected YAML interpretation when the schema files or artifacts are relied on downstream. +* If a regex `pattern` is defined, the example values will be checked against it. +* If `expected_values` is defined, the example value(s) will be checked against the list. +Example error when running with `--strict`: +``` +$ python scripts/generator.py --ref v1.4.0 --semconv-version v1.38.0 --strict +Loading schemas from git ref v1.4.0 +Running generator. ECS version 1.4.0 +... +ValueError: Short descriptions must be single line, and under 120 characters (current length: 134). +Offending field or field set: number +Short description: + Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet. ``` -$ python scripts/generator.py --template-settings-legacy ../myproject/es-overrides/template.json --mapping-settings ../myproject/es-overrides/mappings.json + +Without `--strict`, the same issue produces a warning and the script continues: ``` +$ python scripts/generator.py --ref v1.4.0 --semconv-version v1.38.0 +Loading schemas from git ref v1.4.0 +Running generator. ECS version 1.4.0 +~/dev/ecs/scripts/generators/ecs_helpers.py:176: UserWarning: Short descriptions must be single line, and under 120 characters (current length: 134). +Offending field or field set: number +Short description: + Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet. -The `--template-settings-legacy` argument defines [index level settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings) that will be applied to the legacy index template in the generated artifacts. The `--template-settings` argument now defines those same settings, but for the composable template in the generated artifacts. +This will cause an exception when running in strict mode. +``` +**`--template-settings`** / **`--mapping-settings`** - Custom Elasticsearch template settings +```bash +python scripts/generator.py \ + --semconv-version v1.38.0 \ + --template-settings ../myproject/template.json \ + --mapping-settings ../myproject/mappings.json +``` -This is an example `template.json` to be passed with `--template-setting-legacy`: +This is an example `template.json` to be passed with `--template-settings-legacy`: ```json { @@ -371,7 +261,7 @@ This is an example `template.json` to be passed with `--template-setting-legacy` } ``` -`--mapping-settings` works in the same way except now with the [mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) settings for the index. This is an example `mapping.json` file: +This is an example `mapping.json` to be passed with `--mapping-settings`: ```json { @@ -394,61 +284,32 @@ This is an example `template.json` to be passed with `--template-setting-legacy` } ``` -For `template.json`, the `mappings` object is left empty: `{}`. Likewise the `properties` object remains empty in the `mapping.json` example. This will be filled in automatically by the script. - -#### Strict Mode - -The `--strict` argument enables "strict mode". Strict mode performs a stricter validation step against the schema's contents. - -Basic usage: - -``` -$ python scripts/generator.py --strict -``` - -Strict mode requires the following conditions, else the script exits on an exception: - -* Short descriptions must be less than or equal to 120 characters. -* Example values containing arrays or objects must be quoted to avoid unexpected YAML interpretation when the schema files or artifacts are relied on downstream. -* If a regex `pattern` is defined, the example values will be checked against it. -* If `expected_values` is defined, the example value(s) will be checked against the list. - -The current artifacts generated and published in the ECS repo will always be created using strict mode. However, older ECS versions (pre `v1.5.0`) will cause -an exception if attempting to generate them using `--strict`. This is due to schema validation checks introduced after that version was released. +The `mappings` object in `template.json` and the `properties` object in `mapping.json` are left empty — they will be filled in automatically by the script. -Example: +**`--intermediate-only`** - Generate only intermediate files (for debugging) -``` -$ python scripts/generator.py --ref v1.4.0 --strict -Loading schemas from git ref v1.4.0 -Running generator. ECS version 1.4.0 -... -ValueError: Short descriptions must be single line, and under 120 characters (current length: 134). -Offending field or field set: number -Short description: - Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet. -``` +**`--force-docs`** - Generate docs even with `--subset`/`--include`/`--exclude` -Removing `--strict` will display a warning message, but the script will finish its run successfully: +## Additional Resources -``` -$ python scripts/generator.py --ref v1.4.0 -Loading schemas from git ref v1.4.0 -Running generator. ECS version 1.4.0 -/Users/ericbeahan/dev/ecs/scripts/generators/ecs_helpers.py:176: UserWarning: Short descriptions must be single line, and under 120 characters (current length: 134). -Offending field or field set: number -Short description: - Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet. +### Complete Documentation -This will cause an exception when running in strict mode. -``` +* **[scripts/docs/README.md](scripts/docs/README.md)** - Developer documentation index +* **[scripts/docs/schema-pipeline.md](scripts/docs/schema-pipeline.md)** - Complete pipeline documentation with: + * Detailed field reuse explanation with visual examples + * Comprehensive subset filtering guide with real-world examples + * Troubleshooting section for common issues +* **[scripts/generator.py](scripts/generator.py)** - Comprehensive inline documentation -#### Intermediate-Only +### Module-Specific Guides -The `--intermediate-only` argument is used for debugging purposes. It only generates the ["intermediate files"](generated/ecs), `ecs_flat.yml` and `ecs_nested.yml`, without generating the rest of the artifacts. -More information on the different intermediate files can be found in the generated directory's [README](generated/README.md). +* [OTel Integration](scripts/docs/otel-integration.md) - OpenTelemetry mapping validation +* [Elasticsearch Templates](scripts/docs/es-template.md) - Template generation details +* [Beats Configs](scripts/docs/beats-generator.md) - Beats field definitions +* [CSV Export](scripts/docs/csv-generator.md) - CSV field reference +* [Markdown Docs](scripts/docs/markdown-generator.md) - Documentation generation -#### Force-docs +### Contributing -By default, running the generator with `--subset`, `--include`, or `--exclude` flags will not generate the ECS docs in the `docs` directory. Use `--force-docs` to force the documentation to generate -even if one of those flags is also present. +* [CONTRIBUTING.md](CONTRIBUTING.md) - Contribution guidelines +* [Schema Format](https://github.com/elastic/ecs/tree/main/schemas#fields-supported-in-schemasyml) - YAML field definition format diff --git a/docs/reference/ecs-artifacts.md b/docs/reference/ecs-artifacts.md index e5b089c51e..1aa94dfda9 100644 --- a/docs/reference/ecs-artifacts.md +++ b/docs/reference/ecs-artifacts.md @@ -8,7 +8,7 @@ applies_to: # Generated artifacts [ecs-artifacts] -ECS maintains a collection of artifacts which are generated based on the schema. Examples include Elasticsearch index templates, CSV, and Beats field mappings. The maintained artifacts can be found in the [ECS Github repo](https://github.com/elastic/ecs/blob/master/generated#artifacts-generated-from-ecs). +ECS maintains a collection of artifacts which are generated based on the schema. Examples include Elasticsearch index templates, CSV, and Beats field mappings. The maintained artifacts can be found in the [ECS Github repo](https://github.com/elastic/ecs/blob/main/generated#artifacts-generated-from-ecs). -Users can generate custom versions of these artifacts using the ECS project’s tooling. See the tooling [usage documentation](https://github.com/elastic/ecs/blob/master/USAGE.md) for more detail. +Users can generate custom versions of these artifacts using the ECS project’s tooling. See the tooling [usage documentation](https://github.com/elastic/ecs/blob/main/USAGE.md) for more detail. diff --git a/docs/reference/ecs-converting.md b/docs/reference/ecs-converting.md index aadb59bda6..ac10f0aa45 100644 --- a/docs/reference/ecs-converting.md +++ b/docs/reference/ecs-converting.md @@ -22,7 +22,7 @@ Before you start a conversion, be sure that you understand the basics below. Make sure you understand the distinction between Core and Extended fields, as explained in the [Guidelines and Best Practices](/reference/ecs-guidelines.md). -Core and Extended fields are documented in the [*ECS Field Reference*](/reference/ecs-field-reference.md) or, for a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv). +Core and Extended fields are documented in the [*ECS Field Reference*](/reference/ecs-field-reference.md) or, for a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv). ### An approach to mapping an existing implementation [ecs-conv] diff --git a/docs/reference/ecs-field-reference.md b/docs/reference/ecs-field-reference.md index 86467c1aeb..d0fe6143d8 100644 --- a/docs/reference/ecs-field-reference.md +++ b/docs/reference/ecs-field-reference.md @@ -16,7 +16,7 @@ ECS defines multiple groups of related fields. They are called "field sets". The All other field sets are defined as objects in Elasticsearch, under which all fields are defined. -For a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv). +For a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv). ## Field sets [ecs-fieldsets] diff --git a/docs/reference/ecs-user-usage.md b/docs/reference/ecs-user-usage.md index 7fbe044b20..047d94b6e3 100644 --- a/docs/reference/ecs-user-usage.md +++ b/docs/reference/ecs-user-usage.md @@ -345,5 +345,5 @@ Like the other fields in the [related](/reference/ecs-related.md) field set, `re ## Mapping examples [ecs-user-usage-mappings] -For examples of mapping events from various sources, you can look at [RFC 0007 in section Source Data](https://github.com/elastic/ecs/blob/master/rfcs/text/0007-multiple-users.md#source-data). +For examples of mapping events from various sources, you can look at [RFC 0007 in section Source Data](https://github.com/elastic/ecs/blob/main/rfcs/text/0007-multiple-users.md#source-data). diff --git a/docs/reference/index.md b/docs/reference/index.md index b1af3f00ae..d5b8b53ea3 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -41,5 +41,5 @@ ECS is a permissive schema. If your events have additional data that cannot be m ECS improvements are released following [Semantic Versioning](https://semver.org/). Major ECS releases are planned to be aligned with major Elastic Stack releases. -Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the [Contribution Guidelines](https://github.com/elastic/ecs/blob/master/CONTRIBUTING.md). +Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the [Contribution Guidelines](https://github.com/elastic/ecs/blob/main/CONTRIBUTING.md). diff --git a/scripts/docs/README.md b/scripts/docs/README.md new file mode 100644 index 0000000000..4341e9f712 --- /dev/null +++ b/scripts/docs/README.md @@ -0,0 +1,159 @@ +# ECS Scripts Developer Documentation + +This directory contains developer-focused documentation for the ECS generation scripts. + +## Purpose + +The ECS repository includes a comprehensive toolchain for generating various artifacts from schema definitions. These developer guides explain: + +- **How each component works** internally +- **Architecture and design decisions** +- **How to make changes** and extend functionality +- **Troubleshooting** common issues + +## Documentation Structure + +### Module-Specific Guides + +Each major generator module has its own detailed guide: + +- **[otel-integration.md](otel-integration.md)** - OpenTelemetry Semantic Conventions integration + - Validation of ECS ↔ OTel mappings + - Loading OTel definitions from GitHub + - Generating alignment summaries + +- **[markdown-generator.md](markdown-generator.md)** - Markdown documentation generation + - Rendering ECS schemas to human-readable docs + - Jinja2 template system and customization + - OTel alignment documentation + - Adding new page types + +- **[intermediate-files.md](intermediate-files.md)** - Intermediate file generation + - Flat and nested format representations + - Bridge between schema processing and artifact generation + - Top-level vs. reusable fieldsets + - Data structure reference + +- **[es-template.md](es-template.md)** - Elasticsearch template generation + - Composable vs. legacy template formats + - Field type mapping conversion + - Template customization and settings + - Installation and troubleshooting + +- **[csv-generator.md](csv-generator.md)** - CSV field reference generation + - Spreadsheet-compatible field export + - Column structure and multi-field handling + - Analysis and integration examples + - Usage in Excel, Google Sheets, databases + +- **[beats-generator.md](beats-generator.md)** - Beats field definition generation + - YAML field definitions for Elastic Beats + - Default field selection and allowlist + - Contextual naming and field groups + - Integration with Beat modules + +### Quick Reference + +For high-level usage information, see: +- **[../../USAGE.md](../../USAGE.md)** - User guide for running the generators +- **[../../CONTRIBUTING.md](../../CONTRIBUTING.md)** - Contribution guidelines + +## Scripts Overview + +The `scripts/` directory contains several key components: + +### Core Modules + +| Module | Purpose | Documentation | +|--------|---------|---------------| +| `generator.py` | **Main entry point** - orchestrates complete pipeline | Comprehensive docstrings in file | +| `generators/otel.py` | OTel integration and validation | [otel-integration.md](otel-integration.md) | +| `generators/markdown_fields.py` | Markdown documentation generation | [markdown-generator.md](markdown-generator.md) | +| `generators/intermediate_files.py` | Intermediate format generation | [intermediate-files.md](intermediate-files.md) | +| `generators/es_template.py` | Elasticsearch template generation | [es-template.md](es-template.md) | +| `generators/csv_generator.py` | CSV field reference export | [csv-generator.md](csv-generator.md) | +| `generators/beats.py` | Beats field definition generation | [beats-generator.md](beats-generator.md) | +| `generators/ecs_helpers.py` | Shared utility functions | See docstrings in file | + +### Schema Processing + +The schema processing pipeline transforms YAML schema definitions through multiple stages. See [schema-pipeline.md](schema-pipeline.md) for complete pipeline documentation. + +| Module | Purpose | Documentation | +|--------|---------|---------------| +| **Pipeline Overview** | Complete schema processing flow | **[schema-pipeline.md](schema-pipeline.md)** | +| `schema/loader.py` | Load and parse YAML schemas → nested structure | [schema-pipeline.md#1-loaderpy---schema-loading](schema-pipeline.md#1-loaderpy---schema-loading) | +| `schema/cleaner.py` | Validate, normalize, apply defaults | [schema-pipeline.md#2-cleanerpy---validation--normalization](schema-pipeline.md#2-cleanerpy---validation--normalization) | +| `schema/finalizer.py` | Perform field reuse, calculate names | [schema-pipeline.md#3-finalizerpy---field-reuse--name-calculation](schema-pipeline.md#3-finalizerpy---field-reuse--name-calculation) | +| `schema/visitor.py` | Traverse field hierarchies (visitor pattern) | [schema-pipeline.md#visitorpy---field-traversal](schema-pipeline.md#visitorpy---field-traversal) | +| `schema/subset_filter.py` | Filter to include only specified fields | [schema-pipeline.md#4-subset_filterpy---subset-filtering-optional](schema-pipeline.md#4-subset_filterpy---subset-filtering-optional) | +| `schema/exclude_filter.py` | Explicitly remove specified fields | [schema-pipeline.md#5-exclude_filterpy---exclude-filtering-optional](schema-pipeline.md#5-exclude_filterpy---exclude-filtering-optional) | + +### Types + +| Module | Purpose | +|--------|---------| +| `ecs_types/schema_fields.py` | Core ECS type definitions | +| `ecs_types/otel_types.py` | OTel-specific types | + +## Getting Started + +If you're new to the ECS generator codebase: + +1. **Start with the main orchestrator**: Read `generator.py` docstrings to understand the pipeline +2. **Understand schema processing**: Read [schema-pipeline.md](schema-pipeline.md) +3. **Pick a generator**: Choose a specific generator that interests you +4. **Read its documentation**: Start with the module-specific guide +5. **Explore the code**: Read the source with the guide as reference +6. **Run it**: Try generating artifacts to see it in action + +### Quick Command Reference + +```bash +# Standard generation (from local schemas) +python scripts/generator.py --semconv-version v1.24.0 + +# From specific git version +python scripts/generator.py --ref v8.10.0 --semconv-version v1.24.0 + +# With custom schemas +python scripts/generator.py --include custom/schemas/ --semconv-version v1.24.0 + +# Generate subset only +python scripts/generator.py --subset schemas/subsets/minimal.yml --semconv-version v1.24.0 + +# Strict validation mode +python scripts/generator.py --strict --semconv-version v1.24.0 + +# Intermediate files only (fast iteration) +python scripts/generator.py --intermediate-only --semconv-version v1.24.0 +``` + +See `generator.py` docstrings for complete argument documentation. + +## Contributing Documentation + +When adding or modifying generator code: + +1. **Update docstrings**: Add comprehensive Python docstrings to all functions and classes +2. **Update/create guide**: Ensure a markdown guide exists explaining the component +3. **Update this README**: Add links to new documentation +4. **Include examples**: Show practical usage examples +5. **Document edge cases**: Explain tricky parts and gotchas + +### Documentation Standards + +- **Python docstrings**: Use Google-style docstrings with Args, Returns, Raises, Examples +- **Markdown guides**: Include Overview, Architecture, Usage Examples, Troubleshooting +- **Code examples**: Should be runnable (or clearly marked as pseudocode) +- **Diagrams**: Use ASCII/Unicode diagrams for flow visualization +- **Tables**: Use markdown tables for structured comparisons + +## Questions? + +For questions about: +- **Using the tools**: See [USAGE.md](../../USAGE.md) or ask in the [Elastic community forums](https://discuss.elastic.co/) +- **Contributing**: See [CONTRIBUTING.md](../../CONTRIBUTING.md) +- **Architecture**: Read the relevant module guide in this directory +- **Bugs**: [Open an issue](https://github.com/elastic/ecs/issues) + diff --git a/scripts/docs/beats-generator.md b/scripts/docs/beats-generator.md new file mode 100644 index 0000000000..7abee657e0 --- /dev/null +++ b/scripts/docs/beats-generator.md @@ -0,0 +1,488 @@ +# Beats Field Definition Generator + +## Overview + +The Beats Generator (`generators/beats.py`) creates field definitions for Elastic Beats in YAML format. Beats (Filebeat, Metricbeat, Packetbeat, Winlogbeat, etc.) are lightweight data shippers that need field definitions to validate data structure, configure field behavior, and provide user documentation. + +### Purpose + +Beats are Elastic's lightweight data collection agents that ship data to Elasticsearch or Logstash. They need field definitions to: + +1. **Validate Data** - Ensure collected data matches expected structure +2. **Configure Behavior** - Control indexing, doc_values, multi-fields +3. **Document Fields** - Provide field reference to users +4. **Manage Defaults** - Determine which fields are enabled by default + +The challenge: Beats can't load all ~850 ECS fields by default due to memory and performance constraints. The generator uses an allowlist to mark essential fields as `default_field: true`, while keeping others available but not loaded by default. + +## Architecture + +### High-Level Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ generator.py (main) │ +│ │ +│ Load → Clean → Finalize → Generate Intermediate Files │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ intermediate_files.generate() │ +│ │ +│ Returns: (nested, flat) │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ nested structure +┌─────────────────────────────────────────────────────────────────┐ +│ beats.generate() │ +│ │ +│ 1. Filter non-root fieldsets │ +│ 2. Process 'base' fieldset (fields at root) │ +│ 3. Process other fieldsets (as groups or root) │ +│ 4. Load default_fields allowlist │ +│ 5. Set default_field flags recursively │ +│ 6. Wrap in 'ecs' top-level group │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Output: fields.ecs.yml │ +│ │ +│ - key: ecs │ +│ title: ECS │ +│ fields: │ +│ - name: '@timestamp' │ +│ type: date │ +│ default_field: true │ +│ - name: agent │ +│ type: group │ +│ default_field: true │ +│ fields: │ +│ - name: id │ +│ type: keyword │ +│ default_field: true │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. generate() + +**Entry Point**: `generate(ecs_nested, ecs_version, out_dir)` + +Orchestrates the entire generation process: +- Filters fieldsets (removes top_level=false) +- Processes base fieldset first +- Processes other fieldsets as groups or root fields +- Applies default_field settings +- Writes YAML output + +#### 2. fieldset_field_array() + +**Purpose**: Convert ECS fields to Beats format + +**Transformations**: +- Filter to Beats-relevant properties +- Convert field names to contextual (relative) names +- Process multi-fields +- Sort fields alphabetically + +**Example**: +``` +ECS field name: http.request.method +Beats name (in http group): request.method +``` + +#### 3. set_default_field() + +**Purpose**: Mark fields that should be loaded by default + +**Logic**: +- Reads allowlist from `beats_default_fields_allowlist.yml` +- Recursively applies default_field flags +- Groups inherit and propagate settings +- Multi-fields inherit from parent field + +#### 4. write_beats_yaml() + +**Purpose**: Save formatted YAML with warning header + +## Beats YAML Structure + +### Top-Level Structure + +```yaml +# WARNING! Do not edit this file directly... + +- key: ecs + title: ECS + description: ECS Fields. + fields: + - name: '@timestamp' + type: date + default_field: true + description: Date/time when the event originated + + - name: agent + type: group + default_field: true + description: Agent fields + fields: + - name: id + type: keyword + default_field: true + description: Unique agent identifier +``` + +### Field Groups + +Field sets become groups in Beats: + +```yaml +- name: http + type: group + default_field: false # Group not default + title: HTTP + description: Fields related to HTTP activity + fields: + - name: request.method + type: keyword + default_field: true # But some fields within are + description: HTTP request method + + - name: request.bytes + type: long + default_field: false # Others are not + description: Request size in bytes +``` + +### Contextual Naming + +Beats uses relative field names within groups: + +| ECS Full Name | Beats Group | Beats Field Name | +|---------------|-------------|------------------| +| @timestamp | (root) | @timestamp | +| agent.id | agent | id | +| http.request.method | http | request.method | +| user.email | user | email | + +### Multi-Fields + +Multi-fields follow the same structure: + +```yaml +- name: message + type: match_only_text + default_field: true + description: Log message + multi_fields: + - name: text + type: match_only_text + default_field: true +``` + +### Root vs Group Fields + +**Root Fields** (root=true in schema): +- Appear directly in top-level fields array +- No group wrapper +- Example: base fieldset fields + +**Group Fields** (root=false or not specified): +- Wrapped in group with metadata +- Nested under group's fields array +- Example: http, user, process fieldsets + +## Default Fields Concept + +### The Challenge + +Beats face a trade-off: +- **More fields** = More memory/CPU usage +- **Fewer fields** = Less data captured + +All ~850 ECS fields would consume too many resources for many use cases. + +### The Solution: default_field + +Fields marked `default_field: true` are: +- Loaded by Beat on startup +- Available for immediate use +- Included in index mappings + +Fields marked `default_field: false`: +- Not loaded by default +- Can be enabled in Beat configuration +- Won't appear in index unless explicitly enabled + +### Allowlist File + +`beats_default_fields_allowlist.yml` contains ~400 essential fields: + +```yaml +!!set +# Core timestamp +'@timestamp': null + +# Essential agent fields +agent.id: null +agent.name: null +agent.type: null +agent.version: null + +# Common network fields +client.ip: null +client.port: null +server.ip: null +server.port: null + +# Essential event categorization +event.kind: null +event.category: null +event.type: null +event.outcome: null + +# Common message/log fields +message: null +log.level: null +... +``` + +### Inheritance Rules + +**Groups**: +- Top-level groups: `default_field: true` +- Nested groups: Inherit from parent + +**Fields**: +- In allowlist: `default_field: true` +- Parent is default: Children are default +- Otherwise: `default_field: false` + +**Multi-fields**: +- Always inherit from parent field + +## Usage Examples + +See [README.md](README.md) for generator invocation commands. + +### Programmatic Usage + +```python +from generators.beats import generate +from generators.intermediate_files import generate as gen_intermediate + +# Generate intermediate files +nested, flat = gen_intermediate(fields, 'generated/ecs', True) + +# Generate Beats fields +generate(nested, '8.11.0', 'generated') +# Creates generated/beats/fields.ecs.yml +``` + +### Loading in Beat Module + +```yaml +# In a Beat module (e.g., Filebeat module) +# Reference the generated file: + +--- +- name: http + type: group + description: HTTP fields from ECS + fields: + !include ../../../generated/beats/fields.ecs.yml +``` + +### Checking default_field Settings + +```python +import yaml + +with open('generated/beats/fields.ecs.yml') as f: + beats_def = yaml.safe_load(f) + +def count_default_fields(fields, count={'default': 0, 'non_default': 0}): + for field in fields: + if field.get('default_field', False): + count['default'] += 1 + else: + count['non_default'] += 1 + + if 'fields' in field: + count_default_fields(field['fields'], count) + if 'multi_fields' in field: + count_default_fields(field['multi_fields'], count) + + return count + +counts = count_default_fields(beats_def[0]['fields']) +print(f"Default fields: {counts['default']}") +print(f"Non-default fields: {counts['non_default']}") +``` + +## Making Changes + +### Adding Fields to Allowlist + +To make a field load by default in Beats: + +1. **Edit allowlist**: +```yaml +# beats_default_fields_allowlist.yml +# Add new field +new.field.name: null +``` + +2. **Regenerate**: +```bash +make clean +make SEMCONV_VERSION=v1.24.0 +``` + +3. **Verify**: +```bash +grep "name: field.name" generated/beats/fields.ecs.yml -A 1 +# Should show: default_field: true +``` + +### Removing Fields from Allowlist + +To stop a field from loading by default: + +1. Remove from `beats_default_fields_allowlist.yml` +2. Regenerate as above +3. Verify field now has `default_field: false` + +### Adding New Field Properties + +To include additional properties in Beats output: + +```python +def fieldset_field_array(...): + allowed_keys: List[str] = [ + 'name', + 'level', + # ... existing keys ... + 'new_property', # Add here + ] + # ... rest of function +``` + +### Changing Contextual Naming Logic + +To modify how field names are made relative: + +```python +def fieldset_field_array(...): + # Current logic + if '' == fieldset_prefix: + contextual_name = nested_field_name + else: + contextual_name = '.'.join(nested_field_name.split('.')[1:]) + + # Custom logic example: keep full names + contextual_name = nested_field_name + + # Or: different prefix handling + if fieldset_prefix: + contextual_name = nested_field_name.replace(fieldset_prefix + '.', '') +``` + +## Troubleshooting + +### Common Issues + +#### Fields missing default_field property + +**Symptom**: Some fields don't have `default_field` set + +**Check**: +```bash +# Count fields without default_field +grep -c "name:" generated/beats/fields.ecs.yml +grep -c "default_field:" generated/beats/fields.ecs.yml +# Should be equal (or close, accounting for structure) +``` + +**Solution**: Ensure `set_default_field()` is being called after field processing + +#### Allowlist changes not applying + +**Symptom**: Modified allowlist but field still has old default_field value + +**Solution**: +```bash +# Clean build directory +make clean + +# Regenerate from scratch +make SEMCONV_VERSION=v1.24.0 + +# Verify allowlist was loaded +grep "your.field.name" scripts/generators/beats_default_fields_allowlist.yml +grep "your.field.name" -A 1 generated/beats/fields.ecs.yml +``` + +#### Contextual names incorrect + +**Symptom**: Field names still show full ECS path instead of relative + +**Debug**: +```python +# In fieldset_field_array() +print(f"Field: {nested_field_name}") +print(f"Prefix: {fieldset_prefix}") +print(f"Contextual: {contextual_name}") +``` + +**Check**: +- Is fieldset_prefix being passed correctly? +- Is the split('.')[1:] logic working for your case? + +## Integration with Beats + +### In Beat Modules + +Beats modules include field definitions: + +```yaml +# module/http/access/_meta/fields.yml +- name: http + type: group + description: Fields related to HTTP + fields: + !include ../../../../../../generated/beats/fields.ecs.yml +``` + +### Loading Custom Fields + +Users can enable non-default fields: + +```yaml +# filebeat.yml +filebeat.modules: + - module: httpmodule + access: + enabled: true + var.additional_fields: + - http.request.body.bytes + - http.request.referrer +``` + +### Field Conflicts + +If a Beat defines custom fields with same names as ECS: +- ECS fields take precedence +- Merge is automatic +- Custom fields should use different names or namespaces + +## References + +- [Beats Documentation](https://www.elastic.co/guide/en/beats/libbeat/current/index.html) +- [Beats Developer Guide](https://www.elastic.co/guide/en/beats/devguide/current/index.html) +- [Beats Fields YAML Format](https://www.elastic.co/guide/en/beats/devguide/current/fields-yml.html) +- [ECS Beats Integration](https://www.elastic.co/guide/en/ecs/current/ecs-beats.html) + diff --git a/scripts/docs/csv-generator.md b/scripts/docs/csv-generator.md new file mode 100644 index 0000000000..12a98b58e9 --- /dev/null +++ b/scripts/docs/csv-generator.md @@ -0,0 +1,318 @@ +# CSV Field Reference Generator + +## Overview + +The CSV Generator (`generators/csv_generator.py`) produces a spreadsheet-compatible field reference for all ECS fields. It exports field definitions to a simple CSV (Comma-Separated Values) format that can be easily imported into spreadsheet applications, databases, or custom analysis tools. + +### Purpose + +This generator creates a human-readable, machine-parseable field catalog that's useful for: + +1. **Quick Reference** - Search and filter fields in Excel/Google Sheets +2. **Data Analysis** - Analyze field usage patterns and statistics +3. **Integration** - Parse for custom tooling and automation +4. **Documentation** - Include in presentations or reports +5. **Version Comparison** - Diff CSV files to see field changes + +The CSV format is intentionally simple and widely compatible, making ECS field data accessible to anyone with a spreadsheet application. + +## Architecture + +### High-Level Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ generator.py (main) │ +│ │ +│ Load → Clean → Finalize → Generate Intermediate Files │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ intermediate_files.generate() │ +│ │ +│ Returns: (nested, flat) │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ flat dictionary +┌─────────────────────────────────────────────────────────────────┐ +│ csv_generator.generate() │ +│ 1. base_first() - Sort fields (base fields first) │ +│ 2. save_csv() - Write CSV with header + field rows │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Output: fields.csv │ +│ │ +│ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization │ +│ 8.11.0,true,base,@timestamp,date,core,,2016-05-23... │ +│ 8.11.0,true,http,http.request.method,keyword,extended,... │ +│ ... │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. generate() + +**Entry Point**: `generate(ecs_flat, version, out_dir)` + +Orchestrates CSV generation: +- Creates output directory +- Sorts fields appropriately +- Writes CSV file + +#### 2. base_first() + +**Purpose**: Sort fields for readable output + +**Logic**: +1. Base fields (no dots): @timestamp, message, tags, etc. +2. All other fields alphabetically: agent.*, as.*, client.*, ... + +**Rationale**: Base fields are foundational and referenced frequently, so they appear at the top for easy access. + +#### 3. save_csv() + +**Purpose**: Write field data to CSV format + +**Features**: +- Header row with column names +- One row per field (plus multi-fields) +- Multi-fields get separate rows +- Consistent quoting and line endings + +## CSV Structure + +### Columns + +| Column | Description | Example Values | +|--------|-------------|----------------| +| **ECS_Version** | Version of ECS | 8.11.0, 8.11.0+exp | +| **Indexed** | Whether field is indexed | true, false | +| **Field_Set** | Fieldset name | base, http, user, agent | +| **Field** | Full dotted field name | @timestamp, http.request.method | +| **Type** | Elasticsearch field type | keyword, long, ip, date | +| **Level** | Field level | core, extended, custom | +| **Normalization** | Normalization rules | array, to_lower, array, to_lower | +| **Example** | Example value | GET, 192.0.2.1, 2016-05-23... | +| **Description** | Short field description | HTTP request method, User email | + +### Field Set Logic + +- **Base fields** (no dots in name): field_set = 'base' + - Examples: @timestamp, message, tags, labels +- **Other fields**: field_set = first part before dot + - http.request.method → field_set = 'http' + - user.email → field_set = 'user' + +### Multi-Fields + +Fields with multi-fields (alternate representations) get additional rows: + +```csv +8.11.0,true,event,message,match_only_text,core,,Hello world,Log message +8.11.0,true,event,message.text,match_only_text,core,,,Log message +``` + +Multi-field rows: +- Share version, indexed, field_set, level, description +- Have unique field name and type +- Have empty normalization and example + +## Example Output + +```csv +ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description +8.11.0,true,base,@timestamp,date,core,,2016-05-23T08:05:34.853Z,Date/time when the event originated +8.11.0,true,base,message,match_only_text,core,,Hello World,Log message optimized for viewing +8.11.0,true,base,message.text,match_only_text,core,,,Log message optimized for viewing +8.11.0,true,base,tags,keyword,core,array,"production, eu-west-1",List of keywords for event +8.11.0,true,agent,agent.build.original,keyword,core,,,Extended build information +8.11.0,true,agent,agent.ephemeral_id,keyword,extended,,8a4f500f,Ephemeral identifier +8.11.0,true,agent,agent.id,keyword,core,,8a4f500d,Unique agent identifier +8.11.0,true,http,http.request.body.bytes,long,extended,,1437,Request body size in bytes +8.11.0,true,http,http.request.method,keyword,extended,array,GET,HTTP request method +8.11.0,true,http,http.response.status_code,long,extended,,404,HTTP response status code +``` + +## Usage Examples + +See [README.md](README.md) for generator invocation commands. + +### Programmatic Usage + +```python +from generators.csv_generator import generate +from generators.intermediate_files import generate as gen_intermediate + +# Generate intermediate files +nested, flat = gen_intermediate(fields, 'generated/ecs', True) + +# Generate CSV +generate(flat, '8.11.0', 'generated') +# Creates generated/csv/fields.csv +``` + +### Analyzing Field Data + +**Count fields by type**: +```python +import csv +from collections import Counter + +with open('generated/csv/fields.csv') as f: + reader = csv.DictReader(f) + types = Counter(row['Type'] for row in reader) + +print("Field types:") +for field_type, count in types.most_common(): + print(f" {field_type}: {count}") +``` + +**Find all extended-level fields**: +```python +import csv + +with open('generated/csv/fields.csv') as f: + reader = csv.DictReader(f) + extended = [row for row in reader if row['Level'] == 'extended'] + +print(f"Extended fields: {len(extended)}") +for field in extended[:5]: + print(f" {field['Field']}") +``` + +**Fields by fieldset**: +```python +import csv +from collections import defaultdict + +with open('generated/csv/fields.csv') as f: + reader = csv.DictReader(f) + by_fieldset = defaultdict(list) + for row in reader: + by_fieldset[row['Field_Set']].append(row['Field']) + +for fieldset in sorted(by_fieldset): + print(f"{fieldset}: {len(by_fieldset[fieldset])} fields") +``` + +## Making Changes + +### Adding New Columns + +To add a new column to the CSV: + +1. **Update header row**: +```python +schema_writer.writerow([ + "ECS_Version", "Indexed", "Field_Set", "Field", + "Type", "Level", "Normalization", "Example", "Description", + "New_Column" # Add here +]) +``` + +2. **Add to data rows**: +```python +schema_writer.writerow([ + version, + indexed, + field_set, + field['flat_name'], + field['type'], + field['level'], + ', '.join(field['normalize']), + field.get('example', ''), + field['short'], + field.get('new_property', 'default_value') # Add here +]) +``` + +3. **Update multi-field rows** similarly + +4. **Update documentation** in this file + +### Changing Field Sorting + +To change sort order: + +```python +def base_first(ecs_flat: Dict[str, Field]) -> List[Field]: + # Custom sorting logic + fields_list = list(ecs_flat.values()) + + # Sort by level, then name + return sorted(fields_list, key=lambda f: (f['level'], f['flat_name'])) + + # Or by fieldset, then name + return sorted(fields_list, key=lambda f: (f['flat_name'].split('.')[0], f['flat_name'])) +``` + +### Changing CSV Format + +To modify CSV formatting: + +```python +schema_writer = csv.writer( + csvfile, + delimiter=';', # Use semicolon instead + quoting=csv.QUOTE_ALL, # Quote all fields + quotechar='"', + lineterminator='\r\n' # Windows line endings +) +``` + +### Filtering Fields + +To exclude certain fields: + +```python +def generate(ecs_flat: Dict[str, Field], version: str, out_dir: str) -> None: + ecs_helpers.make_dirs(join(out_dir, 'csv')) + + # Filter out custom fields + filtered = {k: v for k, v in ecs_flat.items() if v['level'] != 'custom'} + + sorted_fields = base_first(filtered) + save_csv(join(out_dir, 'csv/fields.csv'), sorted_fields, version) +``` + +## Troubleshooting + +### Common Issues + +#### Missing multi-fields + +**Symptom**: Multi-fields not appearing in CSV + +**Check**: +```python +# Verify field has multi_fields +field = flat['message'] +print('multi_fields' in field) +print(field.get('multi_fields')) + +# Check multi-field structure +if 'multi_fields' in field: + for mf in field['multi_fields']: + print(f" {mf['flat_name']}: {mf['type']}") +``` + +#### Empty normalization column + +**Symptom**: Normalization column is always empty + +**Check** field definitions have `normalize` key: +```python +field = flat['some.field'] +print(field.get('normalize', [])) # Should be a list +``` + +## References + +- [CSV Format Specification (RFC 4180)](https://tools.ietf.org/html/rfc4180) +- [Python csv Module Documentation](https://docs.python.org/3/library/csv.html) +- [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) + diff --git a/scripts/docs/es-template.md b/scripts/docs/es-template.md new file mode 100644 index 0000000000..266250f126 --- /dev/null +++ b/scripts/docs/es-template.md @@ -0,0 +1,525 @@ +# Elasticsearch Template Generator + +## Overview + +The Elasticsearch Template Generator (`generators/es_template.py`) converts ECS field schemas into Elasticsearch index templates. These templates define the mapping (field types and properties) for indices that will store ECS-structured data. + +### Purpose + +This generator bridges the gap between ECS schema definitions and Elasticsearch's native mapping format, producing ready-to-install JSON templates that: + +1. **Define field mappings** - Specify types, parameters, and multi-fields +2. **Configure index settings** - Set codec, field limits, refresh intervals +3. **Support two template formats**: + - **Composable** (modern): Modular component templates + - **Legacy** (deprecated): Single monolithic template + +The generated templates can be directly installed into Elasticsearch using the `_index_template` or `_template` APIs. + +## Architecture + +### High-Level Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ generator.py (main) │ +│ │ +│ Load → Clean → Finalize → Generate Intermediate Files │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ es_template.generate() / generate_legacy() │ +│ │ +│ Input: nested or flat fieldsets + version + settings │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ┌──────────────────┴──────────────────┐ + │ │ + ▼ ▼ +┌──────────────────────────┐ ┌──────────────────────────┐ +│ Composable Templates │ │ Legacy Template │ +│ │ │ │ +│ For each fieldset: │ │ All fields in one: │ +│ 1. Build nested props │ │ 1. Build nested props │ +│ 2. Convert fields │ │ 2. Convert fields │ +│ 3. Save component │ │ 3. Save single template │ +│ │ │ │ +│ Save main template │ └──────────────────────────┘ +└──────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Elasticsearch JSON Templates │ +│ │ +│ Composable: │ +│ - generated/elasticsearch/composable/component/base.json │ +│ - generated/elasticsearch/composable/component/agent.json │ +│ - generated/elasticsearch/composable/component/*.json │ +│ - generated/elasticsearch/composable/template.json │ +│ │ +│ Legacy: │ +│ - generated/elasticsearch/legacy/template.json │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. Composable Template Generation + +**Entry Point**: `generate(ecs_nested, ecs_version, out_dir, ...)` + +**Process**: +1. For each fieldset: + - Convert flat field names to nested `properties` structure + - Transform ECS field defs to Elasticsearch mappings + - Save as individual component template +2. Generate main template: + - Build component name list + - Create template that composes all components + - Add index patterns, priority, settings + +**Output Files**: +- `component/base.json`, `component/agent.json`, etc. (one per fieldset) +- `template.json` (main composable template) + +#### 2. Legacy Template Generation + +**Entry Point**: `generate_legacy(ecs_flat, ecs_version, out_dir, ...)` + +**Process**: +1. Iterate all fields in sorted order +2. Convert flat field names to nested properties structure +3. Build single monolithic mappings section +4. Generate template with all mappings included + +**Output File**: +- `legacy/template.json` + +#### 3. Field Mapping Conversion + +**Function**: `entry_for(field)` + +Converts ECS field definitions to Elasticsearch mapping format: + +| ECS Type | ES Mapping | Special Parameters | +|----------|------------|-------------------| +| keyword | keyword | ignore_above, synthetic_source_keep | +| text | text | norms | +| long/integer/short/byte | long/integer/short/byte | - | +| float/double/half_float | float/double/half_float | - | +| scaled_float | scaled_float | scaling_factor | +| boolean | boolean | - | +| date | date | - | +| ip | ip | - | +| geo_point | geo_point | - | +| object | object | enabled (if false) | +| nested | nested | enabled (if false) | +| flattened | flattened | ignore_above | +| constant_keyword | constant_keyword | value | +| alias | alias | path | + +**Multi-fields**: Handled via `multi_fields` array in ECS definition + +**Custom parameters**: Merged from `parameters` dict in field definition + +## Template Formats + +### Composable Template (Modern) + +Recommended for Elasticsearch 7.8+. Provides modularity and flexibility. + +**Component Template** (one per fieldset): +```json +{ + "template": { + "mappings": { + "properties": { + "http": { + "properties": { + "request": { + "properties": { + "method": { + "type": "keyword", + "ignore_above": 1024 + } + } + } + } + } + } + } + }, + "_meta": { + "ecs_version": "8.11.0", + "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-http.html" + } +} +``` + +**Main Template**: +```json +{ + "index_patterns": ["try-ecs-*"], + "composed_of": [ + "ecs_8.11.0_base", + "ecs_8.11.0_agent", + "ecs_8.11.0_http", + "..." + ], + "priority": 1, + "template": { + "settings": { + "index": { + "codec": "best_compression", + "mapping": { + "total_fields": { + "limit": 2000 + } + } + } + }, + "mappings": { + "date_detection": false, + "dynamic_templates": [...] + } + }, + "_meta": { + "ecs_version": "8.11.0", + "description": "Sample composable template that includes all ECS fields" + } +} +``` + +**Installation**: +```bash +# Install component templates +for file in generated/elasticsearch/composable/component/*.json; do + name=$(basename "$file" .json) + curl -X PUT "localhost:9200/_component_template/ecs_8.11.0_$name" \ + -H 'Content-Type: application/json' -d @"$file" +done + +# Install main template +curl -X PUT "localhost:9200/_index_template/ecs" \ + -H 'Content-Type: application/json' \ + -d @generated/elasticsearch/composable/template.json +``` + +### Legacy Template (Deprecated) + +For Elasticsearch < 7.8 or backwards compatibility. + +**Structure**: +```json +{ + "index_patterns": ["try-ecs-*"], + "order": 1, + "settings": { + "index": { + "mapping": { + "total_fields": { + "limit": 10000 + } + }, + "refresh_interval": "5s" + } + }, + "mappings": { + "_meta": { + "version": "8.11.0" + }, + "date_detection": false, + "dynamic_templates": [...], + "properties": { + "agent": {...}, + "http": {...}, + "...": "all fields in one place" + } + } +} +``` + +**Installation**: +```bash +curl -X PUT "localhost:9200/_template/ecs" \ + -H 'Content-Type: application/json' \ + -d @generated/elasticsearch/legacy/template.json +``` + +## Usage Examples + +See [README.md](README.md) for generator invocation commands. + +### Programmatic Usage + +```python +from generators.es_template import generate, generate_legacy +from generators.intermediate_files import generate as gen_intermediate + +# Generate intermediate files +nested, flat = gen_intermediate(fields, 'generated/ecs', True) + +# Generate composable templates +generate( + ecs_nested=nested, + ecs_version='8.11.0', + out_dir='generated', + mapping_settings_file=None, # Use defaults + template_settings_file=None # Use defaults +) + +# Generate legacy template +generate_legacy( + ecs_flat=flat, + ecs_version='8.11.0', + out_dir='generated', + mapping_settings_file=None, + template_settings_file=None +) +``` + +### Custom Settings + +**Custom Mapping Settings** (`mapping_settings.json`): +```json +{ + "date_detection": true, + "numeric_detection": false, + "dynamic_templates": [ + { + "strings_as_text": { + "match_mapping_type": "string", + "mapping": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } + } + } + ] +} +``` + +**Custom Template Settings** (`template_settings.json`): +```json +{ + "index_patterns": ["logs-*", "metrics-*"], + "priority": 100, + "template": { + "settings": { + "index": { + "number_of_shards": 1, + "number_of_replicas": 1, + "codec": "best_compression", + "mapping": { + "total_fields": { + "limit": 5000 + } + } + } + } + } +} +``` + +**Usage**: +```python +generate( + ecs_nested=nested, + ecs_version='8.11.0', + out_dir='generated', + mapping_settings_file='mapping_settings.json', + template_settings_file='template_settings.json' +) +``` + +## Making Changes + +### Adding Support for New Field Type + +To add a new Elasticsearch field type: + +1. **Update entry_for() function**: +```python +def entry_for(field: Field) -> Dict: + field_entry: Dict = {'type': field['type']} + try: + # ... existing type handling ... + + elif field['type'] == 'new_type': + ecs_helpers.dict_copy_existing_keys( + field, field_entry, + ['param1', 'param2'] # Type-specific parameters + ) + + # ... rest of function ... +``` + +2. **Update schema definitions** to use new type +3. **Test** with sample field +4. **Document** in this guide + +### Customizing Component Template Naming + +To change the naming convention for component templates: + +```python +def component_name_convention( + ecs_version: str, + ecs_nested: Dict[str, FieldNestedEntry] +) -> List[str]: + version: str = ecs_version.replace('+', '-') + names: List[str] = [] + for (fieldset_name, fieldset) in ecs_helpers.remove_top_level_reusable_false(ecs_nested).items(): + # Change naming pattern here + names.append("my_prefix_{}_{}".format(version, fieldset_name)) + return names +``` + +**Note**: If you change component names, update any deployment scripts that reference them. + +### Adding Custom Metadata + +To add custom metadata to templates: + +**For component templates**: +```python +def save_component_template(...): + # ... existing code ... + template['_meta']['custom_field'] = 'custom_value' + template['_meta']['team'] = 'security' + # ... save ... +``` + +**For main template**: +```python +def finalize_template(...): + # ... existing code ... + if not is_legacy: + template['_meta']['custom_info'] = {...} +``` + +### Modifying Default Settings + +To change default template settings: + +```python +def default_template_settings(ecs_version: str) -> Dict: + return { + "index_patterns": ["your-pattern-*"], # Change pattern + "priority": 500, # Higher priority + "template": { + "settings": { + "index": { + "number_of_shards": 1, # Add shard config + "codec": "default", # Change codec + "mapping": { + "total_fields": { + "limit": 5000 # Increase limit + } + } + } + }, + } + } +``` + +## Troubleshooting + +### Common Issues + +#### "Total fields limit exceeded" + +**Symptom**: Error when installing template or indexing documents + +``` +illegal_argument_exception: Limit of total fields [1000] has been exceeded +``` + +**Cause**: Elasticsearch default limit is 1000 fields, ECS has 800+ + +**Solutions**: +1. Increase limit in template settings: + ```json + { + "settings": { + "index": { + "mapping": { + "total_fields": { + "limit": 2000 + } + } + } + } + } + ``` + +2. Use composable templates (smaller field count per component) + +3. Use selective field sets (only fields you need) + +#### Component template not found + +**Symptom**: Error installing main composable template + +``` +resource_not_found_exception: component template [ecs_8.11.0_http] not found +``` + +**Cause**: Component templates must be installed before main template + +**Solution**: Install components first, then main template: +```bash +# Install all components +for file in generated/elasticsearch/composable/component/*.json; do + # ... install component +done + +# Then install main template +curl -X PUT "localhost:9200/_index_template/ecs" ... +``` + +#### Mapping conflicts + +**Symptom**: Cannot update mapping with different type + +``` +illegal_argument_exception: mapper [field] cannot be changed from type [keyword] to [text] +``` + +**Cause**: Trying to change existing field type + +**Solutions**: +1. Delete and recreate index: + ```bash + curl -X DELETE "localhost:9200/my-index" + # Recreate with new mapping + ``` + +2. Reindex to new index with updated mapping: + ```bash + curl -X POST "localhost:9200/_reindex" -d '{ + "source": {"index": "old-index"}, + "dest": {"index": "new-index"} + }' + ``` + +3. Use index aliases to transparently switch + +### Debugging Tips + +- **Compare versions**: `diff -u old/template.json new/template.json` +- **Test installation**: Use a local Elasticsearch instance with `docker run -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:8.11.0` + +## References + +- [Elasticsearch Composable Templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) +- [Elasticsearch Mapping Types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) +- [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) +- [Index Template Best Practices](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html#avoid-index-pattern-collisions) + diff --git a/scripts/docs/intermediate-files.md b/scripts/docs/intermediate-files.md new file mode 100644 index 0000000000..4af0046d5e --- /dev/null +++ b/scripts/docs/intermediate-files.md @@ -0,0 +1,478 @@ +# Intermediate File Generator + +## Overview + +The Intermediate File Generator (`generators/intermediate_files.py`) is a critical component in the ECS build pipeline. It transforms processed schemas into standardized intermediate representations that serve as the foundation for all downstream artifact generation. + +### Purpose + +This generator bridges the gap between schema processing and artifact generation by creating two normalized formats: + +1. **Flat Format** (`ecs_flat.yml`) - Single-level field dictionary +2. **Nested Format** (`ecs_nested.yml`) - Hierarchical fieldset organization + +These intermediate files provide: +- **Stable Interface**: Consistent data structure for all generators +- **Separation of Concerns**: Schema processing logic separate from artifact generation +- **Debugging Aid**: Human-readable checkpoints in the pipeline +- **Multiple Consumers**: CSV, Elasticsearch templates, Beats, documentation + +## Architecture + +### Pipeline Position + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Schema Processing │ +│ │ +│ 1. loader.py - Load YAML schemas from files │ +│ 2. cleaner.py - Normalize and validate │ +│ 3. finalizer.py - Apply transformations │ +│ 4. subset_filter.py - Optional filtering │ +│ 5. exclude_filter.py - Optional exclusions │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ intermediate_files.generate() │ +│ [THIS MODULE] │ +│ │ +│ Input: Dict[str, FieldEntry] (processed schemas) │ +│ │ +│ ┌───────────────────────┐ ┌───────────────────────┐ │ +│ │ generate_flat_fields()│ │generate_nested_fields()│ │ +│ │ │ │ │ │ +│ │ • Filter non-root │ │ • Keep all fieldsets │ │ +│ │ • Flatten hierarchy │ │ • Group by fieldset │ │ +│ │ • Index by flat_name │ │ • Preserve metadata │ │ +│ └───────────┬───────────┘ └──────────┬─────────────┘ │ +│ │ │ │ +│ ▼ ▼ │ +│ ecs_flat.yml (850 fields) ecs_nested.yml (45 fieldsets) │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Artifact Generators │ +│ │ +│ • CSV Generator - Uses ecs_flat.yml │ +│ • Elasticsearch - Uses ecs_nested.yml │ +│ • Beats Generator - Uses ecs_nested.yml │ +│ • Markdown Generator - Uses ecs_nested.yml │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Data Flow + +``` +Input: Processed Schemas + ↓ +{ + 'http': { + 'field_details': {...}, + 'schema_details': {...}, + 'fields': { + 'request': { + 'fields': { + 'method': { + 'field_details': { + 'flat_name': 'http.request.method', + 'type': 'keyword', + ... + } + } + } + } + } + } +} + ↓ + ├─── generate_flat_fields() ───→ Flat Format + │ { + │ 'http.request.method': { + │ 'name': 'method', + │ 'type': 'keyword', + │ ... + │ } + │ } + │ + └─── generate_nested_fields() ──→ Nested Format + { + 'http': { + 'name': 'http', + 'title': 'HTTP', + 'fields': { + 'http.request.method': {...} + } + } + } +``` + +## File Formats + +### Flat Format (ecs_flat.yml) + +**Purpose**: Quick lookup and iteration over all fields + +**Structure**: +```yaml +# Single-level dictionary, fields indexed by full dotted name +http.request.method: + name: method + flat_name: http.request.method + type: keyword + description: HTTP request method + example: GET + level: extended + normalize: + - array + otel: + - relation: match + stability: stable + +http.response.status_code: + name: status_code + flat_name: http.response.status_code + type: long + description: HTTP response status code + example: 404 + level: extended +``` + +**Characteristics**: +- **Keys**: Full dotted field names (e.g., `http.request.method`) +- **Values**: Complete field definitions +- **Excludes**: Non-root reusable fieldsets (top_level=false) +- **Excludes**: Intermediate structural fields +- **Count**: ~850 fields in standard ECS + +**Use Cases**: +- CSV generation (one row per field) +- Simple field lookups by name +- Validation scripts +- Field counting and statistics + +### Nested Format (ecs_nested.yml) + +**Purpose**: Preserve logical grouping and fieldset metadata + +**Structure**: +```yaml +# Top-level: fieldsets +http: + name: http + title: HTTP + group: 2 + description: Fields related to HTTP activity + type: group + reusable: + top_level: true + expected: + - client + - server + reused_here: + - full: http.request + short: request + schema_name: http.request + fields: + # Flat dictionary of all fields in this fieldset + http.request.method: + name: method + flat_name: http.request.method + type: keyword + description: HTTP request method + ... + http.response.status_code: + name: status_code + flat_name: http.response.status_code + type: long + ... + +user: + name: user + title: User + group: 2 + description: User fields + reusable: + top_level: true + expected: + - client + - destination + - server + - source + fields: + user.email: + name: email + ... +``` + +**Characteristics**: +- **Keys**: Fieldset names (e.g., `http`, `user`, `process`) +- **Values**: Fieldset metadata + fields dictionary +- **Includes**: All fieldsets (even top_level=false) +- **Fields**: Stored in nested `fields` dict (still flat, not hierarchical) +- **Count**: ~45 fieldsets in standard ECS + +**Use Cases**: +- Documentation generation (one page per fieldset) +- Elasticsearch templates (field grouping) +- Beats configuration +- Understanding field relationships + +## Key Concepts + +### Top-Level vs. Non-Root Reusable Fieldsets + +Some fieldsets are designed ONLY to be reused in specific locations: + +**Non-Root Reusable** (top_level=false): +```yaml +# geo fieldset - only valid under client.geo, source.geo, etc. +geo: + reusable: + top_level: false # Never appears as geo.* at root + expected: + - client.geo + - destination.geo + - source.geo +``` + +**Root Reusable** (top_level=true): +```yaml +# user fieldset - valid at root AND reused locations +user: + reusable: + top_level: true # Can appear as user.* at root + expected: + - client.user + - destination.user + - source.user +``` + +**Filtering Behavior**: +- **Flat format**: Excludes top_level=false fieldsets +- **Nested format**: Includes all fieldsets (consumers decide) + +### Intermediate Fields + +Some fields exist only for structural purposes: + +```yaml +# Intermediate field - creates hierarchy but isn't a real field +http.request: + intermediate: true # Not a field itself + fields: + method: {...} # Actual field: http.request.method + body: {...} # Actual field: http.request.body +``` + +These are excluded from intermediate files as they don't represent actual data. + +### Internal Attributes + +Attributes removed from final output: +- `node_name`: Internal tree traversal identifier +- `intermediate`: Flag for structural-only fields +- `dashed_name`: Alternative name format (not needed in output) + +## Usage Examples + +See [README.md](README.md) for generator invocation commands. + +### Programmatic Usage + +```python +from schema import loader, cleaner, finalizer +from generators.intermediate_files import generate + +# Process schemas +fields = loader.load_schemas() +cleaner.clean(fields) +finalizer.finalize(fields) + +# Generate intermediate files +nested, flat = generate( + fields=fields, + out_dir='generated/ecs', + default_dirs=True # Also save raw ecs.yml +) + +# Use the returned structures +print(f"Total fields: {len(flat)}") +print(f"Total fieldsets: {len(nested)}") + +# Access specific field +method_field = flat['http.request.method'] +print(f"Type: {method_field['type']}") + +# Access fieldset +http_fieldset = nested['http'] +print(f"Title: {http_fieldset['title']}") +print(f"Fields in HTTP: {len(http_fieldset['fields'])}") +``` + +### Reading Generated Files + +```python +import yaml + +# Load flat format +with open('generated/ecs/ecs_flat.yml') as f: + flat = yaml.safe_load(f) + +# Iterate all fields +for field_name, field_def in flat.items(): + print(f"{field_name}: {field_def['type']}") + +# Load nested format +with open('generated/ecs/ecs_nested.yml') as f: + nested = yaml.safe_load(f) + +# Process by fieldset +for fieldset_name, fieldset in nested.items(): + print(f"\n{fieldset['title']} ({fieldset_name})") + for field_name in fieldset['fields']: + print(f" - {field_name}") +``` + +## Making Changes + +### Adding New Field Attributes + +If you add a new attribute to field definitions: + +1. **Update schema files** (in `schemas/*.yml`) +2. **Update type definitions** (in `ecs_types/schema_fields.py`) +3. **No changes needed here** - attributes pass through automatically +4. **Update downstream consumers** if they need to use the new attribute + +Example: Adding a `sensitivity` attribute +```yaml +# In schema +- name: password + type: keyword + sensitivity: high # NEW attribute + +# Automatically appears in both formats: +# ecs_flat.yml +user.password: + name: password + type: keyword + sensitivity: high # Passed through + +# ecs_nested.yml +user: + fields: + user.password: + sensitivity: high # Passed through +``` + +### Filtering Additional Attributes + +To remove an attribute from intermediate files: + +```python +def remove_internal_attributes(field_details: Field) -> None: + """Remove internal-only attributes.""" + field_details.pop('node_name', None) + field_details.pop('intermediate', None) + field_details.pop('new_internal_attr', None) # Add this +``` + +### Changing Flat Format Filtering + +To change which fields appear in the flat format: + +```python +def generate_flat_fields(fields: Dict[str, FieldEntry]) -> Dict[str, Field]: + """Generate flat field representation.""" + filtered: Dict[str, FieldEntry] = remove_non_root_reusables(fields) + + # Add additional filtering + filtered = remove_deprecated_fields(filtered) # NEW + + flattened: Dict[str, Field] = {} + visitor.visit_fields_with_memo(filtered, accumulate_field, flattened) + return flattened +``` + +### Modifying Nested Format Structure + +To change fieldset-level attributes: + +```python +def generate_nested_fields(fields: Dict[str, FieldEntry]) -> Dict[str, FieldNestedEntry]: + """Generate nested fieldset representation.""" + nested: Dict[str, FieldNestedEntry] = {} + + for (name, details) in fields.items(): + fieldset_details = { + **copy.deepcopy(details['field_details']), + **copy.deepcopy(details['schema_details']) + } + + # Add custom processing + if 'beta' in fieldset_details: + fieldset_details['stability'] = 'beta' # NEW + + # ... rest of processing ... +``` + +## Troubleshooting + +### Common Issues + +#### Missing fields in flat format + +**Symptom**: Field appears in schema but not in ecs_flat.yml + +**Possible causes**: +1. Field is in a fieldset with `top_level: false` + - **Check**: Look at fieldset's `reusable.top_level` setting + - **Solution**: If field should be at root, set `top_level: true` + +2. Field is marked as `intermediate: true` + - **Check**: Look for `intermediate` attribute in schema + - **Solution**: Remove if field should be included + +3. Field is in a filtered subset + - **Check**: Are you using `--subset` or `--exclude` flags? + - **Solution**: Adjust filtering or run without filters + +#### Fieldset missing from nested format + +**Symptom**: Fieldset defined in schema but not in ecs_nested.yml + +**Unlikely**: The nested format includes all fieldsets by design. + +**Check**: +- Verify fieldset is properly defined in schema +- Check for schema validation errors earlier in pipeline +- Ensure schema file is in the loaded directory + +#### Unexpected attributes in output + +**Symptom**: Internal attributes appearing in intermediate files + +**Solution**: Add to `remove_internal_attributes()`: +```python +def remove_internal_attributes(field_details: Field) -> None: + field_details.pop('node_name', None) + field_details.pop('intermediate', None) + field_details.pop('unwanted_attr', None) # Add this +``` + +### Debugging Tips + +- Use `default_dirs=True` to generate `ecs.yml` with raw processed schemas +- Compare outputs: `diff ecs_flat_old.yml ecs_flat_new.yml` +- Count fields with `len(yaml.safe_load(open('ecs_flat.yml')))` + +## References + +- [ECS Schema Structure](../../USAGE.md) +- [Visitor Pattern Documentation](../schema/visitor.py) +- [ECS Type Definitions](../ecs_types/schema_fields.py) +- [CSV Generator](csv-generator.md) +- [Elasticsearch Template Generator](es-template.md) + diff --git a/scripts/docs/markdown-generator.md b/scripts/docs/markdown-generator.md new file mode 100644 index 0000000000..8052370447 --- /dev/null +++ b/scripts/docs/markdown-generator.md @@ -0,0 +1,447 @@ +# Markdown Documentation Generator + +## Overview + +The Markdown Generator (`generators/markdown_fields.py`) transforms ECS field schemas into human-readable documentation published on the Elastic documentation site. It's the final step in the documentation pipeline, converting structured YAML field definitions into comprehensive markdown pages. + +### Purpose + +This generator creates the official ECS reference documentation, including: + +1. **Field Reference Pages** - Complete catalog of all fields +2. **Fieldset Pages** - Detailed documentation for each fieldset (e.g., HTTP, User, Process) +3. **OTel Alignment Documentation** - Showing convergence with OpenTelemetry +4. **Index and Navigation** - Entry points and cross-references + +The output is human-friendly markdown that integrates with Elastic's documentation infrastructure. + +## Architecture + +### High-Level Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ generator.py (main) │ +│ │ +│ 1. Load schemas │ +│ 2. Clean and finalize │ +│ 3. Generate intermediate files │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ markdown_fields.generate() - Entry Point │ +│ │ +│ Input: Nested fieldsets + OTel generator + version info │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Page Generation Functions │ +│ │ +│ ├─ page_index() → index.md │ +│ ├─ page_field_reference() → ecs-field-reference.md │ +│ ├─ page_otel_alignment_overview() → ecs-otel-alignment-*.md │ +│ ├─ page_otel_alignment_details() → ecs-otel-alignment-*.md │ +│ └─ page_fieldset() [for each] → ecs-{name}.md │ +│ │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Jinja2 Template Rendering │ +│ │ +│ Templates (scripts/templates/): │ +│ - index.j2 │ +│ - fieldset.j2 │ +│ - ecs_field_reference.j2 │ +│ - otel_alignment_overview.j2 │ +│ - otel_alignment_details.j2 │ +│ - field_values.j2 │ +│ - macros.j2 (shared template macros) │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Markdown Files Output │ +│ │ +│ Written to: docs/reference/ │ +│ - index.md │ +│ - ecs-field-reference.md │ +│ - ecs-otel-alignment-overview.md │ +│ - ecs-otel-alignment-details.md │ +│ - ecs-http.md, ecs-user.md, ecs-process.md, ... │ +│ (one per fieldset) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. Generate Function + +**Entry Point**: `generate(nested, docs_only_nested, ecs_version, semconv_version, otel_generator, out_dir)` + +Orchestrates the entire markdown generation process: +- Creates output directory +- Generates each page type +- Saves rendered markdown to files + +**Called by**: `generator.py` main script after all schema processing is complete + +#### 2. Helper Functions + +These prepare data for template consumption: + +| Function | Purpose | +|----------|---------| +| `render_fieldset_reuse_text()` | Extract expected nesting locations | +| `render_nestings_reuse_section()` | Build reuse section data | +| `extract_allowed_values_key_names()` | Get allowed value names | +| `sort_fields()` | Sort and enrich field lists | +| `check_for_usage_doc()` | Check for usage doc existence | + +#### 3. Page Generation Functions + +Each decorated with `@templated()` for automatic rendering: + +| Function | Template | Output File | Purpose | +|----------|----------|-------------|---------| +| `page_index()` | index.j2 | index.md | Main landing page | +| `page_field_reference()` | ecs_field_reference.j2 | ecs-field-reference.md | All fields catalog | +| `page_fieldset()` | fieldset.j2 | ecs-{name}.md | Individual fieldset docs | +| `page_otel_alignment_overview()` | otel_alignment_overview.j2 | ecs-otel-alignment-overview.md | Alignment statistics | +| `page_otel_alignment_details()` | otel_alignment_details.j2 | ecs-otel-alignment-details.md | Field mappings | +| `page_field_values()` | field_values.j2 | (not saved directly) | Event categorization fields | + +#### 4. Template System + +**Framework**: Jinja2 + +**Configuration**: +```python +template_env = jinja2.Environment( + loader=FileSystemLoader('scripts/templates/'), + keep_trailing_newline=True, # Preserve trailing newlines + trim_blocks=True, # Remove first newline after block + lstrip_blocks=False # Don't strip leading whitespace +) +``` + +**Template Location**: `scripts/templates/` + +**Shared Macros**: `macros.j2` contains reusable template components + +## Template Development + +### Adding a New Page Type + +To add a new documentation page: + +1. **Create the template** in `scripts/templates/`: + ```jinja2 + {# my_new_page.j2 #} + # {{ title }} + + Version: {{ version }} + + {% for item in items %} + ## {{ item.name }} + {{ item.description }} + {% endfor %} + ``` + +2. **Create page function** in `markdown_fields.py`: + ```python + @templated('my_new_page.j2') + def page_my_new_page(items, version): + """Generate my new documentation page. + + Args: + items: List of items to document + version: Version string + + Returns: + Rendered markdown content + """ + return dict( + title="My New Page", + items=items, + version=version + ) + ``` + +3. **Call in generate()** function: + ```python + def generate(nested, docs_only_nested, ecs_version, semconv_version, otel_generator, out_dir): + # ... existing code ... + + save_markdown( + path.join(out_dir, 'my-new-page.md'), + page_my_new_page(some_items, ecs_version) + ) + ``` + +### Template Best Practices + +1. **Use macros for repeated patterns**: + ```jinja2 + {# In macros.j2 #} + {% macro field_row(field) -%} + | {{ field.name }} | {{ field.type }} | {{ field.description }} | + {%- endmacro %} + + {# In your template #} + {% from 'macros.j2' import field_row %} + {% for field in fields %} + {{ field_row(field) }} + {% endfor %} + ``` + +2. **Handle missing data gracefully**: + ```jinja2 + {% if field.example %} + Example: `{{ field.example }}` + {% endif %} + ``` + +3. **Keep formatting consistent**: + - Use consistent heading levels + - Follow markdown best practices + - Include blank lines between sections + +4. **Comment complex logic**: + ```jinja2 + {# Sort fields by type, then name #} + {% for field in fields|sort(attribute='type,name') %} + ... + {% endfor %} + ``` + +## Data Structures + +### Nested Fieldsets Structure + +```python +{ + 'http': { + 'name': 'http', + 'title': 'HTTP', + 'group': 2, + 'description': 'HTTP request and response fields', + 'fields': { + 'http.request.method': { + 'name': 'method', + 'flat_name': 'http.request.method', + 'type': 'keyword', + 'description': 'HTTP request method', + 'example': 'GET', + 'level': 'extended', + 'otel': [{'relation': 'match', 'stability': 'stable'}], + 'allowed_values': [...] # Optional + }, + # ... more fields ... + }, + 'reusable': { # If fieldset is reusable + 'expected': [ + {'full': 'client.http', 'short': 'client.http'}, + {'full': 'server.http', 'short': 'server.http'} + ] + }, + 'reused_here': [ # Fieldsets nested here + { + 'full': 'client.geo', + 'schema_name': 'geo', + 'short': 'geo', + 'beta': '', + 'normalize': [] + } + ] + }, + # ... more fieldsets ... +} +``` + +### OTel Mapping Summary Structure + +```python +{ + 'namespace': 'http', + 'title': 'HTTP', + 'nr_all_ecs_fields': 25, + 'nr_plain_ecs_fields': 20, + 'nr_otel_fields': 18, + 'nr_matching_fields': 10, + 'nr_equivalent_fields': 5, + 'nr_related_fields': 3, + 'nr_conflicting_fields': 1, + 'nr_metric_fields': 0, + 'nr_otlp_fields': 0, + 'nr_not_applicable_fields': 1 +} +``` + +## Usage Examples + +See [README.md](README.md) for generator invocation commands. + +### Programmatic Usage + +```python +from generators import markdown_fields +from generators.otel import OTelGenerator + +# Prepare data +nested = {...} # From intermediate_files.generate() +docs_only = {...} +otel_gen = OTelGenerator('v1.24.0') + +# Generate all markdown docs +markdown_fields.generate( + nested=nested, + docs_only_nested=docs_only, + ecs_generated_version='8.11.0', + semconv_version='v1.24.0', + otel_generator=otel_gen, + out_dir='docs/reference' +) +``` + +### Testing Template Changes + +To test template modifications without full regeneration: + +```python +from generators.markdown_fields import render_template + +# Test a template with sample data +context = { + 'fieldset': {'name': 'http', 'title': 'HTTP'}, + 'sorted_fields': [...] +} + +output = render_template('fieldset.j2', **context) +print(output) +``` + +## Making Changes + +### Modifying Existing Pages + +To change an existing page's content: + +1. **Locate the template**: Find the `.j2` file in `scripts/templates/` +2. **Edit the template**: Modify Jinja2 markup +3. **Update page function** (if needed): Adjust context data in `markdown_fields.py` +4. **Test**: Regenerate documentation and review output +5. **Validate**: Check markdown renders correctly + +Example - Adding a field to fieldset pages: + +```python +# In markdown_fields.py +@templated('fieldset.j2') +def page_fieldset(fieldset, nested, ecs_generated_version): + # ... existing code ... + return dict( + fieldset=fieldset, + sorted_fields=sorted_fields, + # Add new data + field_count=len(sorted_fields), # NEW + # ... rest of context ... + ) +``` + +```jinja2 +{# In fieldset.j2 #} +# {{ fieldset.title }} + +This fieldset contains {{ field_count }} fields. {# NEW #} + +{# ... rest of template ... #} +``` + +### Changing Field Display Order + +To modify how fields are sorted: + +```python +def sort_fields(fieldset): + """Sort fields by custom criteria.""" + fields_list = list(fieldset['fields'].values()) + for field in fields_list: + field['allowed_value_names'] = extract_allowed_values_key_names(field) + + # Change sorting key + return sorted(fields_list, key=lambda f: (f.get('level'), f['name'])) + # Now sorts by level first, then name +``` + +### Adding Conditional Sections + +To show content only for certain fieldsets: + +```jinja2 +{% if fieldset.name == 'event' %} +## Special Event Categorization + +The event fieldset includes special categorization fields... +{% endif %} +``` + +## Troubleshooting + +### Common Issues + +#### "Template not found: xyz.j2" + +**Cause**: Template file doesn't exist or path is wrong + +**Solution**: +- Verify template exists in `scripts/templates/` +- Check template name spelling +- Ensure `TEMPLATE_DIR` path is correct + +#### Markdown not rendering correctly + +**Cause**: Jinja2 whitespace control or markdown syntax issues + +**Solutions**: +- Check for extra/missing blank lines +- Use `{%-` and `-%}` for whitespace control +- Validate markdown with a linter +- Review `trim_blocks` and `lstrip_blocks` settings + +Example whitespace issue: +```jinja2 +{# BAD - Creates unwanted blank lines #} +{% for field in fields %} +{{ field.name }} +{% endfor %} + +{# GOOD - Cleaner output #} +{% for field in fields -%} +{{ field.name }} +{% endfor %} +``` + +#### Context variable not available in template + +**Cause**: Variable not passed in context dictionary + +**Solution**: Update the page function's return dict: +```python +@templated('my_template.j2') +def page_something(...): + return dict( + existing_var=value, + new_var=new_value # Add missing variable + ) +``` + +## References + +- [Jinja2 Documentation](https://jinja.palletsprojects.com/) +- [Markdown Guide](https://www.markdownguide.org/) +- [ECS Documentation](https://www.elastic.co/guide/en/ecs/current/index.html) +- [Elastic Doc Build Tools](https://github.com/elastic/docs) + diff --git a/scripts/docs/otel-integration.md b/scripts/docs/otel-integration.md new file mode 100644 index 0000000000..0e699b2375 --- /dev/null +++ b/scripts/docs/otel-integration.md @@ -0,0 +1,364 @@ +# OpenTelemetry Semantic Conventions Integration + +## Overview + +The OTel integration module (`generators/otel.py`) manages the alignment between Elastic Common Schema (ECS) and OpenTelemetry Semantic Conventions. This is a critical component supporting the ECS donation to OpenTelemetry initiative. + +### Purpose + +As ECS and OTel Semantic Conventions converge into a single standard, this module: + +1. **Validates** that ECS field mappings reference valid OTel attributes and metrics +2. **Enriches** ECS definitions with OTel stability information +3. **Generates** alignment summaries for documentation +4. **Detects** potential unmapped fields that exist in both standards + +## Architecture + +### High-Level Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ generator.py │ +│ (Main Entry Point) │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ OTelGenerator.__init__() │ +│ │ +│ 1. Clone/load OTel semconv repo from GitHub │ +│ 2. Parse all YAML model files │ +│ 3. Extract attributes and metrics │ +│ 4. Build lookup indexes │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ OTelGenerator.validate_otel_mapping() │ +│ │ +│ Pass 1: Validate mapping structure │ +│ - Check relation types are valid │ +│ - Verify required/forbidden properties │ +│ - Confirm referenced attributes/metrics exist │ +│ │ +│ Pass 2: Enrich with stability information │ +│ - Add stability levels from OTel definitions │ +└────────────────────────────┬────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ OTelGenerator.get_mapping_summaries() │ +│ │ +│ Generate statistics for each namespace: │ +│ - Count fields by relation type │ +│ - Calculate coverage percentages │ +│ - Used for documentation generation │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. Model Loading (`get_model_files`, `get_tree_by_url`) + +**Purpose**: Load OTel semantic conventions from GitHub + +- Clones the semantic-conventions repository (or uses cached version) +- Checks out a specific version tag (e.g., `v1.24.0`) +- Recursively collects all YAML model files +- Caches the repository in `./build/otel-semconv/` for performance + +**Key Files**: All `.yml`/`.yaml` files in the `model/` directory of the OTel semconv repo + +#### 2. Attribute/Metric Extraction (`get_attributes`, `get_metrics`) + +**Purpose**: Parse model files and build lookup indexes + +- Extracts non-deprecated attributes from `attribute_group` entries +- Extracts non-deprecated metrics from `metric` entries +- Applies prefixes to attribute IDs (e.g., `http.` prefix) +- Preserves display names for documentation + +**Output**: Dictionaries keyed by attribute ID / metric name + +#### 3. Validation (`OTelGenerator.validate_otel_mapping`) + +**Purpose**: Ensure mapping integrity + +- Uses visitor pattern to traverse all ECS fields +- Validates each OTel mapping configuration +- Checks existence of referenced attributes/metrics +- Enriches mappings with stability levels +- Prints warnings for potential unmapped fields + +#### 4. Summary Generation (`OTelGenerator.get_mapping_summaries`) + +**Purpose**: Generate documentation statistics + +- Counts fields by relation type for each namespace +- Identifies OTel-only namespaces (not yet in ECS) +- Produces data structure consumed by markdown generators +- Sorted alphabetically for consistent output + +## OTel Mapping Configuration + +### Relation Types + +ECS fields can have one or more OTel mappings, each with a `relation` type: + +#### `match` +Names and semantics are identical. + +```yaml +- name: method + flat_name: http.request.method + otel: + - relation: match +``` + +**Requirements**: No additional properties +**Generated stability**: From OTel attribute definition + +#### `equivalent` +Semantically equivalent but different names. + +```yaml +- name: status_code + flat_name: http.response.status_code + otel: + - relation: equivalent + attribute: http.response.status_code +``` + +**Requirements**: Must specify `attribute` +**Generated stability**: From OTel attribute definition + +#### `related` +Related concepts but not semantically identical. + +```yaml +- name: original + flat_name: url.original + otel: + - relation: related + attribute: url.full + note: Similar but may have different encoding +``` + +**Requirements**: Must specify `attribute` +**Optional**: `note` explaining the relationship + +#### `conflict` +Conflicting definitions that need resolution. + +```yaml +- name: bytes + flat_name: http.request.body.bytes + otel: + - relation: conflict + attribute: http.request.body.size + note: ECS uses bytes, OTel uses size +``` + +**Requirements**: Must specify `attribute` +**Optional**: `note` explaining the conflict + +#### `metric` +Maps to an OTel metric rather than an attribute. + +```yaml +- name: duration + flat_name: http.client.request.duration + otel: + - relation: metric + metric: http.client.request.duration +``` + +**Requirements**: Must specify `metric` +**Forbidden**: `attribute`, `otlp_field` + +#### `otlp` +Maps to an OTLP protocol-specific field. + +```yaml +- name: trace_id + flat_name: trace.id + otel: + - relation: otlp + otlp_field: trace_id + stability: stable +``` + +**Requirements**: Must specify `otlp_field` and `stability` +**Forbidden**: `attribute`, `metric` + +#### `na` +Not applicable for OTel mapping. + +```yaml +- name: ecs_version + flat_name: ecs.version + otel: + - relation: na + note: ECS-specific field +``` + +**Requirements**: None +**Forbidden**: `attribute`, `metric`, `otlp_field`, `stability` + +### Validation Rules + +The validator enforces strict rules for each relation type: + +| Relation | Required Properties | Forbidden Properties | Validates Existence | +|----------|---------------------|---------------------|---------------------| +| `match` | - | attribute, metric, otlp_field, stability | Yes (attribute) | +| `equivalent` | attribute | metric, otlp_field, stability | Yes (attribute) | +| `related` | attribute | metric, otlp_field, stability | Yes (attribute) | +| `conflict` | attribute | metric, otlp_field, stability | Yes (attribute) | +| `metric` | metric | attribute, otlp_field, stability | Yes (metric) | +| `otlp` | otlp_field, stability | attribute, metric | No | +| `na` | - | attribute, metric, otlp_field, stability | No | + +## Usage Examples + +See [README.md](README.md) for generator invocation commands. + +### Programmatic Usage + +```python +from generators.otel import OTelGenerator +from schema import loader + +# Initialize generator with specific OTel version +generator = OTelGenerator('v1.24.0') + +# Load ECS schemas +fields = loader.load_schemas() + +# Validate all OTel mappings +generator.validate_otel_mapping(fields) + +# Generate summaries for documentation +from generators.intermediate_files import generate_nested_fields +nested = generate_nested_fields(fields) +summaries = generator.get_mapping_summaries(nested) + +# Use summaries in documentation +for summary in summaries: + print(f"{summary['namespace']}: {summary['nr_matching_fields']} matches") +``` + +## Making Changes + +### Adding New Relation Types + +If a new relation type is needed: + +1. **Update validation** in `OTelGenerator.__check_mapping()`: + ```python + elif otel['relation'] == 'new_type': + must_have(ecs_field_name, otel, otel['relation'], 'required_property') + must_not_have(ecs_field_name, otel, otel['relation'], 'forbidden_property') + # Add validation logic + ``` + +2. **Update summary counting** in `OTelGenerator.get_mapping_summaries()`: + ```python + elif otel['relation'] == "new_type": + summary['nr_new_type_fields'] += 1 + ``` + +3. **Update type definition** in `ecs_types/otel_types.py`: + ```python + class OTelMappingSummary(TypedDict, total=False): + # ... existing fields ... + nr_new_type_fields: int + ``` + +4. **Update documentation templates** in `templates/otel_alignment_*.j2` + +5. **Update this documentation** with the new relation type + +### Updating OTel Semconv Version + +To use a newer version of OTel semantic conventions: + +1. **Check available versions**: + Visit https://github.com/open-telemetry/semantic-conventions/tags + +2. **Update version file**: + ```bash + echo "v1.25.0" > otel-semconv-version + ``` + +3. **Regenerate**: + ```bash + make clean + make SEMCONV_VERSION=v1.25.0 + ``` + +4. **Handle validation errors**: + - If attributes were renamed: Update ECS mappings in `schemas/*.yml` + - If attributes were deprecated: Update or remove mappings + - If validation rules changed: Update `otel.py` validator + +### Testing Changes + +After modifying the OTel generator: + +1. **Run validation**: + ```bash + python scripts/generator.py --semconv-version v1.24.0 + ``` + +2. **Check generated files**: + - `docs/reference/otel-*.md` - Alignment documentation + - Verify summary statistics are correct + +3. **Run tests** (if applicable): + ```bash + python -m pytest scripts/tests/ + ``` + +## Troubleshooting + +### Common Issues + +#### "Attribute 'X' does not exist in Semantic Conventions version Y" + +**Cause**: ECS field references an OTel attribute that doesn't exist in the specified version + +**Solutions**: +- Check if attribute was renamed in OTel +- Update the `attribute` value in the ECS schema +- Verify the semconv version is correct +- Check if attribute was deprecated/removed + +#### "OTel mapping must specify the property 'attribute'" + +**Cause**: Mapping has relation type requiring the `attribute` property, but it's missing + +**Solution**: Add the required property to the mapping: +```yaml +otel: + - relation: equivalent + attribute: otel.attribute.name # Add this +``` + + +#### "WARNING: Field 'X' exists in OTel but is not mapped" + +**Cause**: Field name matches OTel attribute but has no mapping defined + +**Action**: Consider if this should be mapped: +- If yes: Add appropriate OTel mapping to schema +- If no: Add `otel: [{relation: na}]` to suppress warning + +## References + +- [ECS Documentation](https://www.elastic.co/guide/en/ecs/current/index.html) +- [OTel Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) +- [ECS-OTel Convergence Announcement](https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/) +- [Semantic Conventions Repository](https://github.com/open-telemetry/semantic-conventions) + diff --git a/scripts/docs/schema-pipeline.md b/scripts/docs/schema-pipeline.md new file mode 100644 index 0000000000..8e54b3e6fe --- /dev/null +++ b/scripts/docs/schema-pipeline.md @@ -0,0 +1,1368 @@ +# ECS Schema Processing Pipeline + +## Overview + +The ECS schema processing pipeline transforms YAML schema definitions into various output formats (Elasticsearch templates, Beats configs, markdown docs, etc.). It's a multi-stage pipeline where each stage has a specific responsibility. + +**Pipeline Stages:** +``` +┌─────────────┐ +│ YAML Schema │ Raw schema files in schemas/*.yml +│ Files │ +└──────┬──────┘ + │ + v +┌─────────────┐ +│ loader.py │ Load & nest: YAML → deeply nested dict +└──────┬──────┘ + │ + v +┌─────────────┐ +│ cleaner.py │ Validate, normalize, apply defaults +└──────┬──────┘ + │ + v +┌─────────────┐ +│finalizer.py │ Perform field reuse, calculate names +└──────┬──────┘ + │ + v (Optional filters) +┌─────────────┐ ┌────────────────┐ +│subset_filter│─>│exclude_filter │ +│ .py │ │ .py │ +└──────┬──────┘ └────────┬───────┘ + │ │ + v v +┌─────────────────────────────┐ +│ intermediate_files.py │ Generate flat & nested YAML +└──────────────┬──────────────┘ + │ + v + ┌────────────────────┐ + │ Generators │ + ├────────────────────┤ + │ • es_template.py │ Elasticsearch templates + │ • beats.py │ Beats field definitions + │ • csv_generator.py │ CSV field export + │ • markdown_fields │ Markdown documentation + └────────────────────┘ +``` + +## Quick Reference + +### Field Reuse Cheat Sheet + +| Concept | What | When to Use | Example | +|---------|------|-------------|---------| +| **Foreign Reuse** | Copy fieldset to different location | Same fields needed elsewhere | `user` → `destination.user` | +| **Transitive** | Reuse carries nested reuses | Automatic composition | If `group` in `user`, `destination.user` gets `group` too | +| **Self-Nesting** | Copy fieldset into itself | Parent/child relationships | `process` → `process.parent` | +| **Non-Transitive** | Self-nesting stays local | Avoid unwanted propagation | `process.parent` NOT at `source.process.parent` | +| **order: 1** | High priority reuse | Has dependencies | `group` reused before `user` | +| **order: 2** | Default priority | Most fieldsets | Standard reuse timing | + +**Quick Syntax:** +```yaml +# Foreign reuse (goes to other fieldsets) +fieldset: + reusable: + expected: + - destination # Simple: reuse as same name + - at: process # Complex: reuse with different name + as: parent + +# Self-nesting (stays in same fieldset) +process: + reusable: + expected: + - at: process # ← Same name as fieldset = self-nesting + as: parent +``` + +### Subset Definition Cheat Sheet + +| Syntax | Meaning | Result | +|--------|---------|--------| +| `fields: '*'` | Include all fields | Every field in fieldset | +| `fields: { field: {} }` | Include specific field | Just that one field | +| `fields: { parent: { fields: '*' }}` | Include all nested | All fields under parent | +| `index: false` | Don't index field | Field exists but not searchable | +| `docs_only: true` | Documentation only | In docs, not in artifacts | + +**Quick Syntax:** +```yaml +name: my_subset +fields: + base: + fields: '*' # All base fields + + http: + fields: + request: + fields: + method: {} # Just this field + response: + fields: '*' # All response fields + + destination: + fields: + user: # Reused fieldset + fields: + name: {} # Specific user fields +``` + +### Common Patterns + +#### Pattern 1: Network Endpoint Fields (Foreign Reuse) + +**Problem:** Need same fields for source, destination, client, server + +**Solution:** Create reusable fieldset, reuse at all locations +```yaml +# In geo schema +geo: + reusable: + top_level: false # Only via reuse + expected: + - client + - destination + - host + - observer + - server + - source + fields: + - name: city_name + - name: country_name + - name: location # latitude/longitude +``` + +**Result:** `source.geo.city_name`, `destination.geo.city_name`, etc. + +#### Pattern 2: Parent-Child Hierarchy (Self-Nesting) + +**Problem:** Need to represent parent process, effective user, session leader + +**Solution:** Self-nesting +```yaml +process: + reusable: + expected: + - at: process + as: parent + - at: process + as: session_leader + fields: + - name: pid + - name: name +``` + +**Result:** `process.pid`, `process.parent.pid`, `process.session_leader.pid` + +#### Pattern 3: Minimal Web Subset + +**Problem:** Only need basic HTTP fields for web logs + +**Solution:** +```yaml +name: web_minimal +fields: + base: { fields: '*' } + http: + fields: + request: { fields: { method: {}, bytes: {} }} + response: { fields: { status_code: {}, bytes: {} }} + url: { fields: { domain: {}, path: {} }} +``` + +**Result:** ~10-15 fields instead of 850 + +#### Pattern 4: Security Monitoring Subset + +**Problem:** Need security-relevant fields only + +**Solution:** +```yaml +name: security +fields: + base: { fields: '*' } + event: { fields: { action: {}, category: {}, type: {}, outcome: {} }} + source: { fields: { ip: {}, port: {}, user: { fields: { name: {} }}}} + destination: { fields: { ip: {}, port: {} }} + process: + fields: + name: {} + pid: {} + parent: { fields: { name: {}, pid: {} }} + file: + fields: + path: {} + hash: { fields: { sha256: {} }} +``` + +**Result:** Security-focused field set + +--- + +## Core Concepts + +### Deeply Nested Structure + +All pipeline stages work with a deeply nested dictionary structure: + +```python +{ + 'fieldset_name': { + 'schema_details': { # Fieldset-level metadata + 'root': bool, + 'group': int, + 'reusable': {...}, + 'title': str + }, + 'field_details': { # Properties of the fieldset itself + 'name': str, + 'description': str, + 'type': 'group' + }, + 'fields': { # Nested fields + 'field_name': { + 'field_details': {...}, + 'fields': {...} # Recursive + } + } + } +} +``` + +### Intermediate Fields + +Auto-created parent fields for nesting structure: +- Created automatically for dotted names: `request.method` → creates `request` intermediate +- Marked with `intermediate: true` +- Type: `object` +- Skipped by some validation/processing steps + +### Field Reuse + +**Why Field Reuse Exists:** + +Without reuse, we'd need to duplicate the same fields everywhere: +```yaml +# Without reuse - lots of duplication! ❌ +source: + - name: ip + - name: port + - name: address +destination: + - name: ip # Duplicated! + - name: port # Duplicated! + - name: address # Duplicated! +client: + - name: ip # Duplicated again! + - name: port # Duplicated again! + # ... and so on +``` + +With reuse, we define fields once and reuse them: +```yaml +# With reuse - define once, reuse everywhere! ✅ +user: + reusable: + top_level: false # Not at root + expected: + - destination # Reuse at destination.user + - source # Reuse at source.user + - client # Reuse at client.user + fields: + - name: name + - name: email + - name: id +``` + +**Two Types of Reuse:** + +#### 1. Foreign Reuse (Transitive) - Copy Across Fieldsets + +**What it does:** Copies a fieldset into a completely different fieldset + +**Example:** `user` fields appear at `destination.user.*`, `source.user.*` + +**Why "transitive":** If A is reused in B, and B is reused in C, then C automatically gets A too. + +**Visual Example:** +``` +Before Reuse: +┌──────────┐ ┌─────────────┐ +│ user │ │ destination │ +├──────────┤ ├─────────────┤ +│ • name │ │ • ip │ +│ • email │ │ • port │ +│ • id │ └─────────────┘ +└──────────┘ + +After Reuse (user → destination.user): +┌─────────────────────────────┐ +│ destination │ +├─────────────────────────────┤ +│ • ip │ +│ • port │ +│ • user ← (reused!) │ +│ ├─ name │ +│ ├─ email │ +│ └─ id │ +└─────────────────────────────┘ + +Result: destination.user.name, destination.user.email, destination.user.id +``` + +**Transitivity in Action:** +``` +Step 1: group → user.group +┌──────────┐ ┌──────────────────┐ +│ group │ ───> │ user │ +│ • id │ │ • name │ +│ • name │ │ • email │ +└──────────┘ │ • group (copied) │ + │ ├─ id │ + │ └─ name │ + └──────────────────┘ + +Step 2: user (with group!) → destination.user +┌──────────────────┐ ┌────────────────────────────────┐ +│ user │ ───> │ destination │ +│ • name │ │ • ip │ +│ • email │ │ • port │ +│ • group │ │ • user (copied with group!) │ +│ ├─ id │ │ ├─ name │ +│ └─ name │ │ ├─ email │ +└──────────────────┘ │ └─ group ← (transitive!) │ + │ ├─ id │ + │ └─ name │ + └────────────────────────────────┘ + +Result: destination.user.group.id exists because transitivity! +``` + +#### 2. Self-Nesting (Non-Transitive) - Copy Within Same Fieldset + +**What it does:** Copies a fieldset into itself with a different name + +**Example:** `process` fields appear at `process.parent.*` + +**Why "non-transitive":** This nesting is local only. When the fieldset is reused elsewhere, the self-nesting doesn't come along. + +**Visual Example:** +``` +Before Self-Nesting: +┌──────────┐ +│ process │ +├──────────┤ +│ • pid │ +│ • name │ +│ • args │ +└──────────┘ + +After Self-Nesting (process → process.parent): +┌───────────────────────────┐ +│ process │ +├───────────────────────────┤ +│ • pid │ +│ • name │ +│ • args │ +│ • parent ← (self-nested!) │ +│ ├─ pid │ +│ ├─ name │ +│ └─ args │ +└───────────────────────────┘ + +Result: process.pid, process.name, process.parent.pid, process.parent.name +``` + +**Non-Transitivity in Action:** +``` +Scenario: process has self-nesting, then process is reused at source + +Step 1: process → process.parent (self-nesting) +┌───────────────────────┐ +│ process │ +│ • pid │ +│ • name │ +│ • parent (self-nest) │ +│ ├─ pid │ +│ └─ name │ +└───────────────────────┘ + +Step 2: process → source.process (foreign reuse) +┌─────────────────────────┐ +│ source │ +│ • ip │ +│ • port │ +│ • process │ +│ ├─ pid │ +│ └─ name │ +│ └─ parent? ← NO! ❌ │ +└─────────────────────────┘ + +Result: source.process.parent does NOT exist! +Why? Self-nesting is NOT transitive - it stays local to original fieldset. +``` + +**When to Use Each Type:** + +| Use Case | Type | Example | +|----------|------|---------| +| Same fields needed in multiple places | Foreign Reuse | user at destination, source, client | +| Capture hierarchical relationship | Self-Nesting | process.parent, process.session_leader | +| Build complex nested structures | Foreign Reuse | geo at client.geo, server.geo | +| Represent parent/child relationships | Self-Nesting | user.target, user.effective | + +**Reuse Order:** + +Some fieldsets depend on others being reused first: +```yaml +group: + reusable: + order: 1 # ← Reused FIRST (high priority) + expected: + - user # group goes into user + +user: + reusable: + order: 2 # ← Reused SECOND (default priority) + expected: + - destination # user (now with group) goes into destination +``` + +**Processing Order:** +1. Order 1 fieldsets → Foreign reuse → Self-nesting +2. Order 2 fieldsets → Foreign reuse → Self-nesting + +**Result:** `destination.user.group.*` exists because group was reused into user before user was reused into destination. + +## Pipeline Stages + +### 1. loader.py - Schema Loading + +**Purpose:** Load YAML schema files and create initial nested structure + +**Input:** +- YAML schema files (`schemas/*.yml`) +- Optional: git ref for specific version +- Optional: custom/experimental schemas + +**Processing:** +1. Load schemas from filesystem or git +2. Nest dotted field names into hierarchical structure +3. Merge multiple sources (ECS + experimental + custom) +4. Create intermediate fields for parents + +**Output:** Deeply nested field dictionary with minimal defaults + +**Key Functions:** +- `load_schemas()`: Main entry point +- `deep_nesting_representation()`: Convert flat to nested +- `nest_fields()`: Build nested hierarchy +- `merge_fields()`: Merge multiple sources + +**Example:** +```python +from schema import loader +fields = loader.load_schemas() +# Or from specific version: +fields = loader.load_schemas(ref='v8.10.0') +``` + +### 2. cleaner.py - Validation & Normalization + +**Purpose:** Validate schemas and apply sensible defaults + +**Input:** Nested fields from loader + +**Processing:** +1. Validate mandatory attributes present +2. Strip whitespace from all strings +3. Apply type-specific defaults (e.g., `ignore_above=1024` for keywords) +4. Expand shorthand notations (reuse locations) +5. Validate constraints (description length, examples, patterns) + +**Output:** Validated and enriched fields + +**Defaults Applied:** +- `group: 2` (fieldset priority) +- `root: false` (not a root fieldset) +- `ignore_above: 1024` (for keyword fields) +- `norms: false` (for text fields) +- `short: description` (if not specified) + +**Validation:** +- Mandatory attributes: name, title, description, type, level +- Short descriptions < 120 characters (strict mode) +- Valid regex patterns +- Example values match patterns/expected_values +- Field levels: core/extended/custom + +**Key Functions:** +- `clean()`: Main entry point +- `schema_cleanup()`: Process fieldsets +- `field_cleanup()`: Process fields +- `normalize_reuse_notation()`: Expand reuse shorthand + +**Example:** +```python +from schema import loader, cleaner +fields = loader.load_schemas() +cleaner.clean(fields, strict=False) # Warnings +cleaner.clean(fields, strict=True) # Exceptions +``` + +### 3. finalizer.py - Field Reuse & Name Calculation + +**Purpose:** Perform field reuse and calculate final field names + +**Input:** Cleaned fields + +**Processing:** + +**Phase 1: Field Reuse** +1. Organize reuses by order and type (foreign vs self) +2. For each order level: + a. Foreign reuses: Copy fieldset to different location (transitive) + b. Self-nestings: Copy fieldset into itself (non-transitive) +3. Mark reused fields with `original_fieldset` +4. Record reuse metadata in `reused_here` + +**Phase 2: Name Calculation** +1. Traverse all fields with path tracking +2. Calculate `flat_name`: full dotted name +3. Calculate `dashed_name`: kebab-case version +4. Calculate multi-field `flat_names` +5. Apply OTel reuse mappings + +**Output:** Complete field structure with all reuses and final names + +**Reuse Example:** +``` +Order 1: +- group → user.group (foreign reuse) + +Order 2: +- user (now with group) → destination.user (foreign reuse) + Result: destination.user.group exists! (transitive) +- process → process.parent (self-nesting) + Result: source.process.parent does NOT exist (non-transitive) +``` + +**Key Functions:** +- `finalize()`: Main entry point +- `perform_reuse()`: Execute reuse operations +- `calculate_final_values()`: Compute final names +- `field_finalizer()`: Calculate individual field names + +**Example:** +```python +from schema import loader, cleaner, finalizer +fields = loader.load_schemas() +cleaner.clean(fields) +finalizer.finalize(fields) +# Fields now have flat_name, dashed_name calculated +``` + +### 4. subset_filter.py - Subset Filtering (Optional) + +**Purpose:** Filter to include only specified fields + +Subset filtering is like a **whitelist** - you specify exactly which fields to include, and everything else is excluded. + +**Why Use Subsets:** + +- **Reduce field count:** Full ECS has ~850 fields. Subsets let you use only 50-100 fields for specific use cases +- **Performance:** Fewer fields = smaller mappings = better Elasticsearch performance +- **Simplicity:** Only the fields you actually need +- **Domain-specific:** Create subsets for web, security, infrastructure, etc. + +**Input:** Finalized fields (after reuse) + +**Processing:** +1. Load subset definition files +2. Extract matching fields recursively +3. Handle `docs_only` fields separately +4. Merge multiple subsets (union) + +**Output:** +- Filtered fields (main subset) +- Docs-only fields (separate) + +--- + +## Understanding Subset Definitions + +A subset definition is a YAML file that mirrors the field structure, but only includes what you want: + +### Basic Subset Structure + +```yaml +name: minimal # Subset name (used for output directory) +fields: # Top-level: list fieldsets to include + base: # Fieldset name + fields: '*' # '*' = include ALL fields in this fieldset + + http: # Another fieldset + fields: # Specify which fields to include + request: # Nested field + fields: # Go deeper + method: {} # Include this field + bytes: {} # Include this field + response: + fields: '*' # Include ALL response fields +``` + +### Visual Representation + +**Before Subset (Full ECS):** +``` +base +├─ @timestamp +├─ message +├─ tags +└─ labels + +http +├─ request +│ ├─ method +│ ├─ bytes +│ ├─ referrer +│ └─ body +└─ response + ├─ status_code + ├─ bytes + └─ body + +user +├─ name +├─ email +└─ id +``` + +**Subset Definition:** +```yaml +name: minimal +fields: + base: + fields: '*' # All base fields + http: + fields: + request: + fields: + method: {} # Just method + bytes: {} # Just bytes +``` + +**After Subset:** +``` +base ✓ (all fields kept) +├─ @timestamp +├─ message +├─ tags +└─ labels + +http ✓ (partially kept) +├─ request +│ ├─ method ✓ (explicitly included) +│ ├─ bytes ✓ (explicitly included) +│ ├─ referrer ✗ (not in subset) +│ └─ body ✗ (not in subset) +└─ response ✗ (entire section excluded) + +user ✗ (not in subset at all) +``` + +--- + +## Field Options in Subsets + +Beyond just including fields, you can set options: + +### Disable Indexing + +```yaml +http: + fields: + request: + fields: + body: + index: false # Don't index this field + enabled: false # Don't process at all +``` + +**Result:** `http.request.body` exists but isn't indexed (saves space, still in _source) + +### docs_only Fields + +```yaml +http: + fields: + request: + fields: + referrer: + docs_only: true # In documentation but not artifacts +``` + +**Result:** Field appears in markdown docs but NOT in Elasticsearch templates, Beats configs, etc. + +**Use Case:** Deprecated fields you still want documented for legacy data + +--- + +## Multiple Subsets (Union) + +You can specify multiple subset files - they're merged together: + +```bash +python generator.py \ + --subset subsets/base.yml subsets/web.yml \ + --semconv-version v1.24.0 +``` + +**Merging Logic:** +- Field in ANY subset → Included in result +- `enabled: false` in subset A, `enabled: true` in subset B → Result: `enabled: true` +- Union operation: More permissive wins + +**Example:** + +`subsets/base.yml`: +```yaml +fields: + base: + fields: '*' + http: + fields: + request: + fields: + method: {} +``` + +`subsets/security.yml`: +```yaml +fields: + http: + fields: + request: + fields: + bytes: {} # Different field + source: + fields: + ip: {} +``` + +**Merged Result:** +``` +base.* (from base.yml) +http.request.method (from base.yml) +http.request.bytes (from security.yml) +source.ip (from security.yml) +``` + +--- + +## Common Subset Pitfalls + +### ❌ Mistake 1: Forgetting Intermediate Fields + +**Wrong:** +```yaml +http: + fields: + method: {} # ❌ Wrong! method is under request +``` + +**Right:** +```yaml +http: + fields: + request: # ✓ Need intermediate field + fields: + method: {} +``` + +### ❌ Mistake 2: Including Fieldset Without Fields Key + +**Wrong:** +```yaml +base: {} # ❌ Missing fields key +``` + +**Right:** +```yaml +base: + fields: '*' # ✓ Must have fields +``` + +### ❌ Mistake 3: Using Wildcards at Wrong Level + +**Wrong:** +```yaml +fields: '*' # ❌ Can't wildcard top level +``` + +**Right:** +```yaml +fields: + base: + fields: '*' # ✓ Wildcard inside fieldset + http: + fields: '*' +``` + +--- + +## Subset Best Practices + +1. **Start with base:** Almost always include `base: {fields: '*'}` +2. **Be specific:** Only include fields you actually use +3. **Test thoroughly:** Generate and verify the output +4. **Document why:** Add comments explaining the subset purpose +5. **Version control:** Keep subset definitions in git +6. **Iterate:** Start small, add fields as needed + +--- + +**Key Functions:** +- `filter()`: Main entry point +- `extract_matching_fields()`: Recursive filtering +- `combine_all_subsets()`: Merge multiple subsets + +**Example:** +```python +from schema import subset_filter +fields, docs = subset_filter.filter( + fields, + ['subsets/minimal.yml'], + 'generated' +) +``` + +### 5. exclude_filter.py - Exclude Filtering (Optional) + +**Purpose:** Explicitly remove specified fields + +**Input:** Fields (optionally after subset filter) + +**Processing:** +1. Load exclude definition files +2. Remove specified fields +3. Auto-remove empty parents (except base) + +**Output:** Fields with exclusions removed + +**Exclude Definition:** +```yaml +- name: http + fields: + - name: request.referrer # Remove this field + - name: response.body +``` + +**Key Functions:** +- `exclude()`: Main entry point +- `exclude_fields()`: Remove matching fields +- `pop_field()`: Recursive removal + +**Example:** +```python +from schema import exclude_filter +fields = exclude_filter.exclude( + fields, + ['excludes/deprecated.yml'] +) +``` + +### 6. intermediate_files.py - Generate Intermediate Formats + +**Purpose:** Generate standardized intermediate YAML representations + +**Input:** Final processed fields + +**Processing:** +1. Generate flat format: `{flat_name: field_def}` +2. Generate nested format: `{fieldset: {fields: {...}}}` +3. Remove internal attributes (node_name, intermediate) +4. Filter non-root reusables (flat format only) + +**Output:** +- `ecs_flat.yml`: Flat dictionary +- `ecs_nested.yml`: Nested by fieldset +- `ecs.yml`: Raw debug format (optional) + +**Key Functions:** +- `generate()`: Main entry point +- `generate_flat_fields()`: Create flat representation +- `generate_nested_fields()`: Create nested representation + +**Example:** +```python +from generators import intermediate_files +nested, flat = intermediate_files.generate( + fields, + 'generated/ecs', + default_dirs=True +) +``` + +## Helper Modules + +### visitor.py - Field Traversal + +**Purpose:** Traverse deeply nested structures using visitor pattern + +**Functions:** +- `visit_fields()`: Call different functions for fieldsets vs fields +- `visit_fields_with_path()`: Pass path array to callback +- `visit_fields_with_memo()`: Pass accumulator object + +**Example:** +```python +from schema import visitor + +# Count all fields +count = {'total': 0} +def counter(details, memo): + memo['total'] += 1 +visitor.visit_fields_with_memo(fields, counter, count) +``` + +## Common Patterns + +### Running the Full Pipeline + +```python +from schema import loader, cleaner, finalizer +from generators import intermediate_files + +# Load schemas +fields = loader.load_schemas() + +# Clean and validate +cleaner.clean(fields, strict=False) + +# Perform reuse and calculate names +finalizer.finalize(fields) + +# Generate intermediate files +nested, flat = intermediate_files.generate( + fields, + 'generated/ecs', + default_dirs=True +) + +# Now ready for generators (es_template, beats, etc.) +``` + +### With Subset Filtering + +```python +from schema import subset_filter + +# ... run pipeline through finalizer ... + +# Apply subset filter +fields, docs = subset_filter.filter( + fields, + ['subsets/minimal.yml'], + 'generated' +) + +# Continue with generators +``` + +### With Exclude Filtering + +```python +from schema import exclude_filter + +# ... run pipeline through finalizer ... + +# Apply exclude filter +fields = exclude_filter.exclude( + fields, + ['excludes/deprecated.yml'] +) + +# Continue with generators +``` + +## Debugging Tips + +### View Intermediate Structure + +```python +import yaml + +# After loader +with open('debug_loaded.yml', 'w') as f: + yaml.dump(fields, f, default_flow_style=False) + +# After cleaner +with open('debug_cleaned.yml', 'w') as f: + yaml.dump(fields, f, default_flow_style=False) + +# After finalizer +with open('debug_finalized.yml', 'w') as f: + yaml.dump(fields, f, default_flow_style=False) +``` + +### Check Specific Field + +```python +# Find a specific field +def find_field(details): + if 'flat_name' in details['field_details']: + if details['field_details']['flat_name'] == 'http.request.method': + print(details['field_details']) + +from schema import visitor +visitor.visit_fields(fields, field_func=find_field) +``` + +### Validate Reuse + +```python +# Check what was reused where +for name, schema in fields.items(): + if 'reused_here' in schema['schema_details']: + print(f"{name} contains:") + for reuse in schema['schema_details']['reused_here']: + print(f" - {reuse['full']}") +``` + +## Extending the Pipeline + +### Adding New Validation + +Add to `cleaner.py`: + +```python +def my_custom_validation(field): + if 'my_custom_attr' in field['field_details']: + # Validate it + pass + +# In field_cleanup(): +def field_cleanup(field): + # ... existing code ... + my_custom_validation(field) +``` + +### Adding New Calculated Fields + +Add to `finalizer.py`: + +```python +def field_finalizer(details, path): + # ... existing calculations ... + + # Add new calculated field + details['field_details']['my_calculated'] = calculate_something(path) +``` + +### Adding New Filter Type + +Create new module like `custom_filter.py`: + +```python +def filter(fields, config): + # Your custom filtering logic + return filtered_fields +``` + +## Testing + +### Unit Tests + +Located in `scripts/tests/unit/`: +- `test_loader.py`: Schema loading +- `test_cleaner.py`: Validation +- `test_finalizer.py`: Reuse logic + +### Integration Tests + +Run full pipeline: +```bash +cd scripts +python3 generator.py --strict +``` + +## Related Documentation + +- [otel-integration.md](otel-integration.md) - OTel integration +- [markdown-generator.md](markdown-generator.md) - Markdown docs +- [intermediate-files.md](intermediate-files.md) - Intermediate formats +- [es-template.md](es-template.md) - Elasticsearch templates +- [ecs-helpers.md](ecs-helpers.md) - Utility functions +- [csv-generator.md](csv-generator.md) - CSV export +- [beats-generator.md](beats-generator.md) - Beats configs + +## Troubleshooting + +### Common Errors + +**ValueError: Missing mandatory attribute** +- Fix: Add required attribute to schema YAML +- Required: name, title, description, type, level + +**ValueError: Schema has root=true and cannot be reused** +- Fix: Don't try to reuse base or other root fieldsets +- Root fieldsets appear at document root, can't be nested + +**KeyError during reuse** +- Fix: Check reuse order; dependencies must be reused first +- Use `order: 1` for fieldsets that others depend on + +**Duplicate field names** +- Fix: Check for conflicting custom schemas +- Use `safe_merge_dicts` which raises on conflicts + +--- + +### Field Reuse Troubleshooting + +#### Problem: Field not appearing where expected + +**Symptom:** Expected `destination.user.group.id` but it doesn't exist + +**Cause:** Reuse order is wrong - `group` not reused into `user` before `user` reused into `destination` + +**Solution:** +```yaml +# Ensure correct order +group: + reusable: + order: 1 # ← FIRST + expected: + - user + +user: + reusable: + order: 2 # ← SECOND + expected: + - destination +``` + +**How to verify:** +```python +# Check what's in destination.user +from schema import visitor + +def show_fields(details): + if 'flat_name' in details['field_details']: + name = details['field_details']['flat_name'] + if name.startswith('destination.user'): + print(name) + +visitor.visit_fields(fields, field_func=show_fields) +``` + +#### Problem: Self-nesting appearing in reused locations + +**Symptom:** Expected `source.process.parent` NOT to exist, but it does + +**Cause:** Something went wrong with non-transitive logic, or it's actually foreign reuse + +**Solution:** +1. Check if `process.parent` is foreign reuse (wrong) or self-nesting (correct): +```yaml +process: + reusable: + expected: + - at: process # ← Self-nesting (correct) + as: parent + - source # ← Foreign reuse +``` + +2. If it's self-nesting, it should NOT appear at `source.process.parent` +3. If you WANT it everywhere, change to foreign reuse: +```yaml +# Create separate parent_process fieldset +parent_process: + reusable: + order: 1 + expected: + - at: process + as: parent +``` + +#### Problem: Reused fields have wrong OTel mappings + +**Symptom:** `destination.user.name` has different OTel mapping than `user.name` + +**Cause:** Need to use `otel_reuse` for location-specific mappings + +**Solution:** +```yaml +# In user schema +- name: name + otel_reuse: + - ecs: destination.user.name # ← Specific location + mapping: + relation: equivalent + attribute: destination.user.name + - ecs: source.user.name + mapping: + relation: equivalent + attribute: source.user.name +``` + +#### Problem: Can't reuse fieldset + +**Symptom:** `ValueError: Schema X has attribute root=true and cannot be reused` + +**Cause:** Trying to reuse a root fieldset (`base`, etc.) + +**Why:** Root fieldsets have fields at document root level. Can't nest them. + +**Solution:** Don't reuse root fieldsets. If you need similar functionality, create a new non-root fieldset. + +--- + +### Subset Filtering Troubleshooting + +#### Problem: Subset includes too many fields + +**Symptom:** Wanted 50 fields, got 200 + +**Cause:** Used `fields: '*'` wildcard on wrong fieldsets + +**Solution:** Be more specific: +```yaml +# Too broad +http: + fields: '*' # ← Gets ALL http fields + +# More specific +http: + fields: + request: + fields: + method: {} + bytes: {} +``` + +**How to verify field count:** +```bash +# Count lines in CSV (minus header) +wc -l generated/csv/fields.csv +# Or +grep -c "^" generated/csv/fields.csv +``` + +#### Problem: Subset excludes fields I need + +**Symptom:** Missing `http.request.method` in generated artifacts + +**Cause 1:** Forgot to include it in subset definition + +**Solution:** +```yaml +http: + fields: + request: + fields: + method: {} # ← Must explicitly include +``` + +**Cause 2:** Forgot intermediate fields in path + +**Solution:** +```yaml +# Wrong - missing 'request' intermediate +http: + fields: + method: {} # ❌ + +# Right - include full path +http: + fields: + request: # ✓ + fields: + method: {} +``` + +**How to debug:** +```bash +# Check what's in flat YAML +grep "http.request.method" generated/ecs/ecs_flat.yml + +# If nothing found, field wasn't included in subset +``` + +#### Problem: ValueError: 'fields' key expected, not found + +**Symptom:** `ValueError: 'fields' key expected, not found in subset for http` + +**Cause:** Schema has nested fields but subset doesn't specify them + +**Solution:** +```yaml +# Wrong +http: {} # ❌ Missing fields key + +# Right +http: + fields: '*' # ✓ Or specific fields +``` + +#### Problem: ValueError: 'fields' key not expected + +**Symptom:** `ValueError: 'fields' key not expected, found in subset for @timestamp` + +**Cause:** Trying to add nested fields to a leaf field (one that doesn't have children) + +**Solution:** +```yaml +# Wrong - @timestamp is a leaf field, can't have nested fields +base: + fields: + @timestamp: + fields: # ❌ @timestamp doesn't have nested fields + value: {} + +# Right - @timestamp is included as-is +base: + fields: + @timestamp: {} # ✓ Just include it +``` + +#### Problem: Subset doesn't include reused fields + +**Symptom:** Subset has `destination` but not `destination.user.*` + +**Cause:** Subset filtering happens AFTER reuse, must include destination in subset + +**Solution:** +```yaml +# Include both the parent and nested fields +destination: + fields: + ip: {} + port: {} + user: # ← Include reused fieldset + fields: + name: {} + email: {} +``` + +**Remember:** Subset sees the FINAL structure after reuse. If `user` is reused at `destination.user`, your subset must explicitly include `destination.user` fields. + +#### Problem: Multiple subsets not merging as expected + +**Symptom:** Field in subset A but not in final output + +**Cause:** Typo in subset definition or field path. Check each subset independently and verify field paths match the schema structure. + +--- + +### Strict Mode Issues + +If `--strict` fails with warnings: +- Review the warning messages +- Fix schema YAMLs to meet requirements +- Or run without `--strict` (warnings only) diff --git a/scripts/generator.py b/scripts/generator.py index fafa5abde7..155717fd3c 100644 --- a/scripts/generator.py +++ b/scripts/generator.py @@ -15,6 +15,22 @@ # specific language governing permissions and limitations # under the License. +"""ECS Generator - Main Entry Point. + +Orchestrates the ECS artifact generation pipeline: +1. Load and validate YAML schemas (with optional git ref, custom schemas) +2. Perform field reuse and apply subset/exclude filters +3. Generate intermediate files (ecs_flat.yml, ecs_nested.yml) +4. Generate artifacts: CSV, Elasticsearch templates, Beats configs, Markdown docs + +Usage: + python scripts/generator.py --semconv-version v1.24.0 + python scripts/generator.py --ref v8.10.0 --semconv-version v1.24.0 + python scripts/generator.py --subset subsets/minimal.yml --semconv-version v1.24.0 + +See scripts/docs/schema-pipeline.md for complete documentation. +""" + import argparse import os from typing import ( @@ -40,6 +56,14 @@ def main() -> None: + """Main entry point for ECS artifact generation. + + Runs the complete pipeline: load schemas → clean → finalize (reuse) → filter → + validate OTel → generate intermediate files → generate all artifacts. + + Raises: + KeyError: If --semconv-version not provided + """ args = argument_parser() if not args.semconv_version: @@ -98,6 +122,7 @@ def main() -> None: def argument_parser() -> argparse.Namespace: + """Parse command-line arguments. Run with --help for all options.""" parser = argparse.ArgumentParser() parser.add_argument('--ref', action='store', help='Loads fields definitions from `./schemas` subdirectory from specified git reference. \ Note that "--include experimental/schemas" will also respect this git ref.') @@ -130,6 +155,7 @@ def argument_parser() -> argparse.Namespace: def read_version(ref: Optional[str] = None) -> str: + """Read ECS version from local 'version' file or from a git ref.""" if ref: print('Loading schemas from git ref ' + ref) tree = ecs_helpers.get_tree_by_ref(ref) diff --git a/scripts/generators/beats.py b/scripts/generators/beats.py index 42c2ec09b8..c857e25f36 100644 --- a/scripts/generators/beats.py +++ b/scripts/generators/beats.py @@ -15,6 +15,14 @@ # specific language governing permissions and limitations # under the License. +"""Beats Field Definition Generator. + +Generates field definitions for Elastic Beats (generated/beats/fields.ecs.yml). +Uses an allowlist (beats_default_fields_allowlist.yml) to mark which of the ~850 +ECS fields should be default_field=true (loaded by default for performance reasons). +Fields use contextual (relative) names within their group rather than full dotted paths. +""" + from os.path import join from collections import OrderedDict from typing import ( @@ -35,6 +43,12 @@ def generate( ecs_version: str, out_dir: str ) -> None: + """Generate Beats field definitions from ECS nested schemas. + + Process: filter non-root reusables → process base fieldset → process other + fieldsets (root=true fields added directly, others wrapped in groups) → apply + default_field allowlist → write YAML. + """ # base first ecs_nested = ecs_helpers.remove_top_level_reusable_false(ecs_nested) beats_fields: List[OrderedDict] = fieldset_field_array(ecs_nested['base']['fields'], ecs_nested['base']['prefix']) @@ -70,6 +84,12 @@ def generate( def set_default_field(fields, df_allowlist, df=False, path=''): + """Recursively set default_field flags based on allowlist. + + Default field logic: field in allowlist → true; top-level group → true; + parent is default → children inherit true; otherwise → false. + Modifies fields in place; recursively processes group fields and multi-fields. + """ for fld in fields: fld_df = fld.get('default_field', df) fld_path = fld['name'] @@ -89,6 +109,16 @@ def fieldset_field_array( source_fields: Dict[str, Field], fieldset_prefix: str ) -> List[OrderedDict]: + """Convert ECS fields to Beats field array format (sorted, contextual names). + + Strips to Beats-relevant properties and converts flat_names to relative names + within the group (e.g., 'http.request.method' → 'request.method' in http group). + + Main field properties kept: name, level, required, type, object_type, + ignore_above, multi_fields, format, input/output_format, output_precision, + description, example, enabled, index, doc_values, path, scaling_factor, pattern. + Multi-field properties kept: name, type, norms, default_field, normalizer, ignore_above. + """ allowed_keys: List[str] = [ 'name', 'level', @@ -150,6 +180,10 @@ def write_beats_yaml( ecs_version: str, out_dir: str ) -> None: + """Write Beats field definitions to YAML with a DO-NOT-EDIT header. + + Wraps the beats_file in an array because Beats expects a YAML array of documents. + """ ecs_helpers.make_dirs(join(out_dir, 'beats')) warning: str = file_header().format(version=ecs_version) ecs_helpers.yaml_dump(join(out_dir, 'beats/fields.ecs.yml'), [beats_file], preamble=warning) @@ -159,6 +193,7 @@ def write_beats_yaml( def file_header() -> str: + """Return DO-NOT-EDIT header string with {version} placeholder.""" return """ # WARNING! Do not edit this file directly, it was generated by the ECS project, # based on ECS version {version}. diff --git a/scripts/generators/csv_generator.py b/scripts/generators/csv_generator.py index 719b8914ce..12f8f9599c 100644 --- a/scripts/generators/csv_generator.py +++ b/scripts/generators/csv_generator.py @@ -15,6 +15,13 @@ # specific language governing permissions and limitations # under the License. +"""CSV Field Reference Generator. + +Generates generated/csv/fields.csv with one row per field (plus a row per multi-field). +Columns: ECS_Version, Indexed, Field_Set, Field, Type, Level, Normalization, Example, Description. +Base fields (no dots in name) are sorted first; all others alphabetically. +""" + import _csv import csv import sys @@ -31,12 +38,14 @@ def generate(ecs_flat: Dict[str, Field], version: str, out_dir: str) -> None: + """Generate generated/csv/fields.csv from the flat field dictionary.""" ecs_helpers.make_dirs(join(out_dir, 'csv')) sorted_fields = base_first(ecs_flat) save_csv(join(out_dir, 'csv/fields.csv'), sorted_fields, version) def base_first(ecs_flat: Dict[str, Field]) -> List[Field]: + """Sort fields: base fields (no dots) first, then all others alphabetically.""" base_list: List[Field] = [] sorted_list: List[Field] = [] for field_name in sorted(ecs_flat): @@ -48,6 +57,11 @@ def base_first(ecs_flat: Dict[str, Field]) -> List[Field]: def save_csv(file: str, sorted_fields: List[Field], version: str) -> None: + """Write sorted fields to CSV. Multi-fields get their own rows with empty normalization. + + Field_Set is 'base' for fields with no dots, otherwise the first path segment. + Indexed is 'true'/'false' (lowercase). Normalization is comma-separated or empty. + """ open_mode: str = "wb" if sys.version_info >= (3, 0): open_mode: str = "w" diff --git a/scripts/generators/ecs_helpers.py b/scripts/generators/ecs_helpers.py index ab688f8aea..1ed43fbbbc 100644 --- a/scripts/generators/ecs_helpers.py +++ b/scripts/generators/ecs_helpers.py @@ -15,6 +15,12 @@ # specific language governing permissions and limitations # under the License. +"""ECS Generator Helper Utilities. + +Shared utilities for dictionary operations, file I/O, git access, list manipulation, +and field introspection. Used across all ECS generator scripts. +""" + import glob import os import yaml @@ -43,6 +49,7 @@ def dict_copy_keys_ordered(dct: Field, copied_keys: List[str]) -> Field: + """Copy specified keys in order. Keys not in source are skipped.""" ordered_dict = OrderedDict() for key in copied_keys: if key in dct: @@ -51,12 +58,14 @@ def dict_copy_keys_ordered(dct: Field, copied_keys: List[str]) -> Field: def dict_copy_existing_keys(source: Field, destination: Field, keys: List[str]) -> None: + """Copy keys from source to destination if they exist in source.""" for key in keys: if key in source: destination[key] = source[key] def dict_sorted_by_keys(dct: FieldNestedEntry, sort_keys: List[str]) -> List[FieldNestedEntry]: + """Sort dictionary values by one or more keys, returning sorted list.""" if not isinstance(sort_keys, list): sort_keys = [sort_keys] @@ -80,6 +89,7 @@ def ordered_dict_insert( before_key: Optional[str] = None, after_key: Optional[str] = None ) -> None: + """Insert key-value pair before or after specified key in ordered dict.""" output = OrderedDict() inserted: bool = False for key, value in dct.items(): @@ -98,7 +108,7 @@ def ordered_dict_insert( def safe_merge_dicts(a: Dict[Any, Any], b: Dict[Any, Any]) -> Dict[Any, Any]: - """Merges two dictionaries into one. If duplicate keys are detected a ValueError is raised.""" + """Merge two dicts, raising ValueError on duplicate keys.""" c = deepcopy(a) for key in b: if key not in c: @@ -109,6 +119,7 @@ def safe_merge_dicts(a: Dict[Any, Any], b: Dict[Any, Any]) -> Dict[Any, Any]: def fields_subset(subset, fields): + """Recursively filter fields to only those specified in subset. 'fields': '*' includes all.""" retained_fields = {} allowed_options = ['fields'] for key, val in subset.items(): @@ -126,6 +137,7 @@ def fields_subset(subset, fields): def yaml_ordereddict(dumper, data): + """YAML representer for OrderedDict that preserves key order.""" # YAML representation of an OrderedDict will be like a dictionary, but # respecting the order of the dictionary. # Almost sure it's unndecessary with Python 3. @@ -137,11 +149,12 @@ def yaml_ordereddict(dumper, data): return yaml.nodes.MappingNode(u'tag:yaml.org,2002:map', value) +# Register the representer globally yaml.add_representer(OrderedDict, yaml_ordereddict) def dict_clean_string_values(dict: Dict[Any, Any]) -> None: - """Remove superfluous spacing in all field values of a dict""" + """Strip leading/trailing whitespace from all string values in dict (in place).""" for key in dict: value = dict[key] if isinstance(value, str): @@ -155,12 +168,12 @@ def dict_clean_string_values(dict: Dict[Any, Any]) -> None: def is_yaml(path: str) -> bool: - """Returns True if path matches an element of the yaml extensions set""" + """Check if a file path has a YAML extension (.yml or .yaml).""" return set(path.split('.')[1:]).intersection(YAML_EXT) != set() def safe_list(o: Union[str, List[str]]) -> List[str]: - """converts o to a list if it isn't already a list""" + """Convert a comma-separated string or list to a list.""" if isinstance(o, list): return o else: @@ -168,7 +181,7 @@ def safe_list(o: Union[str, List[str]]) -> List[str]: def glob_yaml_files(paths: List[str]) -> List[str]: - """Accepts string, or list representing a path, wildcard or folder. Returns list of matched yaml files""" + """Find all YAML files matching given paths, wildcards, or directories. Returns sorted list.""" all_files: List[str] = [] for path in safe_list(paths): if is_yaml(path): @@ -180,12 +193,14 @@ def glob_yaml_files(paths: List[str]) -> List[str]: def get_tree_by_ref(ref: str) -> git.objects.tree.Tree: + """Get git tree object for a specific branch, tag, or commit SHA.""" repo: git.repo.base.Repo = git.Repo(os.getcwd()) commit: git.objects.commit.Commit = repo.commit(ref) return commit.tree def path_exists_in_git_tree(tree: git.objects.tree.Tree, file_path: str) -> bool: + """Return True if file_path exists in the given git tree object.""" try: _ = tree[file_path] except KeyError: @@ -194,6 +209,7 @@ def path_exists_in_git_tree(tree: git.objects.tree.Tree, file_path: str) -> bool def usage_doc_files() -> List[str]: + """Return filenames matching ecs-*-usage.md in docs/reference, or [] if dir doesn't exist.""" usage_docs_dir: str = os.path.join(os.path.dirname(__file__), '../../docs/reference') usage_docs_path: pathlib.PosixPath = pathlib.Path(usage_docs_dir) if usage_docs_path.is_dir(): @@ -202,12 +218,17 @@ def usage_doc_files() -> List[str]: def ecs_files() -> List[str]: - """Return the schema file list to load""" + """Return sorted list of all YAML files in the schemas/ directory.""" schema_glob: str = os.path.join(os.path.dirname(__file__), '../../schemas/*.yml') return sorted(glob.glob(schema_glob)) def make_dirs(path: str) -> None: + """Create directory and all parent directories if they don't exist. + + Raises: + OSError: If directory creation fails (with descriptive message) + """ try: os.makedirs(path, exist_ok=True) except OSError as e: @@ -220,6 +241,7 @@ def yaml_dump( data: Dict[str, FieldNestedEntry], preamble: Optional[str] = None ) -> None: + """Write data to a YAML file, optionally prepending preamble text.""" with open(filename, 'w') as outfile: if preamble: outfile.write(preamble) @@ -227,6 +249,7 @@ def yaml_dump( def yaml_load(filename: str) -> Set[str]: + """Load and parse a YAML file using safe_load.""" with open(filename) as f: return yaml.safe_load(f.read()) @@ -234,12 +257,12 @@ def yaml_load(filename: str) -> Set[str]: def list_subtract(original: List[Any], subtracted: List[Any]) -> List[Any]: - """Subtract two lists. original = subtracted""" + """Return original with all elements of subtracted removed.""" return [item for item in original if item not in subtracted] def list_extract_keys(lst: List[Field], key_name: str) -> List[str]: - """Returns an array of values for 'key_name', from a list of dictionaries""" + """Extract the value of key_name from each dict in lst.""" acc = [] for d in lst: acc.append(d[key_name]) @@ -250,12 +273,12 @@ def list_extract_keys(lst: List[Field], key_name: str) -> List[str]: def is_intermediate(field: FieldEntry) -> bool: - """Encapsulates the check to see if a field is an intermediate field or a "real" field.""" + """Return True if field is a structural placeholder (not an actual data field).""" return ('intermediate' in field['field_details'] and field['field_details']['intermediate']) def remove_top_level_reusable_false(ecs_nested: Dict[str, FieldNestedEntry]) -> Dict[str, FieldNestedEntry]: - """Returns same structure as ecs_nested, but skips all field sets with reusable.top_level: False""" + """Return ecs_nested excluding fieldsets with reusable.top_level=false.""" components: Dict[str, FieldNestedEntry] = {} for (fieldset_name, fieldset) in ecs_nested.items(): if fieldset.get('reusable', None): @@ -269,11 +292,6 @@ def remove_top_level_reusable_false(ecs_nested: Dict[str, FieldNestedEntry]) -> def strict_warning(msg: str) -> None: - """Call warnings.warn(msg) for operations that would throw an Exception - if operating in `--strict` mode. Allows a custom message to be passed. - - :param msg: custom text which will be displayed with wrapped boilerplate - for strict warning messages. - """ + """Issue a UserWarning that becomes a ValueError when running with --strict.""" warn_message: str = f"{msg}\n\nThis will cause an exception when running in strict mode.\nWarning check:" warnings.warn(warn_message, stacklevel=3) diff --git a/scripts/generators/es_template.py b/scripts/generators/es_template.py index f72bf7ba1b..dd1736a922 100644 --- a/scripts/generators/es_template.py +++ b/scripts/generators/es_template.py @@ -15,6 +15,15 @@ # specific language governing permissions and limitations # under the License. +"""Elasticsearch Template Generator. + +Generates Elasticsearch index templates from ECS schemas: +- Composable (modern, ES 7.8+): one component per fieldset + main template +- Legacy (deprecated): single monolithic template + +Output: generated/elasticsearch/composable/ and generated/elasticsearch/legacy/ +""" + import json import sys from typing import ( @@ -42,13 +51,14 @@ def generate( mapping_settings_file: str, template_settings_file: str ) -> None: - """This generates all artifacts for the composable template approach""" + """Generate composable component templates (one per fieldset) and main template.json.""" all_component_templates(ecs_nested, ecs_version, out_dir) component_names = component_name_convention(ecs_version, ecs_nested) save_composable_template(ecs_version, component_names, out_dir, mapping_settings_file, template_settings_file) def save_composable_template(ecs_version, component_names, out_dir, mapping_settings_file, template_settings_file): + """Save elasticsearch/composable/template.json referencing all component templates.""" mappings_section = mapping_settings(mapping_settings_file) template = template_settings(ecs_version, mappings_section, template_settings_file, component_names=component_names) @@ -61,7 +71,7 @@ def all_component_templates( ecs_version: str, out_dir: str ) -> None: - """Generate one component template per field set""" + """Generate one component template JSON per fieldset in elasticsearch/composable/component/.""" component_dir: str = join(out_dir, 'elasticsearch/composable/component') ecs_helpers.make_dirs(component_dir) @@ -81,6 +91,7 @@ def save_component_template( out_dir: str, field_mappings: Dict ) -> None: + """Save {template_name}.json component template with field_mappings and _meta. Docs URL added for non-custom fields.""" filename: str = join(out_dir, template_name) + ".json" reference_url: str = "https://www.elastic.co/guide/en/ecs/current/ecs-{}.html".format(template_name) @@ -102,6 +113,7 @@ def component_name_convention( ecs_version: str, ecs_nested: Dict[str, FieldNestedEntry] ) -> List[str]: + """Return list of component template names as 'ecs_{version}_{fieldset}' ('+' → '-' in version).""" version: str = ecs_version.replace('+', '-') names: List[str] = [] for (fieldset_name, fieldset) in ecs_helpers.remove_top_level_reusable_false(ecs_nested).items(): @@ -119,7 +131,7 @@ def generate_legacy( mapping_settings_file: str, template_settings_file: str ) -> None: - """Generate the legacy index template""" + """Generate elasticsearch/legacy/template.json with all fields in a single monolithic template.""" field_mappings = {} for flat_name in sorted(ecs_flat): field = ecs_flat[flat_name] @@ -138,6 +150,7 @@ def generate_legacy_template_version( out_dir: str, template_settings_file: str ) -> None: + """Build and save the legacy template JSON to elasticsearch/legacy/template.json.""" ecs_helpers.make_dirs(join(out_dir, 'elasticsearch', "legacy")) template: Dict = template_settings(ecs_version, mappings_section, template_settings_file, is_legacy=True) @@ -153,6 +166,11 @@ def dict_add_nested( name_parts: List[str], value: Dict ) -> None: + """Recursively place value at the path defined by name_parts using nested 'properties' dicts. + + E.g., ['http', 'request', 'method'] → {http: {properties: {request: {properties: {method: value}}}}} + Skips if leaf already exists as type=object (avoids overwriting). + """ current_nesting: str = name_parts[0] rest_name_parts: List[str] = name_parts[1:] if len(rest_name_parts) > 0: @@ -171,6 +189,13 @@ def dict_add_nested( def entry_for(field: Field) -> Dict: + """Convert an ECS field to an Elasticsearch mapping dict. + + Type-specific params copied: keyword/flattened→ignore_above, text→norms, + alias→path, scaled_float→scaling_factor, constant_keyword→value, + object/nested→enabled (if false), index=false→doc_values. + multi_fields and 'parameters' dicts are merged in. + """ field_entry: Dict = {'type': field['type']} try: if field['type'] == 'object' or field['type'] == 'nested': @@ -214,6 +239,7 @@ def entry_for(field: Field) -> Dict: def mapping_settings(mapping_settings_file: str) -> Dict: + """Return mapping settings from file or default_mapping_settings().""" if mapping_settings_file: with open(mapping_settings_file) as f: mappings = json.load(f) @@ -229,6 +255,7 @@ def template_settings( is_legacy: Optional[bool] = False, component_names: Optional[List[str]] = None ) -> Dict: + """Load template settings from file or defaults, then finalize with mappings and metadata.""" if template_settings_file: with open(template_settings_file) as f: template = json.load(f) @@ -250,6 +277,11 @@ def finalize_template( mappings_section: Dict, component_names: List[str] ) -> None: + """Merge mappings and metadata into template in place. + + Legacy: mappings at root, _meta moved inside mappings. + Composable: mappings under template.mappings, composed_of and _meta at root. + """ if is_legacy: if mappings_section: template['mappings'] = mappings_section @@ -269,6 +301,7 @@ def finalize_template( def save_json(file: str, data: Dict) -> None: + """Write data to file as JSON with 2-space indent, sorted keys, and trailing newline.""" open_mode = "wb" if sys.version_info >= (3, 0): open_mode = "w" @@ -278,6 +311,7 @@ def save_json(file: str, data: Dict) -> None: def default_template_settings(ecs_version: str) -> Dict: + """Return default composable template settings (try-ecs-* pattern, priority=1, best_compression).""" return { "index_patterns": ["try-ecs-*"], "_meta": { @@ -301,6 +335,7 @@ def default_template_settings(ecs_version: str) -> Dict: def default_legacy_template_settings(ecs_version: str) -> Dict: + """Return default legacy template settings (try-ecs-*, order=1, total_fields.limit=10000).""" return { "index_patterns": ["try-ecs-*"], "_meta": {"version": ecs_version}, @@ -319,6 +354,7 @@ def default_legacy_template_settings(ecs_version: str) -> Dict: def default_mapping_settings() -> Dict: + """Return default mapping settings: date_detection=false, strings_as_keyword dynamic template.""" return { "date_detection": False, "dynamic_templates": [ diff --git a/scripts/generators/intermediate_files.py b/scripts/generators/intermediate_files.py index 94787c4156..df045445ae 100644 --- a/scripts/generators/intermediate_files.py +++ b/scripts/generators/intermediate_files.py @@ -15,6 +15,15 @@ # specific language governing permissions and limitations # under the License. +"""Intermediate File Generator. + +Produces two normalized representations of processed ECS schemas: +- Flat (ecs_flat.yml): {dotted_field_name: field_def}, excludes top_level=false fieldsets +- Nested (ecs_nested.yml): {fieldset_name: {metadata, fields: {...}}}, includes all fieldsets + +These are the stable interfaces consumed by all downstream generators. +""" + import copy from os.path import join from typing import ( @@ -36,6 +45,11 @@ def generate( out_dir: str, default_dirs: bool ) -> Tuple[Dict[str, FieldNestedEntry], Dict[str, Field]]: + """Generate flat and nested intermediate YAML files from processed schemas. + + Returns (nested, flat) dicts. Also saves ecs_flat.yml, ecs_nested.yml, + and (if default_dirs=True) ecs.yml for debugging. + """ ecs_helpers.make_dirs(join(out_dir)) # Should only be used for debugging ECS development @@ -50,7 +64,7 @@ def generate( def generate_flat_fields(fields: Dict[str, FieldEntry]) -> Dict[str, Field]: - """Generate ecs_flat.yml""" + """Return {flat_name: field_def} for all non-intermediate, non-reusable-only fields.""" filtered: Dict[str, FieldEntry] = remove_non_root_reusables(fields) flattened: Dict[str, Field] = {} visitor.visit_fields_with_memo(filtered, accumulate_field, flattened) @@ -58,7 +72,7 @@ def generate_flat_fields(fields: Dict[str, FieldEntry]) -> Dict[str, Field]: def accumulate_field(details: FieldEntry, memo: Field) -> None: - """Visitor function that accumulates all field details in the memo dict""" + """Visitor callback: add field to memo dict by flat_name, skipping schema-level and intermediate entries.""" if 'schema_details' in details or ecs_helpers.is_intermediate(details): return field_details: Field = copy.deepcopy(details['field_details']) @@ -69,7 +83,11 @@ def accumulate_field(details: FieldEntry, memo: Field) -> None: def generate_nested_fields(fields: Dict[str, FieldEntry]) -> Dict[str, FieldNestedEntry]: - """Generate ecs_nested.yml""" + """Return {fieldset_name: {metadata, fields: {flat_name: field_def}}} for ALL fieldsets. + + Unlike generate_flat_fields(), this includes top_level=false fieldsets. + Each fieldset's 'fields' dict is flat (keyed by flat_name), not hierarchical. + """ nested: Dict[str, FieldNestedEntry] = {} # Flatten each field set, but keep all resulting fields nested under their # parent/host field set. @@ -101,23 +119,13 @@ def generate_nested_fields(fields: Dict[str, FieldEntry]) -> Dict[str, FieldNest def remove_internal_attributes(field_details: Field) -> None: - """Remove attributes only relevant to the deeply nested structure, but not to ecs_flat/nested.yml.""" + """Remove node_name and intermediate attributes from field definitions before output.""" field_details.pop('node_name', None) field_details.pop('intermediate', None) def remove_non_root_reusables(fields_nested: Dict[str, FieldEntry]) -> Dict[str, FieldEntry]: - """ - Remove field sets that have top_level=false from the root of the field definitions. - - This attribute means they're only meant to be in the "reusable/expected" locations - and not at the root of user's events. - - This is only relevant for the 'flat' field representation. The nested one - still needs to keep all field sets at the root of the YAML file, as it - the official information about each field set. It's the responsibility of - users consuming ecs_nested.yml to skip the field sets with top_level=false. - """ + """Filter out fieldsets with reusable.top_level=false (only applied to flat representation).""" fields: Dict[str, FieldEntry] = {} for (name, field) in fields_nested.items(): if 'reusable' not in field['schema_details'] or field['schema_details']['reusable']['top_level']: diff --git a/scripts/generators/markdown_fields.py b/scripts/generators/markdown_fields.py index 87be2acb8c..dce732bf52 100644 --- a/scripts/generators/markdown_fields.py +++ b/scripts/generators/markdown_fields.py @@ -15,6 +15,13 @@ # specific language governing permissions and limitations # under the License. +"""Markdown Documentation Generator. + +Generates markdown docs using Jinja2 templates from scripts/templates/: +- index.md, ecs-field-reference.md, ecs-{fieldset}.md per fieldset +- ecs-otel-alignment-details.md and ecs-otel-alignment-overview.md +""" + from functools import wraps import os.path as path import os @@ -26,6 +33,7 @@ def generate(nested, docs_only_nested, ecs_generated_version, semconv_version, otel_generator, out_dir): + """Generate all markdown docs: index, field reference, per-fieldset pages, and OTel alignment pages.""" ecs_helpers.make_dirs(out_dir) @@ -48,11 +56,7 @@ def generate(nested, docs_only_nested, ecs_generated_version, semconv_version, o def render_fieldset_reuse_text(fieldset): - """Renders the expected nesting locations - if the the `reusable` object is present. - - :param fieldset: The fieldset to evaluate - """ + """Return sorted list of full reuse paths for reusable fieldsets, or None if not reusable.""" if not fieldset.get('reusable'): return None reusable_fields = fieldset['reusable']['expected'] @@ -61,10 +65,7 @@ def render_fieldset_reuse_text(fieldset): def render_nestings_reuse_section(fieldset): - """Renders the reuse section entries. - - :param fieldset: The target fieldset - """ + """Return sorted list of reused_here entries with flat_nesting (path.*), or None.""" if not fieldset.get('reused_here'): return None rows = [] @@ -81,26 +82,14 @@ def render_nestings_reuse_section(fieldset): def extract_allowed_values_key_names(field): - """Extracts the `name` keys from the field's - allowed_values if present in the field - object. - - :param field: The target field - """ + """Return list of allowed value names for a field, or [] if none.""" if not field.get('allowed_values'): return [] return ecs_helpers.list_extract_keys(field['allowed_values'], 'name') def sort_fields(fieldset): - """Prepares a fieldset's fields for being - passed into the j2 template for rendering. This - includes sorting them into a list of objects and - adding a field for the names of any allowed values - for the field, if present. - - :param fieldset: The target fieldset - """ + """Return fieldset fields as a list sorted by name, with allowed_value_names added to each field.""" fields_list = list(fieldset['fields'].values()) for field in fields_list: field['allowed_value_names'] = extract_allowed_values_key_names(field) @@ -108,19 +97,12 @@ def sort_fields(fieldset): def check_for_usage_doc(fieldset_name, usage_file_list=ecs_helpers.usage_doc_files()): - """Checks if a usage doc exists for the specified - fieldset. - - :param fieldset_name: The name of the target fieldset - """ + """Return True if ecs-{fieldset_name}-usage.md exists in the usage docs directory.""" return f"ecs-{fieldset_name}-usage.md" in usage_file_list def templated(template_name): - """Decorator function to simplify rendering a template. - - :param template_name: the name of the template to be rendered - """ + """Decorator that renders the function's dict return value through the named Jinja2 template.""" def decorator(func): @wraps(func) def decorated_function(*args, **kwargs): @@ -135,18 +117,13 @@ def decorated_function(*args, **kwargs): def render_template(template_name, **context): - """Renders a template from the template folder with the given - context. - - :param template_name: the name of the template to be rendered - :param context: the variables that should be available in the - context of the template. - """ + """Render a Jinja2 template from scripts/templates/ with the given context.""" template = template_env.get_template(template_name) return template.render(**context) def save_markdown(f, text): + """Write markdown text to file, creating parent directories as needed.""" os.makedirs(path.dirname(f), exist_ok=True) with open(f, "w") as outfile: outfile.write(text) @@ -169,6 +146,7 @@ def save_markdown(f, text): @templated('index.j2') def page_index(ecs_generated_version): + """Render index.md.""" return dict(ecs_generated_version=ecs_generated_version) @@ -177,6 +155,7 @@ def page_index(ecs_generated_version): @templated('fieldset.j2') def page_fieldset(fieldset, nested, ecs_generated_version): + """Render ecs-{fieldset.name}.md with fields, reuse, and nesting sections.""" sorted_reuse_fields = render_fieldset_reuse_text(fieldset) render_nestings_reuse_fields = render_nestings_reuse_section(fieldset) sorted_fields = sort_fields(fieldset) @@ -192,6 +171,7 @@ def page_fieldset(fieldset, nested, ecs_generated_version): @templated('ecs_field_reference.j2') def page_field_reference(ecs_generated_version, es, fieldsets): + """Render ecs-field-reference.md with all fieldsets.""" return dict(ecs_generated_version=ecs_generated_version, es=es, fieldsets=fieldsets) @@ -201,6 +181,7 @@ def page_field_reference(ecs_generated_version, es, fieldsets): def page_field_details(nested, docs_only_nested): + """Return concatenated field_details.j2 output for all fieldsets. Merges docs_only_nested in place.""" if docs_only_nested: for fieldset_name, fieldset in docs_only_nested.items(): nested[fieldset_name]['fields'].update(fieldset['fields']) @@ -211,6 +192,7 @@ def page_field_details(nested, docs_only_nested): @templated('field_details.j2') def generate_field_details_page(fieldset): + """Render field_details.j2 for one fieldset.""" # render field reuse text section sorted_reuse_fields = render_fieldset_reuse_text(fieldset) render_nestings_reuse_fields = render_nestings_reuse_section(fieldset) @@ -227,6 +209,7 @@ def generate_field_details_page(fieldset): @templated('otel_alignment_details.j2') def page_otel_alignment_details(nested, ecs_generated_version, semconv_version): + """Render ecs-otel-alignment-details.md for fieldsets with at least one otel mapping.""" fieldsets = [deepcopy(fieldset) for fieldset in ecs_helpers.dict_sorted_by_keys( nested, ['group', 'name']) if is_eligable_for_otel_mapping(fieldset)] for fieldset in fieldsets: @@ -239,6 +222,7 @@ def page_otel_alignment_details(nested, ecs_generated_version, semconv_version): def is_eligable_for_otel_mapping(fieldset): + """Return True if any field in fieldset has an 'otel' mapping.""" for field in fieldset['fields'].values(): if 'otel' in field: return True @@ -249,6 +233,7 @@ def is_eligable_for_otel_mapping(fieldset): @templated('otel_alignment_overview.j2') def page_otel_alignment_overview(otel_generator, nested, ecs_generated_version, semconv_version): + """Render ecs-otel-alignment-overview.md with mapping summaries from otel_generator.""" fieldsets = ecs_helpers.dict_sorted_by_keys(nested, ['group', 'name']) summaries = otel_generator.get_mapping_summaries(fieldsets) return dict(summaries=summaries, @@ -260,6 +245,7 @@ def page_otel_alignment_overview(otel_generator, nested, ecs_generated_version, @templated('field_values.j2') def page_field_values(nested, template_name='field_values_template.j2'): + """Render allowed-values docs for event.kind, event.category, event.type, event.outcome.""" category_fields = ['event.kind', 'event.category', 'event.type', 'event.outcome'] nested_fields = [] for cat_field in category_fields: diff --git a/scripts/generators/otel.py b/scripts/generators/otel.py index c6c0469535..01ded0d51f 100644 --- a/scripts/generators/otel.py +++ b/scripts/generators/otel.py @@ -1,3 +1,26 @@ +# Licensed to Elasticsearch B.V. under one or more contributor +# license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright +# ownership. Elasticsearch B.V. licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""OpenTelemetry Semantic Conventions Integration Module. + +Loads OTel semconv from GitHub, validates ECS field otel mappings, and generates +alignment summaries for documentation. See scripts/docs/otel-integration.md. +""" + import git import os import shutil @@ -27,7 +50,7 @@ def get_model_files( git_repo: str, semconv_version: str, ) -> List[OTelModelFile]: - """Loads OpenTelemetry Semantic Conventions model from GitHub""" + """Load OTel semconv model YAML files from the git repo at the given version. Raises KeyError if 'model' dir absent.""" target_dir = "model" tree: git.objects.tree.Tree = get_tree_by_url(git_repo, semconv_version) if ecs_helpers.path_exists_in_git_tree(tree, target_dir): @@ -39,7 +62,7 @@ def get_model_files( def get_attributes( model_files: List[OTelModelFile] ) -> Dict[str, OTelAttribute]: - """Retrieves (non-deprecated) OTel attributes from the model files""" + """Extract non-deprecated attributes from attribute_group model files, applying group prefixes.""" attributes: Dict[str, OTelAttribute] = {} for model_file in model_files: @@ -58,7 +81,7 @@ def get_attributes( def get_metrics( model_files: List[OTelModelFile] ) -> Dict[str, OTelAttribute]: - """Retrieves (non-deprecated) OTel metrics from the model files""" + """Extract non-deprecated metrics (type='metric') from model files, keyed by metric_name.""" metrics: Dict[str, OTelGroup] = {} for model_file in model_files: @@ -72,6 +95,7 @@ def collectOTelModelFiles( tree: git.objects.tree.Tree, level=0 ) -> List[OTelModelFile]: + """Recursively parse all .yml/.yaml files in a git tree into OTelModelFile objects.""" otel_model_files: List[OTelModelFile] = [] for entry in tree: if entry.type == "tree": @@ -87,6 +111,7 @@ def get_tree_by_url( url: str, git_ref: str, ) -> git.objects.tree.Tree: + """Return git tree for url at git_ref, using cached clone in LOCAL_TARGET_DIR_OTEL_SEMCONV.""" repo: git.repo.base.Repo clone_from_remote = False if os.path.exists(LOCAL_TARGET_DIR_OTEL_SEMCONV): @@ -110,6 +135,7 @@ def get_otel_attribute_name( field: Field, otel: OTelMapping ) -> str: + """Return OTel attribute name: field's flat_name for 'match', else otel['attribute'].""" if otel['relation'] == 'match': return field['flat_name'] elif 'attribute' in otel: @@ -122,20 +148,44 @@ def get_otel_attribute_name( def must_have(ecs_field_name, otel, relation_type, property): + """Validate that a required property exists in an OTel mapping. + + Args: + ecs_field_name: Name of the ECS field being validated + otel: OTel mapping configuration dictionary + relation_type: The relation type requiring this property + property: Name of the required property + + Raises: + ValueError: If the required property is missing + """ if property not in otel: raise ValueError( f"On field '{ecs_field_name}': An OTel mapping with relation type '{relation_type}' must specify the property '{property}'!") def must_not_have(ecs_field_name, otel, relation_type, property): + """Validate that a forbidden property does not exist in an OTel mapping. + + Args: + ecs_field_name: Name of the ECS field being validated + otel: OTel mapping configuration dictionary + relation_type: The relation type forbidding this property + property: Name of the forbidden property + + Raises: + ValueError: If the forbidden property is present + """ if property in otel: raise ValueError( f"On field '{ecs_field_name}': An OTel mapping with relation type '{relation_type}' must not have the property '{property}'!") class OTelGenerator: + """Loads OTel semconv, validates ECS otel mappings, and generates alignment summaries.""" def __init__(self, semconv_version: str): + """Load OTel semconv model files and extract attributes and metrics for the given version.""" model_files = get_model_files(OTEL_SEMCONV_GIT, semconv_version) self.attributes: Dict[str, OTelAttribute] = get_attributes(model_files) @@ -147,6 +197,7 @@ def __init__(self, semconv_version: str): self.semconv_version = semconv_version def __set_stability(self, details): + """Visitor callback: enrich each otel mapping with stability from OTel definitions.""" field_details = details['field_details'] if 'flat_name' in field_details and 'otel' in field_details: for otel in field_details['otel']: @@ -156,17 +207,36 @@ def __set_stability(self, details): otel['stability'] = self.attributes[get_otel_attribute_name(field_details, otel)]['stability'] def __check_metric_name(self, field_name, metric_name): + """Validate that a referenced metric exists in OTel semantic conventions. + + Args: + field_name: Name of the ECS field being validated + metric_name: OTel metric name to verify + + Raises: + ValueError: If the metric doesn't exist in the loaded conventions + """ if not metric_name in self.otel_metric_names: raise ValueError( f"On field '{field_name}': Metric '{metric_name}' does not exist in Semantic Conventions version {self.semconv_version}!") def __check_attribute_name(self, field_details, otel): + """Validate that a referenced attribute exists in OTel semantic conventions. + + Args: + field_details: ECS field definition + otel: OTel mapping configuration + + Raises: + ValueError: If the attribute doesn't exist in the loaded conventions + """ otel_attr_name = get_otel_attribute_name(field_details, otel) if not otel_attr_name in self.otel_attribute_names: raise ValueError( f"On field '{field_details['flat_name']}': Attribute '{otel_attr_name}' does not exist in Semantic Conventions version {self.semconv_version}!") def __check_mapping(self, details): + """Visitor callback: validate otel mapping structure and verify referenced attributes/metrics exist.""" field_details = details['field_details'] if 'flat_name' in field_details and (not 'intermediate' in field_details or not field_details['intermediate']): ecs_field_name = field_details['flat_name'] @@ -215,6 +285,7 @@ def validate_otel_mapping( self, field_entries: Dict[str, FieldEntry] ) -> None: + """Validate all otel mappings then enrich them with stability info.""" visitor.visit_fields(field_entries, None, self.__check_mapping) visitor.visit_fields(field_entries, None, self.__set_stability) @@ -222,6 +293,7 @@ def get_mapping_summaries( self, fieldsets: List[FieldNestedEntry], ) -> List[OTelMappingSummary]: + """Return alignment summary stats per ECS fieldset and OTel namespace, sorted alphabetically.""" summaries: List[OTelMappingSummary] = [] otel_namespaces = set([attr.split('.')[0] for attr in self.attributes.keys()]) diff --git a/scripts/schema/cleaner.py b/scripts/schema/cleaner.py index 84c6abef15..72a46a6e15 100644 --- a/scripts/schema/cleaner.py +++ b/scripts/schema/cleaner.py @@ -15,6 +15,22 @@ # specific language governing permissions and limitations # under the License. +"""Schema Cleaner Module. + +Validates, normalizes, and enriches schema definitions after loading. +Second stage of pipeline: loader.py → cleaner.py → finalizer.py + +Key operations: +- Validates mandatory attributes (name, title, description, type, level) +- Sets defaults (group=2, root=false, ignore_above=1024 for keywords, etc.) +- Expands reuse shorthand notation +- Validates descriptions, examples, and patterns + +In strict mode (--strict), warnings become exceptions. + +See scripts/docs/schema-pipeline.md for complete documentation. +""" + import re from typing import ( Dict, @@ -32,25 +48,19 @@ MultiField, ) -# This script performs a few cleanup functions in place, within the deeply nested -# 'fields' structure passed to `clean(fields)`. -# -# What happens here: -# -# - check that mandatory attributes are present, without which we can't do much. -# - cleans things up, like stripping spaces, sorting arrays -# - makes lots of defaults explicit -# - pre-calculate a few additional helpful fields -# - converts shorthands into full representation (e.g. reuse locations) -# -# This script only deals with field sets themselves and the fields defined -# inside them. It doesn't perform field reuse, and therefore doesn't -# deal with final field names either. - strict_mode: Optional[bool] # work-around from https://github.com/python/mypy/issues/5732 def clean(fields: Dict[str, Field], strict: Optional[bool] = False) -> None: + """Clean, validate, and enrich schema definitions in place. + + Args: + fields: Deeply nested field dictionary from loader.py + strict: If True, warnings become exceptions + + Raises: + ValueError: If mandatory attributes are missing or invalid + """ global strict_mode strict_mode = strict visitor.visit_fields(fields, fieldset_func=schema_cleanup, field_func=field_cleanup) @@ -60,6 +70,7 @@ def clean(fields: Dict[str, Field], strict: Optional[bool] = False) -> None: def schema_cleanup(schema: FieldEntry) -> None: + """Clean and enrich a fieldset: validate, set defaults, expand reuse notation.""" # Sanity check first schema_mandatory_attributes(schema) # trailing space cleanup @@ -87,7 +98,7 @@ def schema_cleanup(schema: FieldEntry) -> None: def schema_mandatory_attributes(schema: FieldEntry) -> None: - """Ensures for the presence of the mandatory schema attributes and raises if any are missing""" + """Validate mandatory attributes (name, title, description) are present.""" current_schema_attributes: List[str] = sorted(list(schema['field_details'].keys()) + list(schema['schema_details'].keys())) missing_attributes: List[str] = ecs_helpers.list_subtract(SCHEMA_MANDATORY_ATTRIBUTES, current_schema_attributes) @@ -105,7 +116,7 @@ def schema_mandatory_attributes(schema: FieldEntry) -> None: def schema_assertions_and_warnings(schema: FieldEntry) -> None: - """Additional checks on a fleshed out schema""" + """Validate short/beta/short_override descriptions after defaults are applied.""" single_line_short_description(schema, strict=strict_mode) if 'beta' in schema['field_details']: single_line_beta_description(schema, strict=strict_mode) @@ -114,19 +125,10 @@ def schema_assertions_and_warnings(schema: FieldEntry) -> None: def normalize_reuse_notation(schema: FieldEntry) -> None: - """ - Replace single word reuse shorthands from the schema YAMLs with the explicit {at: , as:} notation. - - When marking "user" as reusable under "destination" with the shorthand entry - `- destination`, this is expanded to the complete entry - `- { "at": "destination", "as": "user" }`. - The field set is thus nested at `destination.user.*`, with fields such as `destination.user.name`. - - The dictionary notation enables nesting a field set as a different name. - An example is nesting "process" fields to capture parent process details - at `process.parent.*`. - The dictionary notation `- { "at": "process", "as": "parent" }` will yield - fields such as `process.parent.pid`. + """Expand reuse shorthand ('destination') to explicit {at:, as:, full:} dict form. + + Both 'destination' (shorthand) and {'at': 'process', 'as': 'parent'} (explicit) + are normalized to {'at': ..., 'as': ..., 'full': 'at.as'}. """ if 'reusable' not in schema['schema_details']: return @@ -151,6 +153,10 @@ def normalize_reuse_notation(schema: FieldEntry) -> None: def field_cleanup(field: FieldDetails) -> None: + """Validate, strip whitespace, apply defaults, and validate constraints for a field. + + Intermediate fields (auto-generated structural fields) are skipped after validation. + """ field_mandatory_attributes(field) if ecs_helpers.is_intermediate(field): return @@ -163,6 +169,7 @@ def field_cleanup(field: FieldDetails) -> None: def field_defaults(field: FieldDetails) -> None: + """Apply defaults: short=description, normalize=[], type-specific defaults, multi-field names.""" field['field_details'].setdefault('short', field['field_details']['description']) field['field_details'].setdefault('normalize', []) field_or_multi_field_datatype_defaults(field['field_details']) @@ -174,7 +181,8 @@ def field_defaults(field: FieldDetails) -> None: def field_or_multi_field_datatype_defaults(field_details: Union[Field, MultiField]) -> None: - """Sets datatype-related defaults on a canonical field or multi-field entries.""" + """Apply type-specific defaults: keyword→ignore_above=1024, text→norms=false, + wildcard→strip index, index=false→doc_values=false+remove ignore_above.""" if field_details['type'] == 'keyword': field_details.setdefault('ignore_above', 1024) if field_details['type'] == 'text': @@ -192,7 +200,8 @@ def field_or_multi_field_datatype_defaults(field_details: Union[Field, MultiFiel def field_mandatory_attributes(field: FieldDetails) -> None: - """Ensures for the presence of the mandatory field attributes and raises if any are missing""" + """Raise ValueError if name/description/type/level are missing, or path/scaling_factor + for alias/scaled_float fields. Intermediate fields are skipped.""" if ecs_helpers.is_intermediate(field): return current_field_attributes: List[str] = sorted(field['field_details'].keys()) @@ -212,7 +221,10 @@ def field_mandatory_attributes(field: FieldDetails) -> None: def field_assertions_and_warnings(field: FieldDetails) -> None: - """Additional checks on a fleshed out field""" + """Validate short desc length, beta desc, pattern regex, example value, and level. + + Invalid level always raises ValueError. Other checks warn or raise based on strict_mode. + """ if not ecs_helpers.is_intermediate(field): # check short description length if in strict mode single_line_short_description(field, strict=strict_mode) @@ -227,13 +239,14 @@ def field_assertions_and_warnings(field: FieldDetails) -> None: ACCEPTABLE_FIELD_LEVELS) raise ValueError(msg) -# Common +# Common Validation Helpers SHORT_LIMIT = 120 def single_line_short_check(short_to_check: str, short_name: str) -> Union[str, None]: + """Return error message if short description has newlines or exceeds 120 chars, else None.""" short_length: int = len(short_to_check) if "\n" in short_to_check or short_length > SHORT_LIMIT: msg: str = "Short descriptions must be single line, and under {} characters (current length: {}).\n".format( @@ -246,7 +259,7 @@ def single_line_short_check(short_to_check: str, short_name: str) -> Union[str, def strict_warning_handler(message, strict): - """Handles warnings based on --strict mode""" + """Raise ValueError if strict=True, else issue a warning.""" if strict: raise ValueError(message) else: @@ -254,6 +267,7 @@ def strict_warning_handler(message, strict): def single_line_short_description(schema_or_field: FieldEntry, strict: Optional[bool] = True): + """Validate that short description is single line and under 120 chars.""" error: Union[str, None] = single_line_short_check( schema_or_field['field_details']['short'], schema_or_field['field_details']['name']) if error: @@ -261,6 +275,7 @@ def single_line_short_description(schema_or_field: FieldEntry, strict: Optional[ def single_line_short_override_description(schema_or_field: FieldEntry, strict: Optional[bool] = True): + """Validate that short_override descriptions in reuse entries are single line.""" for field in schema_or_field['schema_details']['reusable']['expected']: if not 'short_override' in field: continue @@ -270,9 +285,9 @@ def single_line_short_override_description(schema_or_field: FieldEntry, strict: def check_example_value(field: Union[List, FieldEntry], strict: Optional[bool] = True) -> None: - """ - Checks if value of the example field is of type list or dict. - Fails or warns (depending on strict mode) if so. + """Validate example: not a YAML object/array, matches pattern, in expected_values. + + Array fields (normalize='array') parse each value individually. """ example_value: str = field['field_details'].get('example', '') pattern: str = field['field_details'].get('pattern', '') @@ -309,6 +324,7 @@ def check_example_value(field: Union[List, FieldEntry], strict: Optional[bool] = def single_line_beta_description(schema_or_field: FieldEntry, strict: Optional[bool] = True) -> None: + """Validate that beta description is single line.""" if "\n" in schema_or_field['field_details']['beta']: msg: str = "Beta descriptions must be single line.\n" msg += f"Offending field or field set: {schema_or_field['field_details']['name']}" @@ -316,9 +332,7 @@ def single_line_beta_description(schema_or_field: FieldEntry, strict: Optional[b def validate_pattern_regex(field, strict=True): - """ - Validates if field['pattern'] contains a valid regular expression. - """ + """Validate that the pattern attribute is a syntactically valid regular expression.""" try: re.compile(field['pattern']) except re.error: diff --git a/scripts/schema/exclude_filter.py b/scripts/schema/exclude_filter.py index 324a16807f..526461ff0c 100644 --- a/scripts/schema/exclude_filter.py +++ b/scripts/schema/exclude_filter.py @@ -15,6 +15,13 @@ # specific language governing permissions and limitations # under the License. +"""Schema Exclude Filter Module. + +Removes specified fields from the schema (inverse of subset filtering). Used primarily +for deprecation testing — exclude fields to assess impact before actually removing them. +Parent fields are auto-removed when emptied, except 'base'. Runs after subset filtering. +""" + from typing import ( Dict, List, @@ -27,12 +34,9 @@ FieldNestedEntry, ) -# This script should be run downstream of the subset filters - it takes -# all ECS and custom fields already loaded by the latter and explicitly -# removes a subset, for example, to simulate impact of future removals - def exclude(fields: Dict[str, FieldEntry], exclude_file_globs: List[str]) -> Dict[str, FieldEntry]: + """Load exclude definitions and remove matching fields from schema.""" excludes: List[FieldNestedEntry] = load_exclude_definitions(exclude_file_globs) if excludes: @@ -42,6 +46,7 @@ def exclude(fields: Dict[str, FieldEntry], exclude_file_globs: List[str]) -> Dic def long_path(path_as_list: List[str]) -> str: + """Join path components with dots.""" return '.'.join([e for e in path_as_list]) @@ -51,7 +56,7 @@ def pop_field( path: List[str], removed: List[str] ) -> str: - """pops a field from yaml derived dict using path derived from ordered list of nodes""" + """Remove field at node_path, auto-removing empty parents (except 'base'). Returns flat_name removed.""" if node_path[0] in fields: if len(node_path) == 1: flat_name: str = long_path(path) @@ -81,7 +86,7 @@ def exclude_trace_path( path: List[str], removed: List[str] ) -> None: - """traverses paths to one or more nodes in a yaml derived dict""" + """Remove each field in item list. Raises ValueError if item has nested 'fields' (not supported).""" for list_item in item: node_path: List[str] = path.copy() # cater for name.with.dots @@ -98,7 +103,7 @@ def exclude_trace_path( def exclude_fields(fields: Dict[str, FieldEntry], excludes: List[FieldNestedEntry]) -> Dict[str, FieldEntry]: - """Traverses fields and eliminates any field which matches the excludes""" + """Apply all exclude definitions, removing each specified field and cleaning up empty parents.""" if excludes: for ex_list in excludes: for item in ex_list: @@ -107,6 +112,7 @@ def exclude_fields(fields: Dict[str, FieldEntry], excludes: List[FieldNestedEntr def load_exclude_definitions(file_globs: List[str]) -> List[FieldNestedEntry]: + """Load exclude YAML files. Returns [] if empty. Raises ValueError if none found.""" if not file_globs: return [] excludes: List[FieldNestedEntry] = loader.load_definitions(file_globs) diff --git a/scripts/schema/finalizer.py b/scripts/schema/finalizer.py index 43ede81a19..144d968f1f 100644 --- a/scripts/schema/finalizer.py +++ b/scripts/schema/finalizer.py @@ -15,34 +15,41 @@ # specific language governing permissions and limitations # under the License. +"""Schema Finalizer Module. + +Third stage of the pipeline: loader.py → cleaner.py → finalizer.py → intermediate_files.py + +Performs field reuse (composition) and calculates final field names. + +Two-phase reuse: +- Phase 1 (foreign): Copy fieldset into a different fieldset. Transitive — if 'group' is in + 'user', then 'destination.user' also gets 'destination.user.group.*'. +- Phase 2 (self-nesting): Copy fieldset into itself (e.g., process → process.parent). + NOT transitive — 'source.process.parent' does not exist even though process.parent does. + +The 'order' attribute controls sequence: order=1 runs before order=2 (default). Within each +order, Phase 1 runs before Phase 2. + +After reuse: calculates flat_name, dashed_name, and multi-field flat_names for all fields. +""" + import copy import re from schema import visitor -# This script takes the fleshed out deeply nested fields dictionary as emitted by -# cleaner.py, and performs field reuse in two phases, repeated for each reuse order, from highest -# priority to lowest. -# -# Phase 1 performs field reuse across field sets. E.g. `group` fields should also be under `user`. -# This type of reuse is then carried around if the receiving field set is also reused. -# In other words, user.group.* will be in other places where user is nested: -# source.user.* will contain source.user.group.* - -# Phase 2 performs field reuse where field sets are reused within themselves, with a different name. -# Examples are nesting `process` within itself, as `process.parent.*`, -# or nesting `user` within itself at `user.target.*`. -# This second kind of nesting is not carried around everywhere else the receiving field set is reused. -# So `user.target.*` is *not* carried over to `source.user.target*` when we reuse `user` under `source`. - def finalize(fields): - """Intended entrypoint of the finalizer.""" + """Perform reuse and calculate final field names (flat_name, dashed_name).""" perform_reuse(fields) calculate_final_values(fields) def order_reuses(fields): + """Return (foreign_reuses, self_nestings) each as {order: {schema_name: [reuse_entries]}}. + + Foreign reuses go to a different fieldset; self_nestings stay within the same fieldset. + """ foreign_reuses = {} self_nestings = {} for schema_name, schema in fields.items(): @@ -65,7 +72,7 @@ def order_reuses(fields): def perform_reuse(fields): - """Performs field reuse respecting order for both foreign reuses and self-nestings""" + """Execute all field reuse, processing each order level with Phase 1 (foreign) then Phase 2 (self-nesting).""" foreign_reuses, self_nestings = order_reuses(fields) # Process foreign reuses and self-nestings together, respecting order @@ -123,11 +130,7 @@ def perform_reuse(fields): def ensure_valid_reuse(reused_schema, destination_schema=None): - """ - Raise if either the reused schema or destination schema have root=true. - - Second param is optional, if testing for a self-nesting (where source=destination). - """ + """Raise ValueError if either schema has root=true (root fieldsets cannot participate in reuse).""" if reused_schema['schema_details']['root']: msg = "Schema {} has attribute root=true and therefore cannot be reused.".format( reused_schema['field_details']['name']) @@ -139,7 +142,7 @@ def ensure_valid_reuse(reused_schema, destination_schema=None): def append_reused_here(reused_schema, reuse_entry, destination_schema): - """Captures two ways of denoting what field sets are reused under a given field set""" + """Record reuse metadata on destination_schema: appends to 'nestings' (legacy) and 'reused_here'.""" # Legacy, too limited destination_schema['schema_details'].setdefault('nestings', []) destination_schema['schema_details']['nestings'] = sorted( @@ -163,7 +166,7 @@ def append_reused_here(reused_schema, reuse_entry, destination_schema): def set_original_fieldset(fields, original_fieldset): - """Recursively set the 'original_fieldset' attribute for all fields in a group of fields""" + """Recursively stamp all fields with original_fieldset (uses setdefault, preserves nested values).""" def func(details): # Don't override if already set (e.g. 'group' for user.group.* fields) details['field_details'].setdefault('original_fieldset', original_fieldset) @@ -171,7 +174,10 @@ def func(details): def field_group_at_path(dotted_path, fields): - """Returns the ['fields'] hash at the dotted_path.""" + """Return the 'fields' dict at the given dotted path. Creates it for object/group/nested types. + + Raises ValueError if path is missing or passes through a non-nestable field type. + """ path = dotted_path.split('.') nesting = fields for next_field in path: @@ -190,17 +196,12 @@ def field_group_at_path(dotted_path, fields): def calculate_final_values(fields): - """ - This function navigates all fields recursively. - - It populates a few more values for the fields, especially path-based values - like flat_name. - """ + """Calculate flat_name, dashed_name, multi-field names, and OTel mappings for all fields.""" visitor.visit_fields_with_path(fields, field_finalizer) def field_finalizer(details, path): - """This is the function called by the visitor to perform the work of calculate_final_values""" + """Visitor callback: compute flat_name, dashed_name, multi-field flat_names, and resolve otel_reuse.""" name_array = path + [details['field_details']['node_name']] flat_name = '.'.join(name_array) diff --git a/scripts/schema/loader.py b/scripts/schema/loader.py index 3ac9e8ad20..b4fc5543e3 100644 --- a/scripts/schema/loader.py +++ b/scripts/schema/loader.py @@ -15,6 +15,20 @@ # specific language governing permissions and limitations # under the License. +"""Schema Loader Module. + +Entry point for ECS schema processing pipeline. Loads YAML schemas from filesystem +or git refs and transforms them into a deeply nested structure. + +Key operations: +- Load from schemas/*.yml, experimental/schemas/, or custom paths +- Transform dotted field names (e.g., 'http.request.method') into nested dicts +- Create intermediate parent fields automatically +- Merge multiple schema sources safely + +See scripts/docs/schema-pipeline.md for complete documentation. +""" + import copy import git import glob @@ -35,42 +49,6 @@ SchemaDetails, ) -# Loads main ECS schemas and optional additional schemas. -# They are deeply nested, then merged together. -# This script doesn't fill in defaults other than the bare minimum for a predictable -# deeply nested structure. It doesn't concern itself with what "should be allowed" -# in being a good ECS citizen. It just loads things and merges them together. - -# The deeply nested structured returned by this script looks like this. -# -# [schema name]: { -# 'schema_details': { -# 'reusable': ... -# }, -# 'field_details': { -# 'type': ... -# }, -# 'fields': { -# [field name]: { -# 'field_details': { ... } -# 'fields': { -# -# (dotted key names replaced by deep nesting) -# [field name]: { -# 'field_details': { ... } -# 'fields': { -# } -# } -# } -# } -# } - -# Schemas at the top level always have all 3 keys populated. -# Leaf fields only have 'field_details' populated. -# Any intermediate field with other fields nested within them have 'fields' populated. -# Note that intermediate fields rarely have 'field_details' populated, but it's supported. -# Examples of this are 'dns.answers', 'observer.egress'. - EXPERIMENTAL_SCHEMA_DIR = 'experimental/schemas' @@ -79,7 +57,11 @@ def load_schemas( ref: Optional[str] = None, included_files: Optional[List[str]] = [] ) -> Dict[str, FieldEntry]: - """Loads ECS and custom schemas. They are returned deeply nested and merged.""" + """Load ECS schemas (from filesystem or git ref) plus any custom included_files. + + Experimental schemas are loaded from git if ref is specified. Custom schemas + are always loaded from filesystem. All sources are merged, with custom taking precedence. + """ # ECS fields (from git ref or not) schema_files_raw: Dict[str, FieldNestedEntry] = load_schemas_from_git( ref) if ref else load_schema_files(ecs_helpers.ecs_files()) @@ -103,6 +85,7 @@ def load_schemas( def load_schema_files(files: List[str]) -> Dict[str, FieldNestedEntry]: + """Load and merge multiple schema YAML files. Raises ValueError on duplicate names.""" fields_nested: Dict[str, FieldNestedEntry] = {} for f in files: new_fields: Dict[str, FieldNestedEntry] = read_schema_file(f) @@ -114,6 +97,7 @@ def load_schemas_from_git( ref: str, target_dir: Optional[str] = 'schemas' ) -> Dict[str, FieldNestedEntry]: + """Load YAML schemas directly from git objects at the given ref. Raises KeyError if target_dir is absent.""" tree: git.objects.tree.Tree = ecs_helpers.get_tree_by_ref(ref) fields_nested: Dict[str, FieldNestedEntry] = {} @@ -129,7 +113,7 @@ def load_schemas_from_git( def read_schema_file(file_name: str) -> Dict[str, FieldNestedEntry]: - """Read a raw schema yml file into a dict.""" + """Read and parse a YAML schema file from filesystem.""" with open(file_name) as f: raw: List[FieldNestedEntry] = yaml.safe_load(f.read()) return nest_schema(raw, file_name) @@ -139,7 +123,7 @@ def read_schema_blob( blob: git.objects.blob.Blob, ref: str ) -> Dict[str, FieldNestedEntry]: - """Read a raw schema yml git blob into a dict.""" + """Read and parse a YAML schema from a git blob object.""" content: str = blob.data_stream.read().decode('utf-8') raw: List[FieldNestedEntry] = yaml.safe_load(content) file_name: str = "{} (git ref {})".format(blob.name, ref) @@ -147,12 +131,7 @@ def read_schema_blob( def nest_schema(raw: List[FieldNestedEntry], file_name: str) -> Dict[str, FieldNestedEntry]: - """ - Raw schema files are an array of schema details: [{'name': 'base', ...}] - - This function loops over the array (usually 1 schema per file) and turns it into - a dict with the schema name as the key: { 'base': { 'name': 'base', ...}} - """ + """Convert schema YAML array to dict keyed by 'name'. Raises ValueError if name is missing.""" fields: Dict[str, FieldNestedEntry] = {} for schema in raw: if 'name' not in schema: @@ -162,6 +141,12 @@ def nest_schema(raw: List[FieldNestedEntry], file_name: str) -> Dict[str, FieldN def deep_nesting_representation(fields: Dict[str, FieldNestedEntry]) -> Dict[str, FieldEntry]: + """Transform flat schema definitions into deeply nested field structures. + + Converts dotted field names (e.g., 'request.method') into nested dicts, + separating schema_details from field_details and creating intermediate + parent fields automatically. + """ deeply_nested: Dict[str, FieldEntry] = {} for (name, flat_schema) in fields.items(): @@ -188,6 +173,11 @@ def deep_nesting_representation(fields: Dict[str, FieldNestedEntry]) -> Dict[str def nest_fields(field_array: List[Field]) -> Dict[str, Dict[str, FieldEntry]]: + """Convert flat array of fields with dotted names into nested structure. + + Splits dotted names (e.g., 'request.method') and creates intermediate + parent fields (type='object', intermediate=True) automatically. + """ schema_root: Dict[str, Dict[str, FieldEntry]] = {'fields': {}} for field in field_array: nested_levels: List[str] = field['name'].split('.') @@ -226,6 +216,7 @@ def nest_fields(field_array: List[Field]) -> Dict[str, Dict[str, FieldEntry]]: def array_of_maps_to_map(array_vals: List[MultiField]) -> Dict[str, MultiField]: + """Convert list of multi-field dicts to {name: dict}. Last entry wins on duplicates.""" ret_map: Dict[str, MultiField] = {} for map_val in array_vals: name: str = map_val['name'] @@ -235,6 +226,7 @@ def array_of_maps_to_map(array_vals: List[MultiField]) -> Dict[str, MultiField]: def map_of_maps_to_array(map_vals: Dict[str, MultiField]) -> List[MultiField]: + """Convert {name: dict} to sorted list of multi-field dicts.""" ret_list: List[MultiField] = [] for key in map_vals: ret_list.append(map_vals[key]) @@ -242,13 +234,15 @@ def map_of_maps_to_array(map_vals: Dict[str, MultiField]) -> List[MultiField]: def dedup_and_merge_lists(list_a: List[MultiField], list_b: List[MultiField]) -> List[MultiField]: + """Merge two multi-field lists; list_b takes precedence on duplicate names.""" list_a_map: Dict[str, MultiField] = array_of_maps_to_map(list_a) list_a_map.update(array_of_maps_to_map(list_b)) return map_of_maps_to_array(list_a_map) def merge_fields(a: Dict[str, FieldEntry], b: Dict[str, FieldEntry]) -> Dict[str, FieldEntry]: - """Merge ECS field sets with custom field sets.""" + """Recursively merge field dicts; b takes precedence. normalize/multi_fields are concatenated; + reusable.expected is concatenated; nested fields are merged recursively.""" a = copy.deepcopy(a) b = copy.deepcopy(b) for key in b: @@ -293,17 +287,18 @@ def merge_fields(a: Dict[str, FieldEntry], b: Dict[str, FieldEntry]) -> Dict[str def load_yaml_file(file_name): + """Load and parse a YAML file.""" with open(file_name) as f: return yaml.safe_load(f.read()) -# You know, for silent tests def warn(message: str) -> None: + """Print a warning message. Exists as a function to enable mocking in tests.""" print(message) def eval_globs(globs): - """Accepts an array of glob patterns or file names, returns the array of actual files""" + """Expand glob patterns to file paths. Directories ending with '/' become 'dir/*'. Warns on no matches.""" all_files = [] for g in globs: if g.endswith('/'): @@ -317,6 +312,7 @@ def eval_globs(globs): def load_definitions(file_globs): + """Load YAML definition files matching file_globs (used by subset_filter and exclude_filter).""" sets = [] for f in ecs_helpers.glob_yaml_files(file_globs): raw = load_yaml_file(f) diff --git a/scripts/schema/subset_filter.py b/scripts/schema/subset_filter.py index 8df16e4ba2..9406473671 100644 --- a/scripts/schema/subset_filter.py +++ b/scripts/schema/subset_filter.py @@ -15,6 +15,13 @@ # specific language governing permissions and limitations # under the License. +"""Schema Subset Filter Module. + +Filters the ECS schema to only the fieldsets/fields specified in subset YAML files. +Supports fields='*' (all), fields={...} (specific), docs_only=true (docs but not artifacts), +and multiple subset files (merged as union). See USAGE.md for subset file format. +""" + import copy import os from typing import ( @@ -34,15 +41,13 @@ FieldEntry ) -# This script takes all ECS and custom fields already loaded, and lets users -# filter out the ones they don't need. - def filter( fields: Dict[str, FieldEntry], subset_file_globs: List[str], out_dir: str ) -> Tuple[Dict[str, FieldEntry], Dict[str, FieldEntry]]: + """Return (filtered_fields, docs_only_fields). Writes per-subset intermediate files to out_dir.""" subsets: List[Dict[str, Any]] = load_subset_definitions(subset_file_globs) for subset in subsets: subfields: Dict[str, FieldEntry] = extract_matching_fields(fields, subset['fields']) @@ -126,7 +131,7 @@ def remove_docs_only_entries(paths: List[str], fields: Dict[str, FieldEntry]) -> def combine_all_subsets(subsets: Dict[str, Any]) -> Dict[str, Any]: - """Merges N subsets into one. Strips top level 'name' and 'fields' keys as well as non-ECS field options since we can't know how to merge those.""" + """Merge subset definitions into one union. Non-ECS options are stripped before merging.""" merged_subset = {} for subset in subsets: strip_non_ecs_options(subset['fields']) @@ -135,6 +140,7 @@ def combine_all_subsets(subsets: Dict[str, Any]) -> Dict[str, Any]: def load_subset_definitions(file_globs: List[str]) -> List[Dict[str, Any]]: + """Load subset YAML files. Returns [] if file_globs is empty. Raises ValueError if none found.""" if not file_globs: return [] subsets: List[Dict[str, Any]] = loader.load_definitions(file_globs) @@ -178,7 +184,12 @@ def extract_matching_fields( fields: Dict[str, FieldEntry], subset_definitions: Dict[str, Any] ) -> Dict[str, FieldEntry]: - """Removes fields that are not in the subset definition. Returns a copy without modifying the input fields dict.""" + """Return fields filtered to only those in subset_definitions, applied recursively. + + Fields with fields='*' include all nested fields. Field options (enabled, index) are + applied to field_details. Intermediate fields with options are promoted to real fields. + Raises ValueError if 'fields' presence/absence in subset doesn't match schema. + """ retained_fields: Dict[str, FieldEntry] = {x: fields[x].copy() for x in subset_definitions} for key, val in subset_definitions.items(): retained_fields[key]['field_details'] = fields[key]['field_details'].copy() diff --git a/scripts/schema/visitor.py b/scripts/schema/visitor.py index 1e1ca4441c..ba95e4732e 100644 --- a/scripts/schema/visitor.py +++ b/scripts/schema/visitor.py @@ -15,6 +15,14 @@ # specific language governing permissions and limitations # under the License. +"""Field Visitor Module. + +Three depth-first traversal helpers for the deeply nested field structure from loader.py: +- visit_fields(): dispatch to fieldset_func or field_func based on node type +- visit_fields_with_path(): pass accumulated path array to callback +- visit_fields_with_memo(): pass shared accumulator to callback +""" + from typing import ( Callable, Dict, @@ -34,19 +42,7 @@ def visit_fields( fieldset_func: Optional[Callable[[FieldEntry], None]] = None, field_func: Optional[Callable[[FieldDetails], None]] = None ) -> None: - """ - This function navigates the deeply nested tree structure and runs provided - functions on each fieldset or field encountered (both optional). - - The argument 'fields' should be at the named field grouping level: - {'name': {'schema_details': {}, 'field_details': {}, 'fields': {}} - - The 'fieldset_func(details)' provided will be called for each field set, - with the dictionary containing their details ({'schema_details': {}, 'field_details': {}, 'fields': {}). - - The 'field_func(details)' provided will be called for each field, with the dictionary - containing the field's details ({'field_details': {}, 'fields': {}). - """ + """Depth-first traversal calling fieldset_func for nodes with schema_details, field_func for others.""" for (_, details) in fields.items(): if fieldset_func and 'schema_details' in details: fieldset_func(details) @@ -63,13 +59,9 @@ def visit_fields_with_path( func: Callable[[FieldDetails], None], path: Optional[List[str]] = [] ) -> None: - """ - This function navigates the deeply nested tree structure and runs the provided - function on all fields and field sets. + """Depth-first traversal passing accumulated path to func(details, path). - The 'func' provided will be called for each field, - with the dictionary containing their details ({'field_details': {}, 'fields': {}) - as well as the path array leading to the location of the field in question. + Root fieldsets (root=true) don't add their name to the path. """ for (name, details) in fields.items(): if 'field_details' in details: @@ -87,14 +79,7 @@ def visit_fields_with_memo( func: Callable[[FieldEntry, Field], None], memo: Optional[Dict[str, Field]] = None ) -> None: - """ - This function navigates the deeply nested tree structure and runs the provided - function on all fields and field sets. - - The 'func' provided will be called for each field, - with the dictionary containing their details ({'field_details': {}, 'fields': {}) - as well as the 'memo' you pass in. - """ + """Depth-first traversal passing a shared accumulator to func(details, memo).""" for (name, details) in fields.items(): if 'field_details' in details: func(details, memo) diff --git a/scripts/templates/ecs_field_reference.j2 b/scripts/templates/ecs_field_reference.j2 index 3a1d485ab8..4e0d8e366f 100644 --- a/scripts/templates/ecs_field_reference.j2 +++ b/scripts/templates/ecs_field_reference.j2 @@ -16,7 +16,7 @@ ECS defines multiple groups of related fields. They are called "field sets". The All other field sets are defined as objects in {{ es }}, under which all fields are defined. -For a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv). +For a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv). ## Field sets [ecs-fieldsets] diff --git a/scripts/templates/index.j2 b/scripts/templates/index.j2 index 02c154a4f5..c56a8807c0 100644 --- a/scripts/templates/index.j2 +++ b/scripts/templates/index.j2 @@ -41,5 +41,5 @@ ECS is a permissive schema. If your events have additional data that cannot be m ECS improvements are released following [Semantic Versioning](https://semver.org/). Major ECS releases are planned to be aligned with major Elastic Stack releases. -Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the [Contribution Guidelines](https://github.com/elastic/ecs/blob/master/CONTRIBUTING.md). +Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the [Contribution Guidelines](https://github.com/elastic/ecs/blob/main/CONTRIBUTING.md).