diff --git a/README.md b/README.md index 4e73d59f2..5eb94a08d 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,63 @@ This repository contains the specifications for Elastic Packages, as served up by the [Elastic Package Registry (EPM)](https://github.com/elastic/package-registry). +| :warning: **WARNING** :warning: | +| ----- | +| The specifications in this repository are currently under active development. They are **NOT** ready for general use. | + In the future it may also contain code for validating said specifications. # Purpose Please use this repository to discuss any changes to the specification, either my making issues or PRs to the specification. + +# Specification Format + +An Elastic Package specification describes: +1. the folder structure of packages and expected files within these folders; and +2. the structure of the expected files' contents. + +There may be multiple versions of specifications. At the root of this repository is a `versions` folder. In this folder you will find sub-folders for each active major version of the specification, e.g. `versions/1`, `versions/2`, etc. Read more in the _Specification Versioning_ section below. + +Within each major version folder, there must be a `spec.yml` file. This file is the entry point for the specification for a package's contents. It describes the the folder structure of packages and expected files within these folders (this is point 1. above). The specification is expressed using a schema similar to [JSON Schema](https://json-schema.org/), but with a couple of differences: +- The `type` field can be either `folder` or `file`, +- A new field, `contents` is introduced to (recursively) describe the contents of folders (i.e. when type == folder), and +- The specification is written as YAML for readability. + +Expected package files, e.g. `manifest.yml` themselves have a structure to their contents. This structure is described in specification files using JSON schema (this is point 2. above). These specification files are also written as YAML for readability. + +Note that the specification files primarily defined the structure (syntax) of a package's contents. To a limited extent they may also define some semantics, e.g. enumeration values for certain fields. Richer semantics, however, will need to be expressed as validation code. + +# Specification Versioning + +As mentioned above, package specifications are versioned. Versions follow the [semantic versioning](https://semver.org/) scheme. In the context of package specifications, this means the following. + +* Packages must specify the specification version they are using. This is done via the `format_version` property in the package's root `manifest.yml` file. The value of `format_version` must conform to the semantic versioning scheme. + +* Specifications are organized under the `versions` folder located at the root of this repository. The `versions` folder will contain a sub-folder for each **major version** of the specification, e.g. `versions/1`, `versions/2`, etc. + +* Within each major version folder, there is a `spec.yml` file. It contains a root-level property called `version` which specifies the complete, current version of the specification. The value of `version` conforms to the semantic versioning scheme. + +* Note that the latest version — and _only the latest_ version — of the specifications may include a pre-release suffix, `e.g. 1.4.0-alpha1`. This indicates that this version is still under development and may be changed multiple times. Once the pre-relase suffix is removed, however, the specification at that version becomes immutable. Further changes must follow the process outlined below in _Changing a Specification_. + +## Changing a Specification + +* Consider the **latest** version of the specification. Say it is `x.y.z`. It will be located under the `versions/x` folder, where `x` is the highest major version of the specification. +* Now consider a proposal to change the specification in some way. The version number of the changed specification must be determined as follows: + * If the proposed change makes the specification stricter than it is at `x.y.z`, the new version number will be `X.0.0`, where `X = x + 1`. That is, we bump up the major version. + * Add a new folder named `versions/X`, where `X` is the new major version number. + * The changed specification — in its entirety — must be added to the new version folder. + * Set the root-level `version` property in the specification's root `spec.yml` file to `X.0.0`. + * Start a new `CHANGELOG.yml` file at the root of the `versions/X` folder, add a section for `X.0.0` and make an entry under it explaining your change. If there are multiple changes, please add multiple entries under the new section. + * If the proposed change makes the specification looser than it is at `x.y.z`, the new version number will be `x.Y.0`, where `Y = y + 1`. That is, we bump up the minor version. Note that adding new, but optional, constraints to a specification is a change that makes a specification looser. + * Apply the proposed changes to the existing specification under the `versions/x` folder, where `x` is the major version number of the specification being changed. + * Set the root-level `version` property in the specification's root `spec.yml` file to `x.Y.0`. + * Modify the `CHANGELOG.yml` file at the root of the `versions/x` folder, add an section for `x.Y.0` and make an entry under it explaining your change. If there are multiple changes, please add multiple entries under the new section. +* If the proposed change does not change the strictness of the specification at `x.y.z`, the new version number will be `x.y.Z`, where `Z = z + 1`. That is, we bump the patch version. + * Apply the proposed changes to the existing specification under the `versions/x` folder, where `x` is the major version number of the specification being changed. + * Set the root-level `version` property in the specification's root `spec.yml` file to `x.y.Z`. + * Modify the `CHANGELOG.yml` file at the root of the `versions/x` folder, add an section for `x.y.Z` and make an entry under it explaining your change. If there are multiple changes, please add multiple entries under the new section. + +## Version Compatibility between Packages and Specifications + +A package specifying its `format_version` as `x.y.z` must be valid against specifications in the semantic version range `[x.y.z, X.0.0)`, where `X = x + 1`. diff --git a/versions/1/changelog.spec.yml b/versions/1/changelog.spec.yml new file mode 100644 index 000000000..5e2035b9e --- /dev/null +++ b/versions/1/changelog.spec.yml @@ -0,0 +1,48 @@ +## +## Describes the specification for the package's CHANGELOG file +## +spec: + # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability + type: array + items: + type: object + additionalProperties: false + properties: + version: + description: Package version. + type: string + pattern: '^[0-9]+\.[0-9]+\.[0-9]+$' + examples: + - "1.0.4" + changes: + description: List of changes in package version. + type: array + items: + type: object + additionalProperties: false + properties: + description: + description: Description of change. + type: string + examples: + - "Fix broken template" + type: + description: Type of change. + type: string + enum: + - "breaking-change" + - "bugfix" + - "enhancement" + - "known-issue" + link: + description: Link to issue or PR describing change in detail. + type: string + examples: + - "https://github.com/elastic/beats/issues/13507" + required: + - description + - type + - link + required: + - version + - changes \ No newline at end of file diff --git a/versions/1/changelog.yml b/versions/1/changelog.yml new file mode 100644 index 000000000..ee10f053a --- /dev/null +++ b/versions/1/changelog.yml @@ -0,0 +1,8 @@ +## +## This file documents changes in the package specification. It is NOT a package specification file. +## +- version: 1.0.0-alpha1 + changes: + - description: Initial version of the Elastic Package specifiction. + type: enhancement + link: https://github.com/elastic/package-spec/pulls/1 diff --git a/versions/1/dataset/fields/fields.spec.yml b/versions/1/dataset/fields/fields.spec.yml new file mode 100644 index 000000000..91aa649b2 --- /dev/null +++ b/versions/1/dataset/fields/fields.spec.yml @@ -0,0 +1,49 @@ +## +## Describes the specification for a dataset's various field definition files +## +spec: + # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability + type: object + additionalProperties: true + properties: + name: + description: Name of field + type: string + title: + description: Title of field + type: string + type: + description: Datatype of field + type: string + enum: + - constant_keyword + - text + - keyword + - long + - integer + - short + - byte + - double + - float + - half_float + - scaled_float + - date + - date_nanos + - boolean + - binary + - integer_range + - float_range + - long_range + - double_range + - date_range + - ip_range + - group + description: + description: Short description of field + type: string + fields: + description: Sub-fields, when type is group + $ref: "#" # JSON-schema syntax for pointing to the root of the schema + required: + - name + - type diff --git a/versions/1/dataset/fields/spec.yml b/versions/1/dataset/fields/spec.yml new file mode 100644 index 000000000..580180efa --- /dev/null +++ b/versions/1/dataset/fields/spec.yml @@ -0,0 +1,7 @@ +spec: +- description: Fields definition + type: file + pattern: '[a-z0-9][a-z0-9_]+[a-z0-9]\.yml' + required: true + contentMediaType: "application/x-yaml" + $ref: "./fields.spec.yml" diff --git a/versions/1/dataset/manifest.spec.yml b/versions/1/dataset/manifest.spec.yml new file mode 100644 index 000000000..26755f871 --- /dev/null +++ b/versions/1/dataset/manifest.spec.yml @@ -0,0 +1,149 @@ +## +## Describes the specification for a dataset's manifest.yml file +## +spec: + # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability + type: object + additionalProperties: false + properties: + name: + description: Name of dataset + type: string + title: + description: Title of dataset. + type: string + examples: + - AWS billing metrics + release: + description: Stability of dataset. + type: string + enum: + - experimental + - beta + examples: + - beta + type: + description: Type of dataset + type: string + enum: + - metrics + - logs + examples: + - metrics + streams: + description: Streams offered by dataset. + type: array + items: + type: object + additionalProperties: false + properties: + input: + type: string + examples: + - aws/metrics + - s3 + - file + title: + type: string + examples: + - AWS Billing metrics + description: + type: string + examples: + - Collect AWS billing metrics + template_path: + description: "Path to Elasticsearch index template for stream." + type: string + vars: + description: Input variables. + type: array + items: + type: object + additionalProperties: false + properties: + name: + description: Variable name. + type: string + examples: + - hosts + type: + description: Data type of variable. + type: string + enum: + - text + - yaml + examples: + - text + title: + description: Title of variable. + type: string + examples: + - Hosts + description: + description: Short description of variable. + type: string + examples: + - "Hosts of integration service to connect to" + multi: + description: Can variable contain multiple values? + type: boolean + default: false + examples: + - true + required: + description: Is variable required? + type: boolean + default: false + examples: + - true + show_user: + description: Should this variable be shown to the user by default? + type: boolean + default: true + examples: + - false + default: + description: Default value(s) for variable + type: array + items: + type: string + examples: + - "http://127.0.0.1" + required: + - name + - type + enabled: + description: Is stream enabled? + type: boolean + required: + - title + - description + - input + elasticsearch: + description: Elasticsearch asset definitions + type: object + additionalProperties: false + properties: + index_template: + description: Index template definition + type: object + additionalProperties: false + properties: + settings: + description: Settings section of index template + type: object + mappings: + description: Mappings section of index template + type: object + ingest_pipeline: + description: Elasticsearch ingest pipeline settings + type: object + additionalProperties: false + properties: + name: + description: Ingest pipeline name + type: string + required: + - name + required: + - title diff --git a/versions/1/dataset/spec.yml b/versions/1/dataset/spec.yml new file mode 100644 index 000000000..b4131f567 --- /dev/null +++ b/versions/1/dataset/spec.yml @@ -0,0 +1,45 @@ +spec: +- description: Folder containing a single dataset definition + type: folder + pattern: '[a-z0-9][a-z0-9_]+[a-z0-9]' + minOccurences: 1 + contents: + - description: A dataset's manifest file + type: file + contentMediaType: "application/x-yaml" + name: "manifest.yml" + required: true + $ref: "./manifest.spec.yml" + - description: Folder containing field definitions + type: folder + name: fields + required: true + $ref: "./fields/spec.yml" + - description: Folder containing agent-related definitions + type: folder + name: agent + required: true + contents: + - description: Folder containing agent stream definitions + type: folder + name: stream + required: true + contents: + - description: Agent stream definition + type: file + name: "stream.yml.hbs" + contentMediaType: "application/x-hbs" + required: true + - description: Folder containing Elasticsearch assets + type: folder + name: elasticsearch + contents: + - description: Folder containing Elasticsearch Ingest Node pipeline definitions + type: folder + name: ingest_pipeline + contents: + - description: Default ingest pipeline definition + type: file + name: "default.yml" + contentMediaType: "application/x-yaml" + required: true diff --git a/versions/1/kibana/spec.yml b/versions/1/kibana/spec.yml new file mode 100644 index 000000000..525dcab60 --- /dev/null +++ b/versions/1/kibana/spec.yml @@ -0,0 +1,46 @@ + spec: + - description: Folder containing Kibana dashboard assets + type: folder + name: dashboard + required: false + contents: + - description: A dashboard asset file + type: file + contentMediaType: "application/json" + pattern: '.+\.json' + - description: Folder containing Kibana visualization assets + type: folder + name: visualization + required: false + contents: + - description: A visualization asset file + type: file + contentMediaType: "application/json" + pattern: '.+\.json' + - description: Folder containing Kibana saved search assets + type: folder + name: search + required: false + contents: + - description: A saved search asset file + type: file + contentMediaType: "application/json" + pattern: '.+\.json' + - description: Folder containing Kibana map assets + type: folder + name: map + required: false + contents: + - description: A map asset file + type: file + contentMediaType: "application/json" + pattern: '.+\.json' + - description: Folder containing Kibana index pattern assets + type: folder + name: "index_pattern" + required: false + contents: + - description: An index pattern asset file + type: file + contentMediaType: "application/json" + pattern: '.+\.json' diff --git a/versions/1/manifest.spec.yml b/versions/1/manifest.spec.yml new file mode 100644 index 000000000..f0e8cddee --- /dev/null +++ b/versions/1/manifest.spec.yml @@ -0,0 +1,236 @@ +## +## Describes the specification for the package's main manifest.yml file +## +spec: + # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability + type: object + additionalProperties: false + properties: + format_version: + description: The version of the package specification format used by this package. + type: string + pattern: '^([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+[0-9A-Za-z-]+)?$' + examples: + - "1.0.0" + name: + description: The name of the package. + type: string + pattern: '[a-z0-9_]+' + examples: + - apache + title: + description: The title of the package. + type: string + examples: + - Apache + description: + description: A longer description of the package. + type: string + examples: + - Apache Integration + version: + description: The version of the package. + type: string + pattern: '^([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+[0-9A-Za-z-]+)?$' + examples: + - "0.1.0" + license: + description: The license under which the package is being released. + type: string + enum: + - basic + default: basic + examples: + - basic + type: + description: The type of package. + type: string + enum: + - integration + examples: + - integration + release: + description: The stability of the package. + type: string + enum: + - experimental + - beta + - ga + examples: + - experimental + categories: + description: Categories to which this package belongs. + type: array + items: + type: string + enum: + - aws + - azure + - cloud + - config_management + - containers + - crm + - custom + - datastore + - elastic_stack + - google_cloud + - kubernetes + - languages + - message_queue + - monitoring + - network + - notification + - os_system + - productivity + - security + - support + - ticketing + - version_control + - web + examples: + - web + conditions: + description: Constraints under which this package can be installed. + type: object + additionalProperties: false + properties: + kibana: + description: Kibana versions compatible with this package. + type: object + additionalProperties: false + properties: + version: + type: string + examples: + - ">=7.9.0" + icons: + description: List of icons for by this package. + type: array + items: + type: object + additionalProperties: false + properties: + src: + description: Relative path to the icon's image file. + type: string + examples: + - "/img/logo_apache.svg" + title: + description: Title of icon. + type: string + examples: + - "Apache Logo" + size: + description: Size of the icon. + type: string + examples: + - "32x32" + type: + description: MIME type of the icon image file. + type: string + examples: + - "image/svg+xml" + required: + - src + screenshots: + description: List of screenshots of Kibana assets created by this package. + type: array + items: + type: object + additionalProperties: false + properties: + src: + description: Relative path to the scrennshot's image file. + type: string + examples: + - "/img/apache_httpd_server_status.png" + title: + description: Title of screenshot. + type: string + examples: + - "Apache HTTPD Server Status" + size: + description: Size of the screenshot. + type: string + examples: + - "1215x1199" + type: + description: MIME type of the screenshot image file. + type: string + examples: + - "image/png" + required: + - src + - title + config_templates: + description: List of configuration templates offered by this package. + type: array + items: + type: object + additionalProperties: false + properties: + name: + description: Name of configuration template. + type: string + examples: + - apache + title: + description: Title of configuration template. + type: string + examples: + - Apache logs and metrics + description: + description: Longer description of configuration template. + type: string + examples: + - Collect logs and metrics from Apache instances + inputs: + description: List of inputs supported by configuration template. + type: array + items: + type: object + additionalProperties: false + properties: + type: + description: Type of input. + type: string + title: + description: Title of input. + type: string + examples: + - Collect logs from Apache instances + description: + description: Longer description of input. + type: string + examples: + - Collecting Apache access and error logs + vars: + $ref: "./dataset/manifest.spec.yml#streams/items/properties/vars" + required: + - type + - title + - description + multiple: + type: boolean + required: + - name + - title + - description + datasets: + type: array + items: + $ref: "./dataset/manifest.spec.yml" + owner: + type: object + additionalProperties: false + properties: + github: + description: GitHub repository name of package maintainer. + type: string + required: + - format_version + - name + - title + - description + - version + - release \ No newline at end of file diff --git a/versions/1/spec.yml b/versions/1/spec.yml new file mode 100644 index 000000000..63a29acf4 --- /dev/null +++ b/versions/1/spec.yml @@ -0,0 +1,33 @@ +## +## Entrypoint of package specification. +## +## Describes the folders and files that make up a package. +## +version: 1.0.0-alpha1 +spec: +- description: The main package manifest file + type: file + contentMediaType: "application/x-yaml" + name: "manifest.yml" + required: true + $ref: "./manifest.spec.yml" +- description: The package's CHANGELOG file + type: file + contentMediaType: "application/x-yaml" + name: "changelog.yml" + required: true + $ref: "./changelog.spec.yml" +- description: Folder containing dataset definitions + type: folder + name: dataset + required: true + $ref: "./dataset/spec.yml" +- description: Folder containing documentation for the package + type: folder + name: docs + required: true +- description: Folder containing Kibana assets used by the package + type: folder + name: kibana + required: false + $ref: "./kibana/spec.yml" \ No newline at end of file