Adding initial spec files

README.md

@@ -2,8 +2,63 @@
 
 This repository contains the specifications for Elastic Packages, as served up by the [Elastic Package Registry (EPM)](https://github.com/elastic/package-registry).
 
+| :warning: **WARNING** :warning: |
+| ----- |
+| The specifications in this repository are currently under active development. They are **NOT** ready for general use. |
+
 In the future it may also contain code for validating said specifications.
 
 # Purpose
 
 Please use this repository to discuss any changes to the specification, either my making issues or PRs to the specification.
+
+# Specification Format 
+
+An Elastic Package specification describes:
+1. the folder structure of packages and expected files within these folders; and
+2. the structure of the expected files' contents.
+
+There may be multiple versions of specifications. At the root of this repository is a `versions` folder. In this folder you will find sub-folders for each active major version of the specification, e.g. `versions/1`, `versions/2`, etc. Read more in the _Specification Versioning_ section below.
+
+Within each major version folder, there must be a `spec.yml` file. This file is the entry point for the specification for a package's contents. It describes the the folder structure of packages and expected files within these folders (this is point 1. above). The specification is expressed using a schema similar to [JSON Schema](https://json-schema.org/), but with a couple of differences:
+- The `type` field can be either `folder` or `file`,
+- A new field, `contents` is introduced to (recursively) describe the contents of folders (i.e. when type == folder), and
+- The specification is written as YAML for readability.
+
+Expected package files, e.g. `manifest.yml` themselves have a structure to their contents. This structure is described in specification files using JSON schema (this is point 2. above). These specification files are also written as YAML for readability.
+
+Note that the specification files primarily defined the structure (syntax) of a package's contents. To a limited extent they may also define some semantics, e.g. enumeration values for certain fields. Richer semantics, however, will need to be expressed as validation code.
+
+# Specification Versioning
+
+As mentioned above, package specifications are versioned. Versions follow the [semantic versioning](https://semver.org/) scheme. In the context of package specifications, this means the following.
+
+* Packages must specify the specification version they are using. This is done via the `format_version` property in the package's root `manifest.yml` file. The value of `format_version` must conform to the semantic versioning scheme.
+
+* Specifications are organized under the `versions` folder located at the root of this repository. The `versions` folder will contain a sub-folder for each **major version** of the specification, e.g. `versions/1`, `versions/2`, etc.
+
+* Within each major version folder, there is a `spec.yml` file. It contains a root-level property called `version` which specifies the complete, current version of the specification. The value of `version` conforms to the semantic versioning scheme.
+
+* Note that the latest version — and _only the latest_ version — of the specifications  may include a pre-release suffix, `e.g. 1.4.0-alpha1`. This indicates that this version is still under development and may be changed multiple times. Once the pre-relase suffix is removed, however, the specification at that version becomes immutable. Further changes must follow the process outlined below in _Changing a Specification_.
+
+## Changing a Specification
+
+* Consider the **latest** version of the specification. Say it is `x.y.z`. It will be located under the `versions/x` folder, where `x` is the highest major version of the specification.
+* Now consider a proposal to change the specification in some way. The version number of the changed specification must be determined as follows:
+  * If the proposed change makes the specification stricter than it is at `x.y.z`, the new version number will be `X.0.0`, where `X = x + 1`. That is, we bump up the major version. 
+     * Add a new folder named `versions/X`, where `X` is the new major version number. 
+     * The changed specification — in its entirety — must be added to the new version folder. 
+     * Set the root-level `version` property in the specification's root `spec.yml` file to `X.0.0`.
+     * Start a new `CHANGELOG.yml` file at the root of the `versions/X` folder, add a section for `X.0.0` and make an entry under it explaining your change. If there are multiple changes, please add multiple entries under the new section.
+  * If the proposed change makes the specification looser than it is at `x.y.z`, the new version number will be `x.Y.0`, where `Y = y + 1`. That is, we bump up the minor version. Note that adding new, but optional, constraints to a specification is a change that makes a specification looser.
+     * Apply the proposed changes to the existing specification under the `versions/x` folder, where `x` is the major version number of the specification being changed. 
+     * Set the root-level `version` property in the specification's root `spec.yml` file to `x.Y.0`.
+     * Modify the `CHANGELOG.yml` file at the root of the `versions/x` folder, add an section for `x.Y.0` and make an entry under it explaining your change. If there are multiple changes, please add multiple entries under the new section.
+* If the proposed change does not change the strictness of the specification at `x.y.z`, the new version number will be `x.y.Z`, where `Z = z + 1`. That is, we bump the patch version.
+     * Apply the proposed changes to the existing specification under the `versions/x` folder, where `x` is the major version number of the specification being changed. 
+     * Set the root-level `version` property in the specification's root `spec.yml` file to `x.y.Z`.
+     * Modify the `CHANGELOG.yml` file at the root of the `versions/x` folder, add an section for `x.y.Z` and make an entry under it explaining your change. If there are multiple changes, please add multiple entries under the new section.
+
+## Version Compatibility between Packages and Specifications
+
+A package specifying its `format_version` as `x.y.z` must be valid against specifications in the semantic version range `[x.y.z, X.0.0)`, where `X = x + 1`.

versions/1/changelog.spec.yml

@@ -0,0 +1,48 @@
+##
+## Describes the specification for the package's CHANGELOG file
+##
+spec:
+  # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability
+  type: array
+  items:
+    type: object
+    additionalProperties: false
+    properties:
+      version:
+        description: Package version.
+        type: string
+        pattern: "^[0-9]+\.[0-9]+\.[0-9]+$"
+        examples:
+        - "1.0.4"
+      changes:
+        description: List of changes in package version.
+        type: array
+        items:
+          type: object
+          additionalProperties: false
+          properties:
+            description:
+              description: Description of change.
+              type: string
+              examples:
+              - "Fix broken template"
+            type:
+              description: Type of change.
+              type: string
+              enum:
+              - "breaking-change"
+              - "bugfix"
+              - "enhancement"
+              - "known-issue"
+            link:
+              description: Link to issue or PR describing change in detail.
+              type: string
+              examples:
+              - "https://github.com/elastic/beats/issues/13507"
+          required:
+          - description
+          - type
+          - link
+    required:
+    - version
+    - changes

versions/1/changelog.yml

@@ -0,0 +1,8 @@
+##
+## This file documents changes in the package specification. It is NOT a package specification file.
+##
+- version: 1.0.0-alpha1
+  changes:
+  - description: Initial version of the Elastic Package specifiction.
+    type: enhancement
+    link: https://github.com/elastic/package-spec/pulls/1

versions/1/dataset/fields/fields.spec.yml

@@ -0,0 +1,53 @@
+##
+## Describes the specification for a dataset's various field definition files
+##
+spec:
+  # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability
+  type: object
+  additionalProperties: true
+  properties:
+    name:
+      description: Name of field
+      type: string
+    title:
+      description: Title of field
+      type: string
+    type: 
+      description: Datatype of field
+      type: string
+      enum:
+      - constant_keyword
+      - text
+      - keyword
+      - long
+      - integer
+      - short
+      - byte
+      - double
+      - float
+      - half_float
+      - scaled_float
+      - date
+      - date_nanos
+      - boolean
+      - binary
+      - integer_range
+      - float_range
+      - long_range
+      - double_range
+      - date_range
+      - ip_range
+      - group
+    description:
+      description: Short description of field
+      type: string
+    default_field:
+      description: "Is this field a default? TODO: better description"
+      type: boolean
+      default: true
+    fields:
+      description: Sub-fields, when type is group
+      $ref: "#"     # JSON-schema syntax for pointing to the root of the schema
+  required:
+  - name
+  - type

versions/1/dataset/fields/spec.yml

@@ -0,0 +1,27 @@
+spec:
+- name: base fields
+  description: Base fields definitions
+  type: file
+  name: "base-fields.yml"
+  required: true
+  contentMediaType: "application/x-yaml"
+  $ref: "./fields.spec.yml"
+- name: dataset fields
+  description: Dataset-specific fields definitions
+  type: file
+  name: "fields.yml"
+  required: true
+  contentMediaType: "application/x-yaml"
+  $ref: "./fields.spec.yml"
+- name: package fields
+  description: Package-specific fields definitions
+  type: file
+  name: "package-fields.yml"
+  contentMediaType: "application/x-yaml"
+  $ref: "./fields.spec.yml"
+- name: ECS fields
+  description: ECS fields definitions
+  type: file
+  name: "ecs.yml"
+  contentMediaType: "application/x-yaml"
+  $ref: "./fields.spec.yml"

versions/1/dataset/manifest.spec.yml

@@ -0,0 +1,142 @@
+##
+## Describes the specification for a dataset's manifest.yml file
+##
+spec:
+  # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability
+  type: object
+  additionalProperties: false
+  properties:
+    # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L39
+    name:
+      description: Name of dataset
+      type: string
+    # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L41
+    title:
+      description: Title of dataset.
+      type: string
+      examples:
+      - AWS billing metrics
+    # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L42
+    release:
+      description: Stability of dataset.
+      type: string
+      enum:
+      - experimental
+      - beta
+      examples:
+      - beta
+    # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L38
+    type:
+      description: Type of dataset
+      type: string
+      enum:
+      - metrics
+      - logs
+      examples:
+      - metrics
+    # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L46
+    streams:
+      description: Streams offered by dataset.
+      type: array
+      items:
+        type: object
+        additionalProperties: false
+        properties:
+          # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L66
+          input:
+            type: string
+            examples:
+            - aws/metrics
+            - s3
+            - file
+          # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L71
+          title:
+            type: string
+            examples:
+            - AWS Billing metrics
+          # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L72
+          description:
+            type: string
+            examples:
+            - Collect AWS billing metrics
+          # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L70
+          template_path:
+            description: "Path to Elasticsearch index template for stream."
+            type: string
+          # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L67
+          vars:
+            description: Input variables.
+            type: array
+            items:
+              type: object
+              additionalProperties: false
+              properties:
+                https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L77
+                name:
+                  description: Variable name.
+                  type: string
+                  examples:
+                  - hosts
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L78
+                type:
+                  description: Data type of variable.
+                  type: string
+                  enum:
+                  - text
+                  - yaml
+                  examples:
+                  - text
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L79
+                title:
+                  description: Title of variable.
+                  type: string
+                  examples:
+                  - Hosts
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L80
+                description:
+                  description: Short description of variable.
+                  type: string
+                  examples:
+                  - "Hosts of integration service to connect to"
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L81
+                multi:
+                  description: Can variable contain multiple values?
+                  type: boolean
+                  default: false
+                  examples:
+                  - true
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L82
+                required:
+                  description: Is variable required?
+                  type: boolean
+                  default: false
+                  examples:
+                  - true
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L83
+                show_user:
+                  description: Should this variable be shown to the user by default?
+                  type: boolean
+                  default: true
+                  examples:
+                  - false
+                # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L84
+                default:
+                  description: Default value(s) for variable
+                  type: array
+                  items:
+                    type: string
+                    examples:
+                    - "http://127.0.0.1"
+              required:
+              - name
+              - type                  
+          # https://github.com/elastic/package-registry/blob/v0.8.0/util/dataset.go#L73
+          enabled:
+            description: Is stream enabled?
+            type: boolean
+        required:
+        - title
+        - description
+        - input
+  required:
+  - title

versions/1/dataset/spec.yml

@@ -0,0 +1,45 @@
+spec:
+- description: Folder containing a single dataset definition
+  type: folder
+  pattern: "[a-z0-9][a-z0-9_]+[a-z0-9]"
+  minOccurences: 1
+  contents:
+  - description: A dataset's manifest file
+    type: file
+    contentMediaType: "application/x-yaml"
+    name: "manifest.yml"
+    required: true
+    $ref: "./manifest.spec.yml"
+  - description: Folder containing field definitions
+    type: folder
+    name: fields
+    required: true
+    $ref: "./fields/spec.yml"
+  - description: Folder containing agent-related definitions
+    type: folder
+    name: agent
+    required: true
+    contents:
+    - description: Folder containing agent stream definitions
+      type: folder
+      name: stream
+      required: true
+      contents:
+      - description: Agent stream definition
+        type: file
+        name: "stream.yml.hbs"
+        contentMediaType: "application/x-hbs"
+        required: true
+  - description: Folder containing Elasticsearch assets
+    type: folder
+    name: elasticsearch
+    contents:
+    - description: Folder containing Elasticsearch Ingest Node pipeline definitions
+      type: folder
+      name: ingest_pipeline
+      contents:
+      - description: Default ingest pipeline definition
+        type: file
+        name: "default.yml"
+        contentMediaType: "application/x-yaml"
+        required: true