From 6fe5f340d64f5eeecd56f4cdbdd0c1d04ab11825 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Wed, 14 Jul 2021 10:11:38 +0200 Subject: [PATCH 1/8] Clean up docs. --- semantic-conventions/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/semantic-conventions/README.md b/semantic-conventions/README.md index fee6067d..28f10f3e 100644 --- a/semantic-conventions/README.md +++ b/semantic-conventions/README.md @@ -26,7 +26,7 @@ docker run --rm otel/semconvgen --yaml-root {yaml_folder} markdown --markdown-ro Where `{yaml_folder}` is the absolute path to the directory containing the yaml files and `{markdown_folder}` the absolute path to the directory containing the markdown definitions -(`specification` for [opentelemetry-specification](https://github.com/open-telemetry/opentelemetry-specification/tree/master/)). +(`specification` for [opentelemetry-specification](https://github.com/open-telemetry/opentelemetry-specification/tree/main/)). The tool will automatically replace the tables with the up to date definition of the semantic conventions. To do so, the tool looks for special tags in the markdown. @@ -68,8 +68,8 @@ semantic conventions that have the tag `network`. The image supports [Jinja](https://jinja.palletsprojects.com/en/2.11.x/) templates to generate code from the models. -For example, the following template is used by the [opentelemetry-java-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/master/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/typedspan) -to generate Java classes. [Template Link.](https://gist.github.com/thisthat/7e34742f4a7f1b5df57118f859a19c3b) +For example, opentelemetry-java [generates typed constants for semantic conventions](https://github.com/open-telemetry/opentelemetry-java/blob/main/semconv/src/main/java/io/opentelemetry/semconv/trace/attributes/SemanticAttributes.java) +here using [this template](https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/templates/SemanticAttributes.java.j2). The image can generate code with the following command: From c3b71767bf4f15edbb65eb890da29689090ac44f Mon Sep 17 00:00:00 2001 From: Giovanni Liva Date: Wed, 14 Jul 2021 10:12:10 +0200 Subject: [PATCH 2/8] Add docs about semconv YAML. Co-authored-by: Armin Ruech Co-authored-by: Yuri Shkuro --- semantic-conventions/syntax.md | 235 +++++++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 semantic-conventions/syntax.md diff --git a/semantic-conventions/syntax.md b/semantic-conventions/syntax.md new file mode 100644 index 00000000..49bad47e --- /dev/null +++ b/semantic-conventions/syntax.md @@ -0,0 +1,235 @@ +# Semantic Convention YAML Language + +First, the syntax with a pseudo [EBNF](https://en.wikipedia.org/wiki/Extended_Backus-Naur_form) grammar is presented. +Then, the semantic of each field is described. + +## Syntax + +All attributes are lower case. + +```bnf +groups ::= semconv + | semconv groups + +semconv ::= id brief [note] [prefix] [extends] [span_kind] attributes [constraints] + +id ::= string +brief ::= string +note ::= string + +prefix ::= string + +# extends MUST point to an existing semconv id +extends ::= string + +span_kind ::= "client" + | "server" + | "producer" + | "consumer" + | "internal" + +attributes ::= (id type brief examples | ref [brief] [examples]) [required] [note] + +# ref MUST point to an existing attribute id +ref ::= id + +type ::= "string" + | "int" + | "double" + | "boolean" + | "string[]" + | "int[]" + | "double[]" + | "boolean[]" + | enum + +enum ::= [allow_custom_values] members + +allow_custom_values := boolean + +members ::= member {member} + +member ::= id value [brief] [note] + +required ::= "always" + | "conditional" + +examples ::= {} + +constraints ::= constraint {constraint} + +constraint ::= any_of + | include + +any_of ::= id {id} + +include ::= id + +``` + +## Semantics + +### Groups + +Groups contain the list of semantic conventions and it is the root node of each yaml file. + +### Semantic Convention + +The field `semconv` represents a semantic convention and it is made by: + +- `id`, string that uniquely identifies the semantic convention. +- `brief`, string, a brief description of the semantic convention. +- `note`, optional string, a more elaborate description of the semantic convention. + It defaults to an empty string. +- `prefix`, optional string, prefix for the attributes for this semantic convention. + It defaults to an empty string. +- `extends`, optional string, reference another semantic convention `id`. + It inherits the prefix, constraints, and all attributes defined in the specified semantic convention. +- `span_kind`, optional enum, specifies the kind of the span. +- `attributes`, list of attributes that belong to the semantic convention. +- `constraints`, optional list, additional constraints (See later). It defaults to an empty list. + +### Attributes + +An attribute is defined by: + +- `id`, string that uniquely identifies the attribute. +- `type`, either a string literal denoting the type or an enum definition (See later). + The accepted string literals are: + + * "string": String attributes. + * "int": Integer attributes. + * "double": Double attributes. + * "boolean": Boolean attributes. + * "string[]": Array of strings attributes. + * "int[]": Array of integer attributes. + * "double[]": Array of double attributes. + * "boolean[]": Array of booleans attributes. + + See the [specification of Attributes](../specification/common/common.md#attributes) for the definition of the value types. +- `ref`, optional string, reference an existing attribute, see later. +- `required`, optional, specifies if the attribute is mandatory. + Can be "always", or "conditional". When omitted, the attribute is not required. + When set to "conditional",the string provided as `` MUST specify + the conditions under which the attribute is required. +- `brief`, string, brief description of the attribute. +- `note`, optional string, additional notes to the attribute. It defaults to an empty string. +- `examples`, sequence/dictionary of example values for the attribute. + They are optional for boolean and enum attributes. + Example values must be of the same type of the attribute. + If only a single example is provided, it can directly be reported without encapsulating it into a sequence/dictionary. + +Examples for setting the `examples` field: + +A single example value for a string attribute. All the following three representations are equivalent: + +```yaml +examples: 'this is a single string' +``` + +or + +```yaml +examples: ['this is a single string'] +``` + +or + +```yaml +examples: + - 'this is a single string' +``` + +Attention, the following will throw a type mismatch error because a string type as example value is expected and not an array of string: + +```yaml +examples: + - ['this is an error'] + +examples: [['this is an error']] +``` + +Multiple example values for a string attribute: + +```yaml +examples: ['this is a single string', 'this is another one'] +``` + +or + +```yaml +examples: + - 'this is a single string' + - 'this is another one' +``` + +A single example value for an array of strings attribute: + +```yaml +examples: ['first element of first array', 'second element of first array'] +``` + +or + +```yaml +examples: + - ['first element of first array', 'second element of first array'] +``` + +Attention, the following will throw a type mismatch error because an array of strings as type for the example values is expected and not a string: + +```yaml +examples: 'this is an error' +``` + +Multiple example values for an array of string attribute: + +```yaml +examples: [ ['first element of first array', 'second element of first array'], ['first element of second array', 'second element of second array'] ] +``` + +or + +```yaml +examples: + - ['first element of first array', 'second element of first array'] + - ['first element of second array', 'second element of second array'] +``` + +### Ref + +`ref` MUST have an id of an existing attribute. When it is set, `id` and `type` MUST NOT be present. +`ref` is useful for specifying that an existing attribute of another semantic convention is part of +the current semantic convention and inherit its `brief`, `note`, and `example` values. However, if these +fields are present in the current attribute definition, they override the inherited values. + +### Type + +An attribute type can either be a string, int, double, boolean, array of strings, array of int, array of double, +array of booleans, or an enumeration. If it is an enumeration, additional fields are required: + +- `allow_custom_values`, optional boolean, set to false to not accept values + other than the specified members. It defaults to `true`. +- `members`, list of enum entries. + +An enum entry has the following fields: + +- `id`, string that uniquely identifies the enum entry. +- `value`, string, int, or boolean; value of the enum entry. +- `brief`, optional string, brief description of the enum entry value. It defaults to the value of `id`. +- `note`, optional string, longer description. It defaults to an empty string. + +### Constraints + +Allow to define additional requirements on the semantic convention. +Currently, it supports `any_of` and `include`. + +#### Any Of + +`any_of` accepts a list of sequences. Each sequence contains a list of attribute ids that are required. +`any_of` enforces that all attributes of at least one of the sequences are set. + +#### Include + +`include` accepts a semantic conventions `id`. It includes as part of this semantic convention all constraints +and required attributes that are not already defined in the current semantic convention. From ac5a935e455bb358f77acc1dabdab54e16c6842a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Wed, 14 Jul 2021 13:18:20 +0200 Subject: [PATCH 3/8] Add JSON schema for semantic convention YAML files. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Very useful for example for auto-completion in VS Code. Co-authored-by: Giovanni Liva Co-authored-by: Christian Neumüller Co-authored-by: Gernot Reisinger <17945993+Hollerberg@users.noreply.github.com> --- semantic-conventions/semconv.schema.json | 302 +++++++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 semantic-conventions/semconv.schema.json diff --git a/semantic-conventions/semconv.schema.json b/semantic-conventions/semconv.schema.json new file mode 100644 index 00000000..fa3d78db --- /dev/null +++ b/semantic-conventions/semconv.schema.json @@ -0,0 +1,302 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "description": "YAML schema for semantic convention generator, use for example with VS Code.", + "additionalProperties": false, + "properties": { + "groups": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "brief" + ], + "anyOf": [ + { + "require": [ + "attributs" + ] + }, + { + "require": [ + "extends" + ] + } + ], + "properties": { + "id": { + "type": "string", + "description": "unique string" + }, + "type": { + "type": "string", + "enum": [ + "span", + "resource", + "metric" + ] + }, + "brief": { + "type": "string", + "description": "a brief description of the semantic convention" + }, + "note": { + "type": "string", + "description": "a more elaborate description of the semantic convention. It defaults to an empty string" + }, + "prefix": { + "type": "string", + "description": "prefix of the attribute for this semconv. It defaults to an empty string." + }, + "extends": { + "type": "string", + "description": "reference another semantic convention ID. It inherits all attributes from the specified semconv." + }, + "span_kind": { + "type": "string", + "enum": [ + "client", + "server", + "producer", + "consumer", + "internal" + ], + "description": "specifies the kind of the span. Leaf semconv nodes (in the hierarchy tree) that do not have this field set will generate a warning." + }, + "attributes": { + "type": "array", + "items": { + "$ref": "#/definitions/Attribute" + }, + "description": "list of attributes that belong to the semconv" + }, + "constraints": { + "type": "array", + "items": { + "anyOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "any_of" + ], + "properties": { + "any_of": { + "type": "array", + "description": " accepts a list of sequences. Each sequence contains a list of attribute ids that are required. any_of enforces that all attributes of at least one of the sequences are set.", + "items": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "include" + ], + "properties": { + "include": { + "type": "string", + "description": "accepts a semantic conventions id. It includes as part of this semantic convention all constraints and required attributes that are not already defined in the current semantic convention." + } + } + } + ] + } + } + } + } + } + }, + "definitions": { + "AttributeEnumType": { + "type": "object", + "additionalProperties": false, + "properties": { + "allow_custom_values": { + "type": "boolean" + }, + "members": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "value" + ], + "properties": { + "id": { + "type": "string", + "description": "string unique" + }, + "value": { + "type": [ + "string", + "number" + ], + "description": "string or number, value of the enum entry." + }, + "brief": { + "type": "string", + "description": "brief description of the enum entry value. It defaults to the value of ID." + }, + "note": { + "type": "string", + "description": "longer description. It defaults to an empty string." + } + } + } + } + } + }, + "AttributeFullSpec": { + "required": [ + "id", + "type" + ], + "properties": { + "id": { + "type": "string", + "description": "unique string" + }, + "type": { + "oneOf": [ + { + "type": "string", + "enum": [ + "string", + "int", + "double", + "boolean", + "string[]", + "int[]", + "double[]", + "boolean[]" + ], + "description": "literal denoting the type" + }, + { + "$ref": "#/definitions/AttributeEnumType" + } + ] + } + } + }, + "AttributeReference": { + "type": "object", + "required": [ + "ref" + ], + "properties": { + "ref": { + "type": "string", + "description": "reference an existing attribute" + }, + "tag": { + "type": "string", + "description": "associates a tag to the attribute" + } + } + }, + "ValueType": { + "oneOf": [ + { + "type": [ + "string", + "boolean", + "number" + ] + }, + { + "type": "array", + "items": { + "type": [ + "boolean", + "number", + "string" + ] + } + } + ] + }, + "Attribute": { + "type": "object", + "allOf": [ + { + "properties": { + "required": { + "description": "specifies if the attribute is mandatory. Can be 'always', or 'conditional'. When omitted, the attribute is not required. When set to 'conditional',the string provided as MUST specify the conditions under which the attribute is required.", + "oneOf": [ + { + "type": "string", + "enum": [ + "always" + ] + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "conditional" + ], + "properties": { + "conditional": { + "type": "string" + } + } + } + ] + }, + "sampling_relevant": { + "type": "boolean", + "description": "specifies if it is relevant for sampling. It defaults to false.", + "default": false + }, + "brief": { + "type": "string", + "description": "brief description of the attribute." + }, + "note": { + "type": "string", + "description": "additional notes to the attribute. It defaults to an empty string." + }, + "examples": { + "$ref": "#/definitions/ValueType", + "description": "sequence/dictionary of example values for the attribute. They are optional for boolean and enum attributes. Example values must be of the same type of the attribute. If only a single example is provided, it can directly be reported without encapsulating it into a sequence/dictionary." + }, + "deprecated": { + "type": "string", + "description": "specifies if the attribute is deprecated. The string provided as MUST specify why it's deprecated and/or what to use instead." + } + } + }, + { + "oneOf": [ + { + "$ref": "#/definitions/AttributeFullSpec" + }, + { + "$ref": "#/definitions/AttributeReference" + } + ] + } + ] + } + } +} From fcf0f486b532660d43c1e77073dabacbabce7c9c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Wed, 14 Jul 2021 15:33:19 +0200 Subject: [PATCH 4/8] More doc updates. --- semantic-conventions/README.md | 46 ++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/semantic-conventions/README.md b/semantic-conventions/README.md index 28f10f3e..3be7fa68 100644 --- a/semantic-conventions/README.md +++ b/semantic-conventions/README.md @@ -16,6 +16,24 @@ For help try: docker run --rm otel/semconvgen -h ``` +## Model definition language (YAML input) + +The expected YAML input file format is documented in [syntax.md](./syntax.md). + +There is also a JSON schema definition available for the YAML files, which can +be used e.g. in VS code to get validation and auto-completion: [semconv.schema.json](semconv.schema.json). For example, with the `redhat.vscode-yaml` plugin, use the following snippet in your VS Code `settings.json` to apply it +to the test YAML files: + +```json +{ + "yaml.schemas": { + "./semantic-conventions/semconv.schema.json": [ + "semantic-conventions/src/tests/**/*.yaml" + ] + } +} +``` + ## Markdown Tables Tables can be generated using the command: @@ -31,7 +49,7 @@ Where `{yaml_folder}` is the absolute path to the directory containing the yaml The tool will automatically replace the tables with the up to date definition of the semantic conventions. To do so, the tool looks for special tags in the markdown. -``` +```html ``` @@ -71,10 +89,24 @@ The image supports [Jinja](https://jinja.palletsprojects.com/en/2.11.x/) templat For example, opentelemetry-java [generates typed constants for semantic conventions](https://github.com/open-telemetry/opentelemetry-java/blob/main/semconv/src/main/java/io/opentelemetry/semconv/trace/attributes/SemanticAttributes.java) here using [this template](https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/templates/SemanticAttributes.java.j2). -The image can generate code with the following command: +The commands used to generate that are +[here in the opentelemetry-java repo](https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/generate.sh). +Note especially the `docker run` commands. For example to generate the aforementioned `SemanticAttributes.java`, +the following command is used: ```bash -docker run --rm otel/semconvgen --yaml-root {yaml_folder} code --template {jinja-file} --output {output-file} +docker run --rm \ + -v ${SCRIPT_DIR}/opentelemetry-specification/semantic_conventions/trace:/source \ + -v ${SCRIPT_DIR}/templates:/templates \ + -v ${ROOT_DIR}/semconv/src/main/java/io/opentelemetry/semconv/trace/attributes/:/output \ + otel/semconvgen:$GENERATOR_VERSION \ + --yaml-root /source \ + code \ + --template /templates/SemanticAttributes.java.j2 \ + --output /output/SemanticAttributes.java \ + -Dclass=SemanticAttributes \ + -DschemaUrl=$SCHEMA_URL \ + -Dpkg=io.opentelemetry.semconv.trace.attributes ``` By default, all models are fed into the specified template at once, i.e. only a single file is generated. @@ -89,10 +121,4 @@ This way, multiple files are generated. The value of `pattern` can be one of the - `extends`: The id of the parent semantic convention. Finally, additional value can be passed to the template in form of `key=value` pairs separated by -comma using the `--parameters [{key=value},]+` flag. - -For example, to generate the typed spans used by the [opentelemetry-java-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/master/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/typedspan), the following command can be used: - -```bash -docker run --rm otel/semconvgen --yaml-root ${yamls} code --template typed_span_class.java.j2 --file-per-group semconv_id -o ${output}/Span.java -``` +comma using the `--parameters [{key=value},]+` or `-D` flag. From b68ba38199e2fab7caf76245861274074d5f88fa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Thu, 15 Jul 2021 14:19:24 +0200 Subject: [PATCH 5/8] Apply suggestions from code review Co-authored-by: Armin Ruech --- semantic-conventions/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/semantic-conventions/README.md b/semantic-conventions/README.md index 3be7fa68..20c15c36 100644 --- a/semantic-conventions/README.md +++ b/semantic-conventions/README.md @@ -21,7 +21,8 @@ docker run --rm otel/semconvgen -h The expected YAML input file format is documented in [syntax.md](./syntax.md). There is also a JSON schema definition available for the YAML files, which can -be used e.g. in VS code to get validation and auto-completion: [semconv.schema.json](semconv.schema.json). For example, with the `redhat.vscode-yaml` plugin, use the following snippet in your VS Code `settings.json` to apply it +be used e.g. in VS code to get validation and auto-completion: [semconv.schema.json](./semconv.schema.json). +For example, with the `redhat.vscode-yaml` plugin, use the following snippet in your VS Code `settings.json` to apply it to the test YAML files: ```json @@ -49,7 +50,7 @@ Where `{yaml_folder}` is the absolute path to the directory containing the yaml The tool will automatically replace the tables with the up to date definition of the semantic conventions. To do so, the tool looks for special tags in the markdown. -```html +```markdown ``` From dd28ca08a3527c5d15c498869f33bbf01446ff89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Thu, 15 Jul 2021 14:38:41 +0200 Subject: [PATCH 6/8] Update semantic-conventions/README.md Co-authored-by: Armin Ruech --- semantic-conventions/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/semantic-conventions/README.md b/semantic-conventions/README.md index 20c15c36..7ce2b398 100644 --- a/semantic-conventions/README.md +++ b/semantic-conventions/README.md @@ -88,7 +88,7 @@ semantic conventions that have the tag `network`. The image supports [Jinja](https://jinja.palletsprojects.com/en/2.11.x/) templates to generate code from the models. For example, opentelemetry-java [generates typed constants for semantic conventions](https://github.com/open-telemetry/opentelemetry-java/blob/main/semconv/src/main/java/io/opentelemetry/semconv/trace/attributes/SemanticAttributes.java) -here using [this template](https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/templates/SemanticAttributes.java.j2). +using [this template](https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/templates/SemanticAttributes.java.j2). The commands used to generate that are [here in the opentelemetry-java repo](https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/generate.sh). From 583e29debbd732e2a3b6df9d625c4f1b120b8672 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Fri, 16 Jul 2021 19:13:52 +0200 Subject: [PATCH 7/8] Fix semconv.schema.json. --- semantic-conventions/semconv.schema.json | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/semantic-conventions/semconv.schema.json b/semantic-conventions/semconv.schema.json index fa3d78db..bf7ef96c 100644 --- a/semantic-conventions/semconv.schema.json +++ b/semantic-conventions/semconv.schema.json @@ -11,17 +11,16 @@ "additionalProperties": false, "required": [ "id", - "type", "brief" ], "anyOf": [ { - "require": [ - "attributs" + "required": [ + "attributes" ] }, { - "require": [ + "required": [ "extends" ] } From f46daffaea83cbf6f98496645512ff7d39ecf604 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20Neum=C3=BCller?= Date: Fri, 16 Jul 2021 19:21:09 +0200 Subject: [PATCH 8/8] Add note on keeping syntax.md/semconv.schema.json in sync, add TOC. --- semantic-conventions/syntax.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/semantic-conventions/syntax.md b/semantic-conventions/syntax.md index 49bad47e..d69e4401 100644 --- a/semantic-conventions/syntax.md +++ b/semantic-conventions/syntax.md @@ -3,6 +3,31 @@ First, the syntax with a pseudo [EBNF](https://en.wikipedia.org/wiki/Extended_Backus-Naur_form) grammar is presented. Then, the semantic of each field is described. + + + + +- [JSON Schema](#json-schema) +- [Syntax](#syntax) +- [Semantics](#semantics) + * [Groups](#groups) + * [Semantic Convention](#semantic-convention) + * [Attributes](#attributes) + * [Ref](#ref) + * [Type](#type) + * [Constraints](#constraints) + + [Any Of](#any-of) + + [Include](#include) + + + +## JSON Schema + +A JSON schema description of the syntax is available as [semconv.schema.json](./semconv.schema.json), +see [README.md](./README.md) for how to use it with an editor. The documentation +here in `syntax.md` should be considered more authoritative though. Please keep +`semconv.schema.json` in synch when changing the "grammar" in this file! + ## Syntax All attributes are lower case.