Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,15 @@ Further reading:
- [Weaver Architecture](docs/architecture.md): A document detailing the architecture of the project.
- [Application Telemetry Schema OTEP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/0243-app-telemetry-schema-vision-roadmap.md): A vision and roadmap for the concept of Application Telemetry Schema.

## Schema and Configuration Reference
Comment thread
jsuereth marked this conversation as resolved.
Outdated

When working with Weaver, these schema and configuration references are essential:

- **[Semantic Convention Schema Syntax](./schemas/semconv-syntax.md)** - Complete YAML syntax reference for defining semantic conventions
- **[Semantic Convention JSON Schema](./schemas/semconv.schema.json)** - JSON Schema for validation and IDE support
- **[Weaver Configuration (`weaver.yaml`)](./docs/weaver-config.md)** - Configuration file reference for code generation
- **[Weaver Forge Documentation](./crates/weaver_forge/README.md)** - In-depth guide to Weaver's templating engine

## Examples and How-Tos

- [Weaver Examples](https://github.com/open-telemetry/opentelemetry-weaver-examples) - Official OTel project showcasing weaver use cases through runnable examples.
Expand Down
33 changes: 18 additions & 15 deletions crates/weaver_forge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@ Weaver Forge is a component of OTEL Weaver that facilitates documentation and
code generation from a semantic convention registry. It uses MiniJinja, a
template engine compatible with Jinja2 syntax, which provides extensive
customization options (refer to this [GitHub repository](https://github.com/mitsuhiko/minijinja)
for more details). Some good references to start developing Jinja2 templages are
[1](https://ttl255.com/jinja2-tutorial-part-2-loops-and-conditionals/) and
[2](https://jinja.palletsprojects.com/en/stable/templates).
for more details).

**New to Jinja?** These resources will help you get started:
- [Official Jinja Template Documentation](https://jinja.palletsprojects.com/en/stable/templates) - Complete syntax reference
- [Jinja2 Tutorial - Loops and Conditionals](https://ttl255.com/jinja2-tutorial-part-2-loops-and-conditionals/) - Practical tutorial with examples

To streamline template creation for semantic conventions,
additional filters, functions, tests, and naming conventions have been
integrated with the standard Jinja logic.
Expand Down Expand Up @@ -485,25 +488,25 @@ compatibility extensions that are also enabled in Weaver.
In addition, OTel Weaver provides a set of custom filters to facilitate the
generation of documentation and code.

The following filters are available (the code for all available extension can be found
[here](./src/extensions)):
The following filters are available (implementation can be found in the
[extensions source code](./src/extensions)):

- `prometheus_metric_name`: Generates a single Prometheus metric name from an OpenTelemetry metric
(more details [here](#prometheus-filters)).
- `prometheus_metric_names`: Generates multiple Prometheus metric names for different translation
strategies with optional histogram/summary expansion (more details [here](#prometheus-filters)).
- `prometheus_unit_name`: Converts OpenTelemetry units to Prometheus-compliant unit names (more
details [here](#prometheus-filters)).
- `lower_case`: Converts a string to lowercase.
- `upper_case`: Converts a string to UPPERCASE.
- `title_case`: Converts a string to TitleCase.
- `pascal_case`: Converts a string to PascalCase.
- `camel_case`: Converts a string to camelCase.
- `snake_case`: Converts a string to snake_case.
- `screaming_snake_case`: Converts a string to SCREAMING_SNAKE_CASE.
- `kebab_case`: Converts a string to kebab-case.
- `screaming_kebab_case`: Converts a string to SCREAMING-KEBAB-CASE.
- `capitalize_first`: Capitalizes the first letter of a string.
- `lower_case`: Converts a string to lowercase ([source](./src/extensions/case.rs)).
Comment thread
jsuereth marked this conversation as resolved.
Outdated
- `upper_case`: Converts a string to UPPERCASE ([source](./src/extensions/case.rs)).
- `title_case`: Converts a string to TitleCase ([source](./src/extensions/case.rs)).
- `pascal_case`: Converts a string to PascalCase ([source](./src/extensions/case.rs)).
- `camel_case`: Converts a string to camelCase ([source](./src/extensions/case.rs)).
- `snake_case`: Converts a string to snake_case ([source](./src/extensions/case.rs)).
- `screaming_snake_case`: Converts a string to SCREAMING_SNAKE_CASE ([source](./src/extensions/case.rs)).
- `kebab_case`: Converts a string to kebab-case ([source](./src/extensions/case.rs)).
- `screaming_kebab_case`: Converts a string to SCREAMING-KEBAB-CASE ([source](./src/extensions/case.rs)).
- `capitalize_first`: Capitalizes the first letter of a string ([source](./src/extensions/case.rs)).
- `kebab_case_const`: Generates kebab-case constants which follow semantic convention namespacing rules (underscores are
ignored, but . is meaningful).
- `pascal_case_const`: Generates PascalCase constants which follow semantic convention namespacing rules (underscores
Expand Down
137 changes: 131 additions & 6 deletions docs/codegen.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,32 @@

<kbd>weaver registry generate</kbd>

|Quick Links | [weaver.yaml](weaver-config.md) | [JQ](/crates/weaver_forge/README.md#jq-filters-reference) | [Jinja](/crates/weaver_forge/README.md#jinja-filters-reference) |
|-|-|-|-|
|Quick Links | [weaver.yaml](weaver-config.md) | [Semconv Schema](../schemas/semconv-syntax.md) | [JQ Filters](../crates/weaver_forge/README.md#jq-filters-reference) | [Jinja Filters](../crates/weaver_forge/README.md#jinja-filters-reference) |
|-|-|-|-|-|

A core element of schema-first telemetry are the artifacts it enables:
- **Up-to-Date Documentation**: Created right from the schema, the source of truth. When a metric changes, the docs do so, too.
- **Type-safe Constructors**: `New*` functions for **_your_** telemetry that **always** create schema-conforming results.
- etc.

>[!Note]
>Weaver enables this using code-generation, powered by [minijinja](https://github.com/mitsuhiko/minijinja) and [jaq](https://github.com/01mf02/jaq).
>Full documentation is available at [Weaver Forge](/crates/weaver_forge/README.md)
>Weaver enables this using code-generation, powered by [minijinja](https://github.com/mitsuhiko/minijinja) (Jinja2-compatible templating) and [jaq](https://github.com/01mf02/jaq) (jq-compatible data processing).
>Full documentation is available at [Weaver Forge](../crates/weaver_forge/README.md)

## Learning Jinja Templates

If you're new to Jinja or need a refresher, these resources will help you get started:

- **[Official Jinja Template Documentation](https://jinja.palletsprojects.com/en/stable/templates)** - Comprehensive reference for Jinja template syntax
- **[Jinja2 Tutorial - Loops and Conditionals](https://ttl255.com/jinja2-tutorial-part-2-loops-and-conditionals/)** - Practical tutorial with examples
- **[MiniJinja Documentation](https://docs.rs/minijinja/latest/minijinja/)** - Rust implementation used by Weaver (mostly compatible with Jinja2)

Weaver extends Jinja with custom filters and functions specifically designed for semantic convention generation. See the [Jinja Filters Reference](../crates/weaver_forge/README.md#jinja-filters-reference) for details.

## High-level overview

>[!Tip]
>For a full tutorial, consider the [Step-by-Step Guide](/crates/weaver_forge/README.md#step-by-step-guide)
>For a full tutorial, consider the [Step-by-Step Guide](../crates/weaver_forge/README.md#step-by-step-guide)

Let's say we want to generate Markdown documentation for the metrics of our application or convention.
With Weaver Forge, the following directory layout is used by default:
Expand Down Expand Up @@ -104,10 +114,90 @@ jinja --> file1
jinja --> file2
```

Weaver resolves the entire registry (http and grpc in this case) into a single document. This is passed to the [JQ filter](/crates/weaver_forge/README.md#jq-filters) `semconv_grouped_metrics`. This groups individual metrics by their root namespace (`http` or `grpc`)
Weaver resolves the entire registry (http and grpc in this case) into a single document. This is passed to the [JQ filter](../crates/weaver_forge/README.md#jq-filters) `semconv_grouped_metrics`. This groups individual metrics by their root namespace (`http` or `grpc`)

This output in turn is passed to the `metrics.md.j2` template, evaluated by the [minijinja](https://github.com/mitsuhiko/minijinja) templating engine. Because `application_mode` is set to `each`, the template is invoked for each group, so this yields a `http.md` and a separate `grpc.md`.

## Understanding Filters

The `filter` field in `weaver.yaml` is a powerful feature that preprocesses your schema data before it reaches your templates. Think of filters as data transformations that shape how your templates receive information.

### Default Behavior (No Filter)

When you **don't specify a filter**, the entire resolved registry is passed directly to your template in the `ctx` variable:

```yaml
templates:
- template: "all_data.j2"
application_mode: single
# No filter specified - ctx contains the entire registry
Comment thread
jsuereth marked this conversation as resolved.
```

Your template receives everything:
```jinja
{# ctx.groups contains all semantic convention groups #}
{% for group in ctx.groups %}
{{ group.id }}
{% endfor %}
```

### With a Filter

When you **do specify a filter**, it transforms the data before passing it to the template:

```yaml
templates:
- template: "metrics.md.j2"
filter: semconv_grouped_metrics # This filter groups metrics by namespace
Comment thread
jsuereth marked this conversation as resolved.
Outdated
application_mode: each
```

Now `ctx` contains the output of the filter. For `semconv_grouped_metrics`, each invocation receives:
```jinja
{# ctx now has root_namespace and metrics fields #}
## {{ ctx.root_namespace }}
{% for metric in ctx.metrics %}
- {{ metric.metric_name }}
{% endfor %}
```

### Common Filters

- **`semconv_grouped_attributes`** - Groups attributes by root namespace (e.g., `http`, `db`)
- **`semconv_grouped_metrics`** - Groups metrics by root namespace
- **`semconv_grouped_events`** - Groups events by root namespace

### Filter Options

Many filters accept options to customize their behavior:

```yaml
templates:
- template: "stable_attrs.j2"
filter: >
semconv_grouped_attributes({
"stable_only": true,
"exclude_deprecated": true,
"exclude_root_namespace": ["url", "network"]
})
application_mode: each
```

**Options explained:**
- `stable_only: true` - Only include stable attributes
- `exclude_deprecated: true` - Skip deprecated attributes
- `exclude_root_namespace` - Exclude specific namespaces

### The `ctx` Variable

The `ctx` variable is the primary way your template accesses data:

- **Without a filter**: `ctx` = the entire resolved registry
- **With `application_mode: single`**: `ctx` = the filter's complete output
- **With `application_mode: each`**: `ctx` = one element from the filter's output array

For more details on available filters, see the [JQ Filters Reference](../crates/weaver_forge/README.md#jq-filters-reference).


## Tips and Tricks

Expand All @@ -122,3 +212,38 @@ Jinja2 can be overwhelming and hard to discover. Try putting `debug()` somewhere
```

This gives you a JSON-like dump of every exact variable and corresponding value, identifier, function, filter, test, etc.

## Built-in Helper Functions and Filters

Weaver provides numerous built-in filters and functions to simplify template development. Here are the most commonly used ones:

### Case Conversion Filters

Transform strings between different naming conventions:

- `kebab_case` - Converts to kebab-case: `{{ "Hello World" | kebab_case }}` → `hello-world`
- `snake_case` - Converts to snake_case: `{{ "Hello World" | snake_case }}` → `hello_world`
- `pascal_case` - Converts to PascalCase: `{{ "hello_world" | pascal_case }}` → `HelloWorld`
- `camel_case` - Converts to camelCase: `{{ "hello_world" | camel_case }}` → `helloWorld`
- `screaming_snake_case` - Converts to SCREAMING_SNAKE_CASE
- `screaming_kebab_case` - Converts to SCREAMING-KEBAB-CASE
- `upper_case`, `lower_case`, `title_case` - Standard case conversions
- `capitalize_first` - Capitalizes only the first character

**Example:**
```jinja
file_name: "{{ctx.root_namespace | snake_case}}.md"
Comment thread
lmolkova marked this conversation as resolved.
```

### Other Useful Filters

- `comment` - Format text as comments (language-specific, configured in weaver.yaml)
- `attribute_sort` - Sort attributes by requirement level, then name
- `required` / `not_required` - Filter attributes by requirement level
- `flatten` - Convert nested lists into a single list
- `map_text` - Map values using text_maps in weaver.yaml

For a complete list of all filters, functions, and tests, see:
- **[Jinja Filters Reference](../crates/weaver_forge/README.md#jinja-filters-reference)** - Complete filter documentation
- **[Jinja Functions Reference](../crates/weaver_forge/README.md#jinja-functions-reference)** - Available functions
- **[Source Code](../../crates/weaver_forge/src/extensions)** - Implementation details