Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions crates/weaver_forge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@ Weaver Forge is a component of OTEL Weaver that facilitates documentation and
code generation from a semantic convention registry. It uses MiniJinja, a
template engine compatible with Jinja2 syntax, which provides extensive
customization options (refer to this [GitHub repository](https://github.com/mitsuhiko/minijinja)
for more details). Some good references to start developing Jinja2 templages are
[1](https://ttl255.com/jinja2-tutorial-part-2-loops-and-conditionals/) and
[2](https://jinja.palletsprojects.com/en/stable/templates).
for more details).

**New to Jinja?** These resources will help you get started:
- [Official Jinja Template Documentation](https://jinja.palletsprojects.com/en/stable/templates) - Complete syntax reference
- [Jinja2 Tutorial - Loops and Conditionals](https://ttl255.com/jinja2-tutorial-part-2-loops-and-conditionals/) - Practical tutorial with examples

To streamline template creation for semantic conventions,
additional filters, functions, tests, and naming conventions have been
integrated with the standard Jinja logic.
Expand Down Expand Up @@ -485,8 +488,8 @@ compatibility extensions that are also enabled in Weaver.
In addition, OTel Weaver provides a set of custom filters to facilitate the
generation of documentation and code.

The following filters are available (the code for all available extension can be found
[here](./src/extensions)):
The following filters are available (implementation can be found in the
[extensions source code](./src/extensions)):

- `prometheus_metric_name`: Generates a single Prometheus metric name from an OpenTelemetry metric
(more details [here](#prometheus-filters)).
Expand Down
140 changes: 134 additions & 6 deletions docs/codegen.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,32 @@

<kbd>weaver registry generate</kbd>

|Quick Links | [weaver.yaml](weaver-config.md) | [JQ](/crates/weaver_forge/README.md#jq-filters-reference) | [Jinja](/crates/weaver_forge/README.md#jinja-filters-reference) |
|-|-|-|-|
|Quick Links | [weaver.yaml](weaver-config.md) | [Semconv Schema](../schemas/semconv-syntax.md) | [JQ Filters](../crates/weaver_forge/README.md#jq-filters-reference) | [Jinja Filters](../crates/weaver_forge/README.md#jinja-filters-reference) |
|-|-|-|-|-|

A core element of schema-first telemetry are the artifacts it enables:
- **Up-to-Date Documentation**: Created right from the schema, the source of truth. When a metric changes, the docs do so, too.
- **Type-safe Constructors**: `New*` functions for **_your_** telemetry that **always** create schema-conforming results.
- etc.

>[!Note]
>Weaver enables this using code-generation, powered by [minijinja](https://github.com/mitsuhiko/minijinja) and [jaq](https://github.com/01mf02/jaq).
>Full documentation is available at [Weaver Forge](/crates/weaver_forge/README.md)
>Weaver enables this using code-generation, powered by [minijinja](https://github.com/mitsuhiko/minijinja) (Jinja2-compatible templating) and [jaq](https://github.com/01mf02/jaq) (jq-compatible data processing).
>Full documentation is available at [Weaver Forge](../crates/weaver_forge/README.md)

## Learning Jinja Templates

If you're new to Jinja or need a refresher, these resources will help you get started:

- **[Official Jinja Template Documentation](https://jinja.palletsprojects.com/en/stable/templates)** - Comprehensive reference for Jinja template syntax
- **[Jinja2 Tutorial - Loops and Conditionals](https://ttl255.com/jinja2-tutorial-part-2-loops-and-conditionals/)** - Practical tutorial with examples
- **[MiniJinja Documentation](https://docs.rs/minijinja/latest/minijinja/)** - Rust implementation used by Weaver (mostly compatible with Jinja2)

Weaver extends Jinja with custom filters and functions specifically designed for semantic convention generation. See the [Jinja Filters Reference](../crates/weaver_forge/README.md#jinja-filters-reference) for details.

## High-level overview

>[!Tip]
>For a full tutorial, consider the [Step-by-Step Guide](/crates/weaver_forge/README.md#step-by-step-guide)
>For a full tutorial, consider the [Step-by-Step Guide](../crates/weaver_forge/README.md#step-by-step-guide)

Let's say we want to generate Markdown documentation for the metrics of our application or convention.
With Weaver Forge, the following directory layout is used by default:
Expand Down Expand Up @@ -104,10 +114,93 @@ jinja --> file1
jinja --> file2
```

Weaver resolves the entire registry (http and grpc in this case) into a single document. This is passed to the [JQ filter](/crates/weaver_forge/README.md#jq-filters) `semconv_grouped_metrics`. This groups individual metrics by their root namespace (`http` or `grpc`)
Weaver resolves the entire registry (http and grpc in this case) into a single document. This is passed to the [JQ filter](../crates/weaver_forge/README.md#jq-filters) `semconv_grouped_metrics`. This groups individual metrics by their root namespace (`http` or `grpc`)

This output in turn is passed to the `metrics.md.j2` template, evaluated by the [minijinja](https://github.com/mitsuhiko/minijinja) templating engine. Because `application_mode` is set to `each`, the template is invoked for each group, so this yields a `http.md` and a separate `grpc.md`.

## Understanding Filters

The `filter` field in `weaver.yaml` is a powerful feature that preprocesses your schema data before it reaches your templates. Think of filters as data transformations that shape how your templates receive information.

### Default Behavior (No Filter)

When you **don't specify a filter**, the entire resolved registry is passed directly to your template in the `ctx` variable:

```yaml
templates:
- template: "all_data.j2"
application_mode: single
# No filter specified - ctx contains the entire registry
Comment thread
jsuereth marked this conversation as resolved.
```

Your template receives everything:
```jinja
{# ctx.groups contains all semantic convention groups #}
{% for group in ctx.groups %}
{{ group.id }}
{% endfor %}
```

### With a Filter

When you **do specify a filter**, it transforms the data before passing it to the template. Filters use JQ expressions to select and transform data:

```yaml
templates:
- template: "metrics.md.j2"
filter: '.groups | map(select(.type == "metric"))' # Select only metric groups
application_mode: each
```

Now `ctx` contains the output of the filter - each metric group object:
```jinja
{# ctx is a single metric group #}
## {{ ctx.id }}
{{ ctx.brief }}
{% for attr in ctx.attributes %}
- {{ attr.name }}
{% endfor %}
```

### Common Helper Filters

For convenience, Weaver provides pre-built helper filters for common operations:

- **`semconv_grouped_attributes`** - Groups attributes by root namespace (e.g., `http`, `db`)
- **`semconv_grouped_metrics`** - Groups metrics by root namespace
- **`semconv_grouped_events`** - Groups events by root namespace

### Filter Options

Many filters accept options to customize their behavior:

```yaml
templates:
- template: "stable_attrs.j2"
filter: >
semconv_grouped_attributes({
"stable_only": true,
"exclude_deprecated": true,
"exclude_root_namespace": ["url", "network"]
})
application_mode: each
```

**Options explained:**
- `stable_only: true` - Only include stable attributes
- `exclude_deprecated: true` - Skip deprecated attributes
- `exclude_root_namespace` - Exclude specific namespaces

### The `ctx` Variable

The `ctx` variable is the primary way your template accesses data:

- **Without a filter**: `ctx` = the entire resolved registry
- **With `application_mode: single`**: `ctx` = the filter's complete output
- **With `application_mode: each`**: `ctx` = one element from the filter's output array

For more details on available filters, see the [JQ Filters Reference](../crates/weaver_forge/README.md#jq-filters-reference).


## Tips and Tricks

Expand All @@ -122,3 +215,38 @@ Jinja2 can be overwhelming and hard to discover. Try putting `debug()` somewhere
```

This gives you a JSON-like dump of every exact variable and corresponding value, identifier, function, filter, test, etc.

## Built-in Helper Functions and Filters

Weaver provides numerous built-in filters and functions to simplify template development. Here are the most commonly used ones:

### Case Conversion Filters

Transform strings between different naming conventions:

- `kebab_case` - Converts to kebab-case: `{{ "Hello World" | kebab_case }}` → `hello-world`
- `snake_case` - Converts to snake_case: `{{ "Hello World" | snake_case }}` → `hello_world`
- `pascal_case` - Converts to PascalCase: `{{ "hello_world" | pascal_case }}` → `HelloWorld`
- `camel_case` - Converts to camelCase: `{{ "hello_world" | camel_case }}` → `helloWorld`
- `screaming_snake_case` - Converts to SCREAMING_SNAKE_CASE
- `screaming_kebab_case` - Converts to SCREAMING-KEBAB-CASE
- `upper_case`, `lower_case`, `title_case` - Standard case conversions
- `capitalize_first` - Capitalizes only the first character

**Example:**
```jinja
file_name: "{{ctx.root_namespace | snake_case}}.md"
Comment thread
lmolkova marked this conversation as resolved.
```

### Other Useful Filters

- `comment` - Format text as comments (language-specific, configured in weaver.yaml)
- `attribute_sort` - Sort attributes by requirement level, then name
- `required` / `not_required` - Filter attributes by requirement level
- `flatten` - Convert nested lists into a single list
- `map_text` - Map values using text_maps in weaver.yaml

For a complete list of all filters, functions, and tests, see:
- **[Jinja Filters Reference](../crates/weaver_forge/README.md#jinja-filters-reference)** - Complete filter documentation
- **[Jinja Functions Reference](../crates/weaver_forge/README.md#jinja-functions-reference)** - Available functions
- **[Source Code](../../crates/weaver_forge/src/extensions)** - Implementation details