Skip to content

docs: add distributed tracing section to production deployment guide #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ankurs merged 2 commits into from
Mar 28, 2026
Merged

docs: add distributed tracing section to production deployment guide #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7d7b291
docs: add distributed tracing section to production deployment guide
ankurs Mar 28, 2026
2bffed9
docs: add production deployment link to quickstart and home page
ankurs Mar 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ ColdBrew integrates with the tools you already use:
- **[Getting Started](/getting-started)** — Create your first ColdBrew service
- **[Using ColdBrew](/using)** — Configure and extend your service
- **[How-To Guides](/howto)** — Step-by-step guides for common tasks
- **[Production Deployment](/howto/production)** — Kubernetes, health probes, tracing, and graceful shutdown
- **[Integrations](/integrations)** — Set up monitoring, tracing, and error tracking
- **[FAQ](/faq)** — Common questions and answers

Expand Down
76 changes: 75 additions & 1 deletion howto/production.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
layout: default
title: "Production Deployment"
parent: "How To"
description: "Deploy ColdBrew Go services to production with Kubernetes manifests, health probes, Prometheus, and graceful shutdown"
description: "Deploy ColdBrew Go services to production with Kubernetes manifests, health probes, Prometheus, distributed tracing, and graceful shutdown"
---
## Table of contents
{: .no_toc .text-delta }
Expand Down Expand Up @@ -129,6 +129,7 @@ type: Opaque
stringData:
NEW_RELIC_LICENSE_KEY: "your-license-key"
SENTRY_DSN: "https://your-dsn@sentry.io/123"
OTLP_HEADERS: "x-honeycomb-team=your-api-key" # if your OTLP backend needs auth
```

### Service
Expand Down Expand Up @@ -260,6 +261,79 @@ env:
value: "0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10"
```

## Distributed tracing

ColdBrew sends traces via OpenTelemetry to any OTLP-compatible backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, etc.) or New Relic.

### OTLP backend (Jaeger, Tempo, Honeycomb, etc.)

```yaml
env:
- name: OTLP_ENDPOINT
value: "otel-collector.monitoring:4317" # your OTLP collector
- name: OTLP_SAMPLING_RATIO
value: "0.1" # sample 10% of traces in production
```

For backends that require authentication headers:

```yaml
env:
- name: OTLP_ENDPOINT
value: "api.honeycomb.io:443"
- name: OTLP_HEADERS
value: "x-honeycomb-team=your-api-key"
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example includes an API key inline in the manifest (OTLP_HEADERS: "x-honeycomb-team=your-api-key"). Elsewhere on this page you recommend putting secrets in Kubernetes Secrets; consider showing valueFrom.secretKeyRef here (or explicitly noting the header value should come from a Secret) to avoid encouraging in-repo/plaintext configs.

Suggested change
value: "x-honeycomb-team=your-api-key"
valueFrom:
secretKeyRef:
name: myservice-secrets
key: OTLP_HEADERS

Copilot uses AI. Check for mistakes.
- name: OTLP_SAMPLING_RATIO
value: "0.1"
```

{: .note }
For local development, set `OTLP_INSECURE=true` and point to a local Jaeger instance (`localhost:4317`). See the [config reference](/config-reference#example-local-development-with-jaeger-via-otlp) for a full example.

### New Relic

New Relic tracing is configured separately and can run alongside OTLP:
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The text says “New Relic tracing … can run alongside OTLP”, but the config reference states that when OTLP_ENDPOINT is set it takes precedence over New Relic OpenTelemetry configuration. This is internally inconsistent and could confuse users about whether traces are dual-exported. Please clarify the precedence/interaction here (or link to the canonical behavior) so the guide matches the config reference.

Suggested change
New Relic tracing is configured separately and can run alongside OTLP:
New Relic tracing is configured separately. It can be enabled in the same service as OTLP, but when `OTLP_ENDPOINT` is set it takes precedence over the New Relic OpenTelemetry exporter, so traces are not dual-exported. See the [config reference](/config-reference#distributed-tracing) for details:

Copilot uses AI. Check for mistakes.

```yaml
env:
- name: NEW_RELIC_LICENSE_KEY
valueFrom:
secretKeyRef:
name: myservice-secrets
key: NEW_RELIC_LICENSE_KEY
- name: NEW_RELIC_OPENTELEMETRY
value: "true"
- name: NEW_RELIC_OPENTELEMETRY_SAMPLE
value: "0.2"
```

### What gets traced

ColdBrew automatically creates spans for:

| Source | Span kind | Example |
|--------|-----------|---------|
| Incoming gRPC RPCs | Server | `/pkg.Service/Method` |
| Incoming HTTP requests | Server | `ServeHTTP` |
| Outbound gRPC calls (gateway) | Client | `/pkg.Service/Method` |
| `tracing.NewInternalSpan()` | Internal | Custom business logic spans |
| `tracing.NewDatastoreSpan()` | Client | Database/Redis operations |
| `tracing.NewExternalSpan()` | Client | External HTTP/API calls |

### Sampling in production

Set `OTLP_SAMPLING_RATIO` based on your traffic volume:

| QPS | Recommended ratio | Traces/sec |
|-----|-------------------|------------|
| 100 | `1.0` | 100 |
| 1,000 | `0.1` | 100 |
| 10,000 | `0.01` | 100 |
| 70,000+ | `0.001–0.01` | 70–700 |
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The “Recommended ratio” cell uses inline-code to show a range (0.001–0.01). Since OTLP_SAMPLING_RATIO is a single float, readers may copy/paste the range literally and end up with an invalid value. Consider expressing this as plain text (e.g., “0.001 to 0.01”) or splitting into min/max guidance.

Suggested change
| 70,000+ | `0.001–0.01` | 70–700 |
| 70,000+ | 0.001 to 0.01 | 70–700 |

Copilot uses AI. Check for mistakes.

{: .important }
Sampling is parent-based — if an incoming request already has a sampled trace context, ColdBrew respects that decision regardless of the local ratio.

## gRPC load balancing

gRPC uses HTTP/2 with long-lived connections. A standard Kubernetes Service with `ClusterIP` won't distribute load across pods — all requests go over a single connection to one pod.
Expand Down
1 change: 1 addition & 0 deletions quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -465,5 +465,6 @@ Your service starts on `:9090` (gRPC) and `:9091` (HTTP) with metrics, health ch

- **[Using ColdBrew](/using)** — Configure ports, environment variables, and interceptors
- **[How-To Guides](/howto)** — Tracing, logging, metrics, error handling, and more
- **[Production Deployment](/howto/production)** — Kubernetes manifests, health probes, tracing, and graceful shutdown
- **[Integrations](/integrations)** — Connect New Relic, Prometheus, Sentry, Jaeger
- **[FAQ](/faq)** — Common questions and gotchas
Loading