Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,9 @@ See [Custom HTTP Routes](/howto/APIs/#custom-http-routes) for full examples incl

## Is hystrixprometheus still maintained?

**No.** The `hystrixprometheus` package depends on `afex/hystrix-go`, which is unmaintained. Do not invest in this package for new projects.
**No.** The `hystrixprometheus` package depends on `afex/hystrix-go`, which is unmaintained and will be archived in v1.

For circuit breaking, consider [failsafe-go](https://github.com/failsafe-go/failsafe-go) as an alternative. The client-side interceptors in the `interceptors` package provide retry and circuit breaking functionality that covers most use cases.
ColdBrew now provides a pluggable executor hook (`interceptors.SetDefaultExecutor`) for circuit breaking. Use [failsafe-go](https://github.com/failsafe-go/failsafe-go) as the recommended resilience library. See the [integrations page](/integrations/) for setup examples.

## How do I do cross-package development?

Expand Down
2 changes: 1 addition & 1 deletion Index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ A Kubernetes-native Go microservice framework for building production-grade gRPC
| **gRPC + REST Gateway** | Define your API once in protobuf — get gRPC, REST, and [Swagger docs](/architecture#self-documenting-apis) automatically via [grpc-gateway]. HTTP gateway supports JSON, `application/proto`, and `application/protobuf` [content types](/howto/APIs/#http-content-type) out of the box |
| **Structured Logging** | Native [slog] with custom Handler — per-request context fields, trace ID propagation, and typed attrs for zero-boxing performance |
| **Distributed Tracing** | [OpenTelemetry] and [New Relic] support with automatic span creation via gRPC stats handlers — traces can be sent to any OTLP-compatible backend including [Jaeger] |
| **Prometheus Metrics** | Built-in request latency, error rate, and circuit breaker metrics at `/metrics` |
| **Prometheus Metrics** | Built-in request latency, error rate, and gRPC status code metrics at `/metrics` |
| **Error Tracking** | Stack traces, gRPC status codes, and async notification to [Sentry], Rollbar, or Airbrake |
| **Rate Limiting** | Per-pod token bucket rate limiter — disabled by default, pluggable via custom [`ratelimit.Limiter`](https://pkg.go.dev/github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/ratelimit#Limiter) interface for distributed or per-tenant rate limiting. Config: `RATE_LIMIT_PER_SECOND`. See [interceptors howto](/howto/interceptors#rate-limiting) |
| **Auth Examples** | JWT and API key authentication interceptor examples in the [cookiecutter template][ColdBrew cookiecutter], built on [go-grpc-middleware auth](https://github.com/grpc-ecosystem/go-grpc-middleware/tree/main/interceptors/auth). See [auth howto](/howto/auth/) |
Expand Down
4 changes: 2 additions & 2 deletions Packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ Tracing is a library that provides distributed tracing to Go applications. It of

Documentation can be found at [tracing-docs]

## [Hystrix Prometheus] (Deprecated)
hystrixprometheus provides a Prometheus metrics collector for Hystrix (https://github.com/afex/hystrix-go). This package is deprecated as hystrix-go is unmaintained. Consider migrating to [failsafe-go](https://github.com/failsafe-go/failsafe-go).
## [Hystrix Prometheus] (Deprecated — will be archived in v1)
hystrixprometheus provides a Prometheus metrics collector for Hystrix. This package is deprecated as hystrix-go is unmaintained. Use `interceptors.SetDefaultExecutor` with [failsafe-go](https://github.com/failsafe-go/failsafe-go) instead. See [integrations](/integrations/#circuit-breaker--resilience).
Comment thread
ankurs marked this conversation as resolved.

Documentation can be found at [hystrixprometheus-docs]

Expand Down
39 changes: 14 additions & 25 deletions howto/Metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: "Metrics"
parent: "How To"
nav_order: 6
description: "Prometheus and OpenTelemetry metrics in ColdBrew: default runtime metrics, OTLP export, custom counters and histograms, and Hystrix circuit breaker monitoring"
description: "Prometheus and OpenTelemetry metrics in ColdBrew: default runtime metrics, OTLP export, custom counters and histograms, and circuit breaker monitoring"
---
## Table of contents
{: .no_toc .text-delta }
Expand All @@ -19,7 +19,6 @@ A collection of metrics are collected by default, including:
* Golang runtime metrics (e.g. memory usage, goroutine count, etc.)
* gRPC Client/Server metrics (e.g. request count, request duration, etc.)
* HTTP request metrics (e.g. request count, request duration, etc.)
* [Hystrix-go] circuit breaker metrics (e.g. request count, request duration, etc.) powered by [Hystrix Prometheus]

## Application Metrics Package (Cookiecutter)

Expand Down Expand Up @@ -136,36 +135,26 @@ Health check, readiness, and server reflection RPCs are bucketed under a generic

Both export pipelines use independent metric names and registries, so there is no conflict or double-counting.

## How to use Hystrix Metrics in Prometheus
## Circuit Breaker Metrics

{: .warning }
Hystrix-Go is unmaintained (last updated 2018). Consider migrating to [failsafe-go](https://github.com/failsafe-go/failsafe-go) for circuit breaker functionality.

[Hystrix Prometheus] is a library that provides a Prometheus metrics collector for [Hystrix-go]. To use it, you can register the collector with the default Prometheus registry:
Circuit breaker metrics are the responsibility of the resilience library you plug into ColdBrew's executor hook. For example, with [failsafe-go](https://github.com/failsafe-go/failsafe-go) you can attach event listeners to track circuit state:

Comment thread
ankurs marked this conversation as resolved.
```go

import (
metricCollector "github.com/afex/hystrix-go/hystrix/metric_collector"
"github.com/go-coldbrew/hystrixprometheus"
"github.com/prometheus/client_golang/prometheus"
)

// setupHystrix sets up the hystrix metrics
// This is a workaround for hystrix-go not supporting the prometheus registry
func setupHystrix() {
promC := hystrixprometheus.NewPrometheusCollector("hystrix", nil, prometheus.DefBuckets)
metricCollector.Registry.Register(promC.Collector)
}
cb := circuitbreaker.NewBuilder[any]().
WithFailureThreshold(5).
WithDelay(5 * time.Second).
OnStateChanged(func(e circuitbreaker.StateChangedEvent) {
circuitStateGauge.WithLabelValues(commandName).Set(float64(e.NewState))
}).
Build()
Comment thread
ankurs marked this conversation as resolved.
```

{: .note .note-info }
If you are using the `go-coldbrew/core` package, you can skip the above step as it will automatically register the collector for you.
See [Hystrix Prometheus] for more details.
ColdBrew's built-in `grpc_client_*` Prometheus metrics (request count, duration, status codes) cover request-level observability automatically. See the [integrations page](/integrations/#circuit-breaker--resilience) for full executor setup examples.

{: .warning }
**Legacy:** The previous `hystrixprometheus` package and `core.SetupHystrixPrometheus()` are deprecated. Circuit breaker metrics are now managed by your chosen resilience library, not ColdBrew.

---
[HTTP port]: https://pkg.go.dev/github.com/go-coldbrew/core/config#readme-type-config
[Hystrix Prometheus]: https://pkg.go.dev/github.com/go-coldbrew/hystrixprometheus
[Hystrix-go]: https://github.com/afex/hystrix-go
[Prometheus Go client library]: https://github.com/prometheus/client_golang
[ColdBrew cookiecutter]: /getting-started
139 changes: 126 additions & 13 deletions integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,28 +231,143 @@ func main() {
}
```

## Hystrix-Go
## Circuit Breaker / Resilience

{: .warning }
Hystrix-Go (`afex/hystrix-go`) is unmaintained (last updated 2018). Consider migrating to [failsafe-go] for circuit breaker functionality.
ColdBrew provides a pluggable executor hook for client-side resilience (circuit breaking, retries, bulkheading, etc.). You bring your own resilience library — [failsafe-go] is recommended.

[Hystrix-Go] is a Go implementation of the circuit breaker pattern. ColdBrew provides Prometheus metrics integration for Hystrix.
### How it works

### Initialising
Call `interceptors.SetDefaultExecutor` during init to register your resilience logic. The executor receives the gRPC method name, so you can choose which methods get circuit breaking:

If your app is using [ColdBrew cookiecutter] template, initialisation is done automatically.
```go
import (
"github.com/failsafe-go/failsafe-go"
"github.com/failsafe-go/failsafe-go/circuitbreaker"
"github.com/go-coldbrew/interceptors"
)

func init() {
cb := circuitbreaker.NewBuilder[any]().
WithFailureThreshold(5).
WithDelay(5 * time.Second).
WithSuccessThreshold(2).
Comment thread
ankurs marked this conversation as resolved.
Build()

// Only apply circuit breaking to specific methods
protected := map[string]bool{
"/payment.Service/Charge": true,
"/payment.Service/Refund": true,
}

interceptors.SetDefaultExecutor(func(ctx context.Context, method string, fn func(ctx context.Context) error) error {
if !protected[method] {
return fn(ctx) // passthrough for non-protected methods
}
return failsafe.With[any](cb).WithContext(ctx).Run(func() error {
return fn(ctx)
})
})
Comment thread
ankurs marked this conversation as resolved.
}
Comment thread
ankurs marked this conversation as resolved.
```

### Per-method circuit breakers

If you are using ColdBrew packages in your app, you need to initialise Hystrix Prometheus manually:
Each method can have its own circuit breaker with different limits — sensitive methods trip fast, tolerant methods allow more failures:

```go
import "github.com/go-coldbrew/core"
func init() {
type cbConfig struct {
failureThreshold uint
delay time.Duration
}

func main() {
// SetupHystrixPrometheus registers Hystrix metrics with Prometheus
core.SetupHystrixPrometheus()
configs := map[string]cbConfig{
"/payment.Service/Charge": {failureThreshold: 3, delay: 10 * time.Second},
"/payment.Service/Refund": {failureThreshold: 3, delay: 10 * time.Second},
"/user.Service/GetUser": {failureThreshold: 10, delay: 5 * time.Second},
"/feed.Service/GetFeed": {failureThreshold: 10, delay: 5 * time.Second},
}

var (
mu sync.Mutex
breakers = make(map[string]circuitbreaker.CircuitBreaker[any])
)
Comment thread
ankurs marked this conversation as resolved.

interceptors.SetDefaultExecutor(func(ctx context.Context, method string, fn func(ctx context.Context) error) error {
cfg, ok := configs[method]
if !ok {
return fn(ctx) // no circuit breaker for unconfigured methods
}

mu.Lock()
cb, exists := breakers[method]
if !exists {
cb = circuitbreaker.NewBuilder[any]().
WithFailureThreshold(cfg.failureThreshold).
WithDelay(cfg.delay).
Build()
breakers[method] = cb
}
mu.Unlock()

return failsafe.With[any](cb).WithContext(ctx).Run(func() error {
return fn(ctx)
})
Comment thread
ankurs marked this conversation as resolved.
})
}
```

### Disabling for specific connections

Use `WithoutExecutor()` to skip resilience for internal or loopback connections:

```go
interceptors.DefaultClientInterceptor(
interceptors.WithoutExecutor(),
)
Comment thread
ankurs marked this conversation as resolved.
```

### Per-call executor

`WithExecutor` can be passed directly to any gRPC call to use a specific executor for that call. This works even without calling `SetDefaultExecutor` — you don't need a global executor to use per-call executors. This is useful for per-service circuit breaker tuning.

{: .note .note-info }
Per-call `WithExecutor` requires `ExecutorClientInterceptor` in the interceptor chain. This is included automatically when using `DefaultClientInterceptors`. If you build your own interceptor chain, make sure to include it.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

```go
// Create a sensitive circuit breaker for the payment service
paymentCB := circuitbreaker.NewBuilder[any]().
WithFailureThreshold(3).
WithDelay(10 * time.Second).
Build()

paymentExec := func(ctx context.Context, method string, fn func(ctx context.Context) error) error {
return failsafe.With[any](paymentCB).WithContext(ctx).Run(func() error {
return fn(ctx)
})
}

// Pass it directly to the gRPC call
resp, err := paymentClient.Charge(ctx, req,
interceptors.WithExecutor(paymentExec),
)
```

### Excluded errors

Use `WithExcludedErrors` or `WithExcludedCodes` to prevent expected errors from tripping the circuit breaker. These can also be passed directly to gRPC calls:

```go
Comment thread
ankurs marked this conversation as resolved.
resp, err := client.GetUser(ctx, req,
interceptors.WithExcludedCodes(codes.NotFound, codes.InvalidArgument),
)
```
Comment thread
ankurs marked this conversation as resolved.

See the [interceptors examples](https://github.com/go-coldbrew/interceptors/tree/main/examples) for more patterns including circuit breaker + bulkhead composition.

{: .warning }
**Legacy:** The previous `HystrixClientInterceptor` (wrapping `afex/hystrix-go`) is deprecated and will be removed in v1. Migrate to `SetDefaultExecutor` with [failsafe-go].

Comment thread
ankurs marked this conversation as resolved.
Comment thread
ankurs marked this conversation as resolved.
## Environment Configuration

ColdBrew provides functions to configure the environment and release information used by monitoring tools.
Expand Down Expand Up @@ -320,7 +435,6 @@ To see all the ColdBrew packages, check out the [ColdBrew packages] page.
[Sentry]: https://sentry.io/welcome/
[Opentelemetry]: https://opentelemetry.io/
[Jaeger]: https://www.jaegertracing.io/
[Hystrix-Go]: https://pkg.go.dev/github.com/afex/hystrix-go/hystrix
[Go-grpc-middleware]: https://github.com/grpc-ecosystem/go-grpc-middleware/v2
[Core]: https://github.com/go-coldbrew/core/tree/main#readme
[ColdBrew packages]: /packages
Expand All @@ -337,7 +451,6 @@ To see all the ColdBrew packages, check out the [ColdBrew packages] page.
[OTLPConfig]: https://pkg.go.dev/github.com/go-coldbrew/core#OTLPConfig
[SetupOpenTelemetry]: https://pkg.go.dev/github.com/go-coldbrew/core#SetupOpenTelemetry
[SetupLogger]: https://pkg.go.dev/github.com/go-coldbrew/core#SetupLogger
[SetupHystrixPrometheus]: https://pkg.go.dev/github.com/go-coldbrew/core#SetupHystrixPrometheus
[failsafe-go]: https://github.com/failsafe-go/failsafe-go
[SetupEnvironment]: https://pkg.go.dev/github.com/go-coldbrew/core#SetupEnvironment
[SetupReleaseName]: https://pkg.go.dev/github.com/go-coldbrew/core#SetupReleaseName
Expand Down
4 changes: 2 additions & 2 deletions tests/content.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,10 @@ test.describe("Callouts", () => {
expect(await callout.count()).toBeGreaterThan(0);
});

test("metrics page has warning about hystrix", async ({ page }) => {
test("metrics page has legacy warning about hystrix", async ({ page }) => {
await page.goto("/howto/Metrics/");
const pageText = await page.locator("main, .main-content").first().textContent();
expect(pageText).toContain("unmaintained");
expect(pageText).toContain("SetupHystrixPrometheus");
});
});

Expand Down
Loading