Skip to content

Fix OTel Collector deployment and Winston log shipping to Axiom - #358

Merged
Asherlc merged 3 commits into
mainfrom
Asherlc/debug-sync-failures
Mar 20, 2026
Merged

Asherlc merged 3 commits into
mainfrom
Asherlc/debug-sync-failures

Conversation

@Asherlc

@Asherlc Asherlc commented Mar 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • Restore OTel Collector — the collector service, its config, and OTEL env vars were never deployed to the server (stale docker-compose.yml). Fixed by running collector as root (user: "0:0") so it can read Docker container logs, and targeting the existing dofek-logs Axiom dataset (ingest token is scoped).
  • Fix OTLP log export — the protobuf log exporter doesn't resolve /v1/logs from the generic OTEL_EXPORTER_OTLP_ENDPOINT alone. Added explicit OTEL_EXPORTER_OTLP_LOGS_ENDPOINT to all containers.
  • Replace broken WinstonInstrumentation with direct transport — WinstonInstrumentation relies on CJS require hooks that can't patch ESM imports ("type": "module"). Switched to @opentelemetry/winston-transport which adds an OTel transport directly to the Winston logger, loaded lazily only when OTLP is configured.
  • Add ESM instrumentation hook to entrypoint for future OTel instrumentation compatibility.

Test plan

  • Verified Docker container logs flow to Axiom via collector filelog receiver
  • Verified OTLP log records reach Axiom via collector OTLP receiver
  • Unit tests pass (src/instrumentation.test.ts updated for removed WinstonInstrumentation)
  • After Watchtower deploys new image, verify Winston app logs (sync errors, provider failures) appear in Axiom

🤖 Generated with Claude Code

Asherlc and others added 3 commits March 20, 2026 09:29
The OTel Collector was never deployed to production (stale docker-compose.yml
on server). Three issues fixed:
- Add user: "0:0" to collector so it can read Docker container logs
- Target existing dofek-logs dataset (token is scoped, can't create new ones)
- Added AXIOM_API_TOKEN to server .env for collector auth

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…umentation

WinstonInstrumentation doesn't work in ESM apps (type: module) because
it relies on CJS require hooks that can't patch ESM imports. Replace it
with @opentelemetry/winston-transport which adds an OTel transport
directly to the Winston logger — no module hooking needed.

Also:
- Add explicit OTEL_EXPORTER_OTLP_LOGS_ENDPOINT to docker-compose
  (proto exporter doesn't resolve from generic endpoint alone)
- Add ESM hook to entrypoint.sh for future instrumentation compatibility
- Update instrumentation tests for removed WinstonInstrumentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Knip correctly flagged this as unused after switching to
@opentelemetry/winston-transport.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asherlc
Asherlc merged commit 0034106 into main Mar 20, 2026
10 checks passed
@Asherlc
Asherlc deleted the Asherlc/debug-sync-failures branch March 20, 2026 17:35
Asherlc added a commit that referenced this pull request Mar 31, 2026
* fix: restore OTel Collector and fix log shipping to Axiom

The OTel Collector was never deployed to production (stale docker-compose.yml
on server). Three issues fixed:
- Add user: "0:0" to collector so it can read Docker container logs
- Target existing dofek-logs dataset (token is scoped, can't create new ones)
- Added AXIOM_API_TOKEN to server .env for collector auth

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: bridge Winston to OTel via transport instead of broken ESM instrumentation

WinstonInstrumentation doesn't work in ESM apps (type: module) because
it relies on CJS require hooks that can't patch ESM imports. Replace it
with @opentelemetry/winston-transport which adds an OTel transport
directly to the Winston logger — no module hooking needed.

Also:
- Add explicit OTEL_EXPORTER_OTLP_LOGS_ENDPOINT to docker-compose
  (proto exporter doesn't resolve from generic endpoint alone)
- Add ESM hook to entrypoint.sh for future instrumentation compatibility
- Update instrumentation tests for removed WinstonInstrumentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove unused @opentelemetry/instrumentation-winston dependency

Knip correctly flagged this as unused after switching to
@opentelemetry/winston-transport.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant