Skip to content

1 Getting started

Jump to bottom
Peter Taoussanis edited this page Aug 28, 2024 · 9 revisions
Telemere logo

Introduction

Telemere is a structured telemetry library and next-generation replacement for Timbre. It helps enable the creation of Clojure/Script systems that are highly observable, robust, and debuggable.

Its key function is to help:

  1. Capture data in your running Clojure/Script programs, and
  2. Facilitate processing of that data into useful information / insight.

[Terminology] Telemetry derives from the Greek tele (remote) and metron (measure). It refers to the collection of in situ (in position) data, for transmission to other systems for monitoring/analysis. Logs are the most common form of software telemetry. So think of telemetry as the superset of logging-like activities that help monitor and understand (software) systems.

Signals

The basic unit of data in Telemere is the signal.

Signals include traditional log messages, structured log messages, and events. Telemere doesn't make a hard distinction between these - they're all just signals with various attributes.

And they're represented by plain Clojure/Script maps with those attributes (keys).

Fundamentally all signals:

  • Occur or are observed at a particular location in your code (file, namespace, line, column).
  • Occur or are observed within a particular program state / context.
  • Convey something of value about that program state / context.

Signals may be independently valuable, valuable in the aggregate (e.g. statistically), or valuable in association with other related signals (e.g. while tracing the flow of some logical activity).

Functionality

The basic tools of Telemere are:

  1. Signal creators to conditionally create signal maps at points in your code.
  2. Signal handlers to conditionally handle those signal maps (analyse, write to console/file/queue/db, etc.).

This is just a generalization of traditional logging which:

  • Conditionally creates message strings at points in your code.
  • Usually dumps those message strings somewhere for future parsing by human eyes or automated tools.

Data types and structures

The parsing of traditional log messages is often expensive, fragile, and lossy. So a key principle of structured logging is to avoid parsing, by instead preserving data types and structures whenever possible.

Telemere embraces this principle by making such preservation natural and convenient.

Noise reduction

Not all data is equally valuable.

Too much low-value data is often actively harmful: expensive to process, to store, and to query. Adding noise just interferes with better data, harming your ability to understand your system.

Telemere embraces this principle by making effective filtering likewise natural and convenient:

Telemere sampling

Telemere uses the term filtering as the superset of both random sampling and other forms of data exclusion/reduction.

Structured telemetry

To conclude- Telemere handles structured and traditional logging, tracing, and basic performance monitoring with a simple unified API that:

  • Preserves data types and structures with rich signals, and
  • Offers effective noise reduction with signal filtering.

Its name is a combination of telemetry and telomere:

Telemetry derives from the Greek tele (remote) and metron (measure). It refers to the collection of in situ (in position) data, for transmission to other systems for monitoring/analysis. Logs are the most common form of software telemetry. So think of telemetry as the superset of logging-like activities that help monitor and understand (software) systems.

Telomere derives from the Greek télos (end) and méros (part). It refers to a genetic feature commonly found at the end of linear chromosomes that helps to protect chromosome integrity.

Setup

Add the relevant dependency to your project:

Leiningen: [com.taoensso/telemere               "x-y-z"] ; or
deps.edn:   com.taoensso/telemere {:mvn/version "x-y-z"}

And setup your namespace imports:

(ns my-app (:require [taoensso.telemere :as t]))

Default config

Telemere is configured sensibly out-the-box.
See section 3-Config for customization.

Default minimum level: :info (signals with lower levels will noop).

Default signal handlers:

Signal handlers process created signals to do something with them (analyse them, write them to console/file/queue/db, etc.)

Platform Condition Handler
Clj Always Console handler that prints signals to *out* or *err*
Cljs Always Console handler that prints signals to the browser console

Default interop:

Telemere can create signals from relevant external API calls, etc.

Platform Condition Signals from
Clj SLF4J API and Telemere SLF4J backend present SLF4J logging calls
Clj tools.logging present and tools-logging->telemere! called tools.logging logging calls
Clj streams->telemere! called Output to System/out and System/err streams

Interop can be tough to get configured correctly so the check-interop util is provided to help verify for tests or debugging:

(check-interop) ; =>
{:tools-logging  {:present? false}
 :slf4j          {:present? true, :telemere-receiving? true, ...}
 :open-telemetry {:present? true, :use-tracer? false, ...}
 :system/out     {:telemere-receiving? false, ...}
 :system/err     {:telemere-receiving? false, ...}}

Usage

Creating signals

Use whichever signal creator is most convenient for your needs:

Name Signal kind Main arg Optional arg Returns
log! :log msg opts/level Signal allowed?
event! :event id opts/level Signal allowed?
error! :error error opts/id Given error
trace! :trace form opts/id Form result
spy! :spy form opts/level Form result
catch->error! :error form opts/id Form value or given fallback
signal! <arb> opts - Depends on opts

Checking signals

Use the with-signal or (advanced) with-signals utils to help test/debug the signals that you're creating:

(t/with-signal
  (t/log!
    {:let  [x "x"]
     :data {:x x}}
    ["My msg:" x]))

;; => {:keys [ns inst data msg_ ...]} ; The signal
  • with-signal will return the last signal created by the given form.
  • with-signals will return all signals created by the given form.

Both have several options, see their docstrings (links above) for details.

Filtering

A signal will be provided to a handler iff ALL of the following are true:

    1. Signal creation is allowed by signal filters:
    • a. Compile time: sample rate, kind, ns, id, level, when form, rate limit
    • b. Runtime: sample rate, kind, ns, id, level, when form, rate limit
    1. Signal handling is allowed by handler filters:
    • a. Compile time: not applicable
    • b. Runtime: sample rate, kind, ns, id, level, when fn, rate limit
    1. Signal middleware (fn [signal]) => ?modified-signal does not return nil
    1. Handler middleware (fn [signal]) => ?modified-signal does not return nil

Quick examples of some basic filtering:

(t/set-min-level! :info) ; Set global minimum level
(t/with-signal (t/event! ::my-id1 :info))  ; => {:keys [inst id ...]}
(t/with-signal (t/event! ::my-id1 :debug)) ; => nil (signal not allowed)

(t/with-min-level :trace ; Override global minimum level
  (t/with-signal (t/event! ::my-id1 :debug))) ; => {:keys [inst id ...]}

;; Disallow all signals in matching namespaces
(t/set-ns-filter! {:disallow "some.nosy.namespace.*"})
  • Filtering is always O(1), except for rate limits which are O(n_windows).
  • See help:filters for more about filtering.
  • See section 2-Architecture for a flowchart / visual aid.

Internal help

Telemere includes extensive internal help docstrings:

Var Help with
help:signal-creators Creating signals
help:signal-options Options when creating signals
help:signal-content Signal content (map given to middleware/handlers)
help:filters Signal filtering and transformation
help:handlers Signal handler management
help:handler-dispatch-options Signal handler dispatch options
help:environmental-config Config via JVM properties, environment variables, or classpath resources
Clone this wiki locally