Archived Now bundled in https://github.com/cloudfoundry/loggregator-release
Loggregator is the logging system used in CloudFoundry.
- Real time streaming of logs
- Producers do not experience backpressure
- Logs can be routed to several consumers
- Elastic and horizontal scalability
- High message reliability
- Low latency
- Flexibile consumption
- Security via gRPC with mutual TLS
- Opinionated log structure
- Logs are ephemeral/non-replicated within system
- No guaranteed delivery
- Limited persistence/querying ability
Loggregator uses Google's protocol buffers along with gRPC to deliver logs. Loggregator has defined (via Loggregator API) log types that are contained in a single protocol buffer type, named an envelope. Those types are:
- Log
- Counter
- Gauge
- Timer
- Event
The Loggregator system as a whole does not have an opinion about how frequently any log type is emitted. However, there are recommendations. Counter and Gauge logs should be emitted regularly (e.g., once a minute) so downstream consumers can easily plot the increasing total.
Log, Timer and Event should be emitted when the corresponding action occurred and therefore should be treated as asynchronous data. These data types do not lend themselves to be plotted as a time series.
Loggregator is made up of 4 components:
- Agent
- Router
- ReverseLogProxy (RLP)
- TrafficController (TC)
The Agent is a daemon process that is intended to run on each container/VM. It is the entry point into Loggregator. Any log that is written to Loggregator is written to the Agent.
The Router takes each log and publishes it to any interested consumer. Every log is in flight, and not available for a consumer if the consumer was late to connect. It routes envelopes via the pubsub library that allows a consumer to give the Router "selectors" to whitelist what logs it wants. A consumer does not connect directly to a router.
The ReverseLogProxy (RLP) has gRPC endpoints to allow consumers to read data from Loggregator. Each RLP connects to every Router to ensure it gets all the relevant logs. Each Router only gets a subset of the logs, and therefore without the RLP, a consumer would only get a fraction of the logs. The RLP only speaks V2.
The ReverseLogProxy (RLP) Gateway exposes the RLP API over HTTP.
The API Documentation can be found here
The TrafficController (TC) is like the RLP, but is tuned for CloudFoundry. It authenticates via the UAA and CloudController, which are both CF components. It egresses logs via Websockets. It only speaks V1. Planned deprecation in 2018.
Loggregator chooses to drop logs instead of pushing back on producers. It does so with the diodes library. Therefore if anything upstream slows down, the diode will drop older messages. This allows producers to be resilient to upstream problems.
There is Go client library: go-loggregator. The client library has several useful patterns along with examples to interact with a Loggregator system.
The most common way to deploy Loggregator is via Bosh. There is a Loggregator Bosh release that is used heavily in a plethora of production environments. It is battle tested and well maintained.