diff --git a/SUMMARY.md b/SUMMARY.md index de9f63829..36c4e9a95 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -230,6 +230,7 @@ ## Fluent Bit for developers * [C library API](development/library_api.md) +* [External libraries](development/external-libraries.md) * [Ingest records manually](development/ingest-records-manually.md) * [Golang output plugins](development/golang-output-plugins.md) * [MessagePack format](development/msgpack-format.md) diff --git a/development/developer-guide.md b/development/developer-guide.md index d8151d370..01593ce6a 100644 --- a/development/developer-guide.md +++ b/development/developer-guide.md @@ -6,6 +6,8 @@ If you have some knowledge of C, this guide should help you understand how to ma Most external libraries are embedded in the project in the [/lib](https://github.com/fluent/fluent-bit/tree/master/lib) folder. To keep its footprint low and maximize compatibility in cross-platform builds, Fluent Bit attempts to keep its dependency graph small. +For a complete list of embedded libraries and their purposes, see [External libraries](external-libraries.md). + The external library that you're mostly likely to interact with is [MessagePack](https://github.com/msgpack/msgpack-c). For cryptographic support, Fluent Bit uses the system installed version of OpenSSL. You must install OpenSSL libraries and headers before Building Fluent Bit. diff --git a/development/external-libraries.md b/development/external-libraries.md new file mode 100644 index 000000000..505d6eb19 --- /dev/null +++ b/development/external-libraries.md @@ -0,0 +1,103 @@ +--- +description: External libraries embedded in Fluent Bit +--- + +# External libraries + +Fluent Bit embeds several external libraries in its [/lib](https://github.com/fluent/fluent-bit/tree/master/lib) directory. These libraries are included to minimize external dependencies and maximize cross-platform compatibility. + +## Core fluent libraries + +These libraries are developed and maintained by the Fluent Bit team: + +| Library | Purpose | License | +|---------|---------|---------| +| [`CFL`](https://github.com/fluent/cfl) | Common data structures (strings, lists, key-value pairs, arrays, variants, time utilities, hashing) used across Fluent Bit and related projects. | Apache 2.0 | +| [`Chunk I/O`](https://github.com/fluent/chunkio) | Manages chunks of data in the file system for buffering. Handles memory-mapped I/O with optional CRC32 checksums for data integrity. Powers filesystem buffering for Fluent Bit. | Apache 2.0 | +| [`CMetrics`](https://github.com/fluent/cmetrics) | Creates and manages metrics (counters, gauges, histograms, summaries) with label support. Based on Go Prometheus Client API design. | Apache 2.0 | +| [`CTraces`](https://github.com/fluent/ctraces) | Creates and manages trace contexts with encoding/decoding support for OpenTelemetry and other formats. | Apache 2.0 | +| [`CProfiles`](https://github.com/fluent/cprofiles) | Creates and manages profiling data based on the OpenTelemetry Profiles schema. | Apache 2.0 | +| [`Fluent OTel Proto`](https://github.com/fluent/fluent-otel-proto) | Provides C interfaces for OpenTelemetry protocol buffer definitions (common, resource, trace, logs, metrics). | Apache 2.0 | +| [`flb_libco`](https://github.com/fluent/flb_libco) | Fork of [libco](https://byuu.org/library/libco/) for cooperative multithreading (coroutines). Includes ARMv8 fixes and macOS support. | Internet Systems Consortium | + +## HTTP and networking + +| Library | Purpose | License | +|---------|---------|---------| +| [`Monkey`](https://github.com/monkey/monkey) | Fast, lightweight HTTP server providing the embedded web server HTTP endpoints (health checks, metrics, and so on) for Fluent Bit. | Apache 2.0 | +| [`c-ares`](https://c-ares.org/) | Modern asynchronous DNS resolver library. Enables non-blocking DNS queries for network operations. | Massachusetts Institute of Technology (MIT) | +| [`nghttp2`](https://nghttp2.org/) | HTTP/2 C library implementing the framing layer. Enables HTTP/2 client and server capabilities. | Massachusetts Institute of Technology (MIT) | + +## Data serialization and parsing + +| Library | Purpose | License | +|---------|---------|---------| +| [`msgpack-c`](https://github.com/msgpack/msgpack-c) | MessagePack serialization library. Used for internal data format for records in Fluent Bit. | Boost 1.0 | +| [`Jansson`](https://github.com/akheron/jansson) | JSON encoding, decoding, and manipulation with full Unicode support. | Massachusetts Institute of Technology (MIT) | +| [`yyjson`](https://github.com/ibireme/yyjson) | High-performance JSON library capable of processing gigabytes per second. Used for fast JSON operations. | Massachusetts Institute of Technology (MIT) | +| [`jsmn`](https://github.com/zserge/jsmn) | Minimalistic JSON parser (~200 lines of code). Zero-copy, no dynamic allocation. | Massachusetts Institute of Technology (MIT) | + +## Text processing + +| Library | Purpose | License | +|---------|---------|---------| +| [`Onigmo`](https://github.com/k-takata/Onigmo) | Regular expression library (fork of Oniguruma). Supports Perl 5.10+ expressions like `\K`, `\R`, and conditional patterns. Default regular expression engine for Ruby 2.0+. | BSD | +| [`tutf8e`](https://github.com/papplampe/tutf8e) | Tiny UTF-8 encoder supporting ISO-8859-x and Windows-125x character set conversions. | Massachusetts Institute of Technology (MIT) | +| [`simdutf`](https://github.com/simdutf/simdutf) | SIMD-accelerated Unicode validation and transcoding (UTF-8, UTF-16, UTF-32). | Apache 2.0 | + +## Compression + +| Library | Purpose | License | +|---------|---------|---------| +| [`miniz`](https://github.com/richgel999/miniz) | Single-file deflate/inflate, zlib-subset implementation. Provides gzip compression. | Public Domain | +| [`Snappy`](https://github.com/google/snappy) | Fast compression/decompression library. Prioritizes speed over compression ratio. | BSD | +| [`zstd`](https://github.com/facebook/zstd) | Zstandard compression providing high compression ratios at fast speeds. | BSD | + +## Scripting and webassembly + +| Library | Purpose | License | +|---------|---------|---------| +| [`LuaJIT`](https://luajit.org/) | `Just-in-time (JIT)` compiler for Lua. Powers the [Lua filter](../pipeline/filters/lua.md) plugin for custom record processing. | Massachusetts Institute of Technology (MIT) | +| [`WAMR`](https://github.com/bytecodealliance/wasm-micro-runtime) | WebAssembly Micro Runtime from the Bytecode Alliance. Enables [WASM filter plugins](wasm-filter-plugins.md) and [WASM input plugins](wasm-input-plugins.md). Supports `interpreter`, `AOT`, and `JIT` execution modes. | Apache 2.0 | + +## Integration libraries + +| Library | Purpose | License | +|---------|---------|---------| +| [`librdkafka`](https://github.com/confluentinc/librdkafka) | Apache Kafka C/C++ client library. Enables the [Kafka input](../pipeline/inputs/kafka.md) and [Kafka output](../pipeline/outputs/kafka.md) plugins. Supports `producer`, `consumer`, and `admin` clients with high performance (millions of messages/second). | BSD-2 | +| [`Avro`](https://github.com/apache/avro) | Apache Avro data serialization system. | Apache 2.0 | + +## Storage and utilities + +| Library | Purpose | License | +|---------|---------|---------| +| [`SQLite`](https://sqlite.org/) | Self-contained SQL database engine. Used for persistent storage operations. | Public Domain | +| [`jemalloc`](https://github.com/jemalloc/jemalloc) | General-purpose memory allocator emphasizing fragmentation avoidance and scalable concurrency. Optional alternative to system malloc. | BSD-2 | +| [`lwrb`](https://github.com/MaJerle/lwrb) | Lightweight ring buffer implementation. Thread-safe FIFO buffer with zero-copy DMA support. | Massachusetts Institute of Technology (MIT) | +| [`mpack`](https://github.com/ludocode/mpack) | Simple and fast MessagePack encoder/decoder. | Massachusetts Institute of Technology (MIT) | +| [`libbacktrace`](https://github.com/ianlancetaylor/libbacktrace) | Library for producing symbolic back traces. Used for debugging and error reporting. | BSD | + +## Version information + +When Fluent Bit starts, it logs the versions of key libraries. For example, as of Fluent Bit 4.2: + +```text +[2025/01/15 10:30:00] [ info] [chunkio ] version=1.5.4 +[2025/01/15 10:30:00] [ info] [ctraces ] version=1.0.5 +[2025/01/15 10:30:00] [ info] [monkey ] version=1.8.5 +[2025/01/15 10:30:00] [ info] [wamr ] version=2.4.1 +``` + +## Build options + +Some libraries are optional and can be enabled or disabled at build time: + +| CMake option | Library | Default | +|--------------|---------|---------| +| `FLB_WAMRC` | `WAMR` `AOT` compiler | Off | +| `FLB_LUAJIT` | `LuaJIT` | On | +| `FLB_OUT_KAFKA` | `librdkafka` | Off | +| `FLB_JEMALLOC` | `jemalloc` | Off (platform dependent) | + +For more information on build options, see [Build and install](../installation/downloads/source/build-and-install.md). +