IPFIXcol framework

⚠️ IPFIXcol2 has been released!

The next generation of the collector is more stable, up to 2x faster, and adds support for new features (e.g. biflow, structured data types, etc.). The code was completely rewritten and some plugins might not be available.

Since the release of the new collector, this old framework is not supported anymore! Please, consider upgrading to the new release.

Table of Contents

1. Framework description
2. IPFIXcol
3. Plugins
   • External input plugins
   • External intermediate plugins
   • External storage plugins
4. Built-in tools
5. External tools
   • fbitconvert
   • fbitdump
   • fbitexpire
   • fbitmerge
   • profilesdaemon
6. Howto install
7. Howto build
8. Docker
9. RPM
10. FastBit
11. Contact us * Reporting bugs * Forum

Framework description

IPFIXcol framework is a set of:

• IPFIXcol - collector for capturing IPFIX NetFlow data
• input, intermediate and storage plugins for collector
• tools for data processing etc.

To generate data for the IPFIXcol, look at the list list of supported flow exporters.

IPFIXcol

Described in it's README

Plugins

IPFIX collector comes with several built-in plugins described at IPFIXcol's page.

There are also external plugins that are installed separately

External input plugins

• nfdump - NFDUMP file reader

External intermediate plugins

• geoip - adds country codes into the metadata structure
• profiler - fills metadata informations about profiles and channels
• profile_stats - counts statistic per profile and channel
• stats - counts statistics per ODID
• uid - fills user identity information

External storage plugins

• fastbit - uses FastBit library to store and index data
• fastbit_compression - uses FastBit library to store and index data with optional compression support
• json - converts data into JSON format
• nfdump - stores data in NFDUMP file format
• postgres - stores data into PostgreSQL database
• statistics - uses RRD library to generate statistics for collected data
• unirec - stores data in UniRec format

Built-in tools

ipfixviewer and ipfixconf

Destribed in IPFIXcol's README

External tools

fbitconvert

Converts data from NFDUMP file format into FastBit. Uses IPFIXcol, nfdump input plugin and fastbit storage plugin.

More info in it's README

fbitdump

Tool for manipulating IPFIX data in FastBit database format. It uses FastBit library to read and index data.

More info in it's README

fbitexpire

Daemon for removing old data.

More info in it's README

fbitmerge

Tool for merging FastBit data (saves free disk space, reduces number of files..)

More info in it's README

profilesdaemon

Tool for profiles management and distribution

More info

How to install

Individual packages of the IPFIXcol framework can be installed from Fedora copr repository Just add the repository to your system:

dnf copr enable @CESNET/IPFIXcol 

And install the packages you need (e.g. IPFIXcol framework and JSON output plugin):

dnf install ipfixcol ipfixcol-json-output

If you not are using one of the supported operating systems, you can build the IPFIXcol from sources.

How to build

Dependencies must be installed first. For Fedora, CentOS and RHEL the list of necessary packages is as follows:

autoconf bison docbook-style-xsl doxygen flex 
gcc gcc-c++ git libtool libxml2 libxml2-devel 
libxslt lksctp-tools-devel lzo-devel make 
openssl-devel GeoIP-devel rrdtool-devel
sqlite-devel postgresql-devel corosync corosync-devel rpm-build

Debian and Ubuntu distributions have a different names for some of the packages:

autoconf bison build-essential docbook-xsl doxygen flex
git liblzo2-dev libtool libsctp-dev libssl-dev libxml2
libxml2-dev pkg-config xsltproc libgeoip-dev librrd-dev
libsqlite3-dev libpq-dev libcpg-dev corosync-dev

IPFIXcol does not support openssl1.1, therefore you need to use libssl1.0-dev on Debian Jessie.

Moreover, you need to build the FastBit library

First, download IPFIXcol git repository (do NOT forget to use --recursive option):

git clone --recursive https://github.com/CESNET/ipfixcol.git

Note: If you have updated from a previous revision of the repository without a submodule or if you forgot to add --recursive option, you can just download the submodule manually:

git submodule update --init --recursive

After installing all dependencies and downloading the repository, the whole framework can be build at once with

autoreconf -i 

to generate configure script from configure.ac, Makefile.in from Makefile.am and install missing files.

./configure

to configure packages in subdirectories and generate Makefiles.

make
sudo make install
sudo ldconfig

to build and install all projects. The call to ldconfig is necessary to update the cache of fynamic linker to find dependency library.

Or you can build each part (collector, tool(s), external plugin(s)) separately.

The projects that depend on ipfixcol headers check the reltive path to base/header directory to use headers. When project is separated from the structure, it needs to have the headers installed (ipfixcol-devel package).

Docker

IPFIXcol can be used with Docker. See Docker howto.

Ansible

IPFIXcol can also be installed using Ansible orchestration. See Ansible howto.

RPM

Each part of framework supports building rpm packages by running

make rpm

RPMs can be build only for specific parts, not the whole project.

FastBit

Plugins and tools that uses FastBit file format need FasBit library installed. IPFIXcol framework uses it's own fork of FastBit library to keep compatibility.

IPFIXcol's FastBit library can be found here.

Contact us

Reporting bugs

If you find any bug you can report it into issue tracker here on GitHub.

Contributing to IPFIXcol

We are open to contributions to IPFIXcol which improve the stability and functionality of the collector. To keep the code readable and consistent, please adhere to the coding style document.

Forum

if you have some questions or if you just want to share your ideas about useful features etc., please use this forum.

Back to Top