Code

Initial Kafka support#13190

Code

Merged

reyesj2 merged 148 commits into2.4/devSecurity-Onion-Solutions/securityonion:2.4/devfrom reyesj2/kafkaSecurity-Onion-Solutions/securityonion:reyesj2/kafkaCopy head branch name to clipboardJun 13, 2024

+1,658-73

Commits

Commits on Nov 29, 2023

• Kafka init

  Show description for 86dc7cc
  reyesj2

  committed
  86dc7cc

  Copy full SHA for 86dc7cc

• Update to merge in 2.4/dev

  Show description for 8cf2968
  reyesj2

  committed
  8cf2968

  Copy full SHA for 8cf2968

Commits on Mar 25, 2024

• merge 2.4/dev

  Show description for 446f1ff
  reyesj2

  committed
  446f1ff

  Copy full SHA for 446f1ff

Commits on Apr 2, 2024

• Removed duplicate kafka setup

  Show description for 26abe90
  reyesj2

  committed
  26abe90

  Copy full SHA for 26abe90

• Merge remote-tracking branch 'origin/reyesj2/kafka' into kaffytaffy

  m0duspwnens

  committed
  e25bc8e

  Copy full SHA for e25bc8e

• add kafka to manager nodes

  m0duspwnens

  committed
  780ad9e

  Copy full SHA for 780ad9e

• make manager download so-kafka container

  m0duspwnens

  committed
  f7534a0

  Copy full SHA for f7534a0

• Fix kafka keystore script

  Show description for 1b49c85
  reyesj2

  committed
  1b49c85

  Copy full SHA for 1b49c85

• Update kafka to use manager docker registry

  Show description for b032eed
  reyesj2

  committed
  b032eed

  Copy full SHA for b032eed

• CRLF -> LF

  Show description for 643d483
  reyesj2

  committed
  643d483

  Copy full SHA for 643d483

• Fix kafka storage setup

  Show description for 7f5741c
  reyesj2

  committed
  7f5741c

  Copy full SHA for 7f5741c

• Fix typos and fix error related to elasticsearch saltstate being called from logstash state. Logstash will be removed from kafkanodes in future

  Show description for 82830c8
  reyesj2

  committed
  82830c8

  Copy full SHA for 82830c8

• Merge remote-tracking branch 'origin/reyesj2/kafka' into kaffytaffy

  m0duspwnens

  committed
  c9bf1c8

  Copy full SHA for c9bf1c8

• put alphabetical

  m0duspwnens

  committed
  4e142e0

  Copy full SHA for 4e142e0

Commits on Apr 3, 2024

• add so-manager to kafka.nodes pillar

  m0duspwnens

  committed
  639bf05

  Copy full SHA for 639bf05

• kafka.nodes pillar to lf

  m0duspwnens

  committed
  8e47cc7

  Copy full SHA for 8e47cc7

• listen on 0.0.0.0 for CONTROLLER

  m0duspwnens

  committed
  db106f8

  Copy full SHA for db106f8

• add kafka nodes as extra hosts for the container

  m0duspwnens

  committed
  c3f02a6

  Copy full SHA for c3f02a6

• allow sensor and searchnode to connect to manager kafka ports

  m0duspwnens

  committed
  ed6137a

  Copy full SHA for ed6137a

• port 9093 for kafka docker

  m0duspwnens

  committed
  18f95e8

  Copy full SHA for 18f95e8

• kafka broker and listener on 0.0.0.0

  m0duspwnens

  committed
  b863060

  Copy full SHA for b863060

• Set kafka heap size. To be later configured from SOC

  Show description for dc27bbb
  reyesj2

  committed
  dc27bbb

  Copy full SHA for dc27bbb

• Generate certs for use with elasticfleet kafka output policy

  Show description for 13105c4
  reyesj2

  committed
  13105c4

  Copy full SHA for 13105c4

• allow manager to manager on kafka ports

  m0duspwnens

  committed
  1b8584d

  Copy full SHA for 1b8584d

Commits on Apr 4, 2024

• add so-kafka to manager for firewall

  m0duspwnens

  committed
  7a6b72e

  Copy full SHA for 7a6b72e

• restart container on server config changes

  m0duspwnens

  committed
  4c5b42b

  Copy full SHA for 4c5b42b

• Generate kafka keystore on changes to kafka.key

  Show description for 40b08d7
  reyesj2

  committed
  40b08d7

  Copy full SHA for 40b08d7

• Add kafka signing_policy for client/server auth. Add kafka-client cert on manager so manager can interact with kafka using its own cert

  Show description for 436cbc1
  reyesj2

  committed
  436cbc1

  Copy full SHA for 436cbc1

• Merge remote-tracking branch 'origin/kaffytaffy' into reyesj2/kafka

  reyesj2

  committed
  6202090

  Copy full SHA for 6202090

• Autogenerate kafka topics when a message it sent to non-existing topic

  Show description for 735cfb4
  reyesj2

  committed
  735cfb4

  Copy full SHA for 735cfb4

Commits on Apr 5, 2024

• Generate kafka cluster id if it doesn't exist

  Show description for 433309e
  reyesj2

  committed
  433309e

  Copy full SHA for 433309e

• initial logstash input from kafka over ssl

  Show description for 721e04f
  reyesj2

  committed
  721e04f

  Copy full SHA for 721e04f

• Add client_id to logstash pipeline. To identify which searchnode is pulling messages

  Show description for 65274e8
  reyesj2

  committed
  65274e8

  Copy full SHA for 65274e8

Commits on Apr 8, 2024

• Remove logstash output to kafka pipeline. Add additional topics for searchnodes to ingest and add partition/offset info to event

  Show description for d67ebab
  reyesj2

  committed
  d67ebab

  Copy full SHA for d67ebab

Commits on Apr 9, 2024

• add defaults and jijafy kafka config

  m0duspwnens

  committed
  6217a7b

  Copy full SHA for 6217a7b

• fix var

  m0duspwnens

  committed
  7aa00fa

  Copy full SHA for 7aa00fa

• fix dict update

  m0duspwnens

  committed
  c48436c

  Copy full SHA for c48436c

• items not keys in for loop

  m0duspwnens

  committed
  daa5342

  Copy full SHA for daa5342

• fix client and server properties formatting

  m0duspwnens

  committed
  d38051e

  Copy full SHA for d38051e

• jinja config files

  m0duspwnens

  committed
  bd5fe43

  Copy full SHA for bd5fe43

• Merge remote-tracking branch 'origin/reyesj2/kafka' into kaffytaffy

  m0duspwnens

  committed
  fa7f810

  Copy full SHA for fa7f810

Commits on Apr 10, 2024

• annotations and enable/disable from ui

  m0duspwnens

  committed
  86b9840

  Copy full SHA for 86b9840

• Merge remote-tracking branch 'origin/2.4/dev' into kaffytaffy

  m0duspwnens

  committed
  e9e61ea

  Copy full SHA for e9e61ea

• disable logstash and redis if kafka enabled

  m0duspwnens

  committed
  d3bd56b

  Copy full SHA for d3bd56b

Commits on Apr 11, 2024

• Remove references to kafkanode

  Show description for af53dcd
  reyesj2

  committed
  af53dcd

  Copy full SHA for af53dcd

• Run kafka-clusterid script when pillar values are missing

  Show description for ca7253a
  reyesj2

  committed
  ca7253a

  Copy full SHA for ca7253a

• Update annotation for global.pipeline

  Show description for 6b28dc7
  reyesj2

  committed
  6b28dc7

  Copy full SHA for 6b28dc7

• Use global.pipeline for redis / kafka states

  Show description for 3955587
  reyesj2

  committed
  3955587

  Copy full SHA for 3955587

• add kafka to receiver

  m0duspwnens

  committed
  f514e5e

  Copy full SHA for f514e5e

Commits on Apr 12, 2024

• move kafka_cluster_id to kafka:cluster_id

  m0duspwnens

  committed
  a54a72c

  Copy full SHA for a54a72c

• create kratos local pillar dirs during setup

  m0duspwnens

  committed
  0ed9894

  Copy full SHA for 0ed9894

• Make global.pipeline use GLOBALMERGED value

  Show description for fbd3cff
  reyesj2

  committed
  fbd3cff

  Copy full SHA for fbd3cff

• Merge kaffytaffy

  Show description for af29ae1
  reyesj2

  committed
  af29ae1

  Copy full SHA for af29ae1

• add receiver managersearch and standalone to kafka.nodes pillar

  m0duspwnens

  committed
  04ddcd5

  Copy full SHA for 04ddcd5

• order kafka pillar assignment

  m0duspwnens

  committed
  d73ba7d

  Copy full SHA for d73ba7d

• Note to remove so-kafka-clusterid. Update soup and setup to generate needed kafka pillar values

  Show description for a6ff92b
  reyesj2

  committed
  a6ff92b

  Copy full SHA for a6ff92b

• Typo

  Show description for 911ee57
  reyesj2

  committed
  911ee57

  Copy full SHA for 911ee57

• Merge kaffytaffy

  Show description for fcfbb1e
  reyesj2

  committed
  fcfbb1e

  Copy full SHA for fcfbb1e

• need /opt/so/conf/ca/cacerts on receiver for kafka to run

  m0duspwnens

  committed
  c014508

  Copy full SHA for c014508

• just broker as default process

  m0duspwnens

  committed
  bb983d4

  Copy full SHA for bb983d4

• update default process.role to broker only

  Show description for de6ea29
  reyesj2

  committed
  de6ea29

  Copy full SHA for de6ea29

• Merge remote-tracking branch 'remotes/origin/kaffytaffy' into reyesj2/kafka

  reyesj2

  committed
  eedea2c

  Copy full SHA for eedea2c

Commits on Apr 15, 2024

• restart salt minion if a manager and signing policies change

  m0duspwnens

  committed
  c4994a2

  Copy full SHA for c4994a2

Commits on Apr 16, 2024

• watch pillar files for changes and do something

  m0duspwnens

  committed
  4b79623

  Copy full SHA for 4b79623

Commits on Apr 17, 2024

• Update Kafka nodeid

  Show description for 665b719
  reyesj2

  committed
  665b719

  Copy full SHA for 665b719

• watch a pillar in files and take action

  m0duspwnens

  committed
  4caa6a1

  Copy full SHA for 4caa6a1

Commits on Apr 18, 2024

• more comments, better logging

  m0duspwnens

  committed
  506bbd3

  Copy full SHA for 506bbd3

• improve it

  m0duspwnens

  committed
  610dd2c

  Copy full SHA for 610dd2c

• Update map files to handle empty kafka:nodes pillar

  Show description for 5cc358d
  reyesj2

  committed
  5cc358d

  Copy full SHA for 5cc358d

• match keys better. go through files reverse first found is prio

  m0duspwnens

  committed
  1f6eb9c

  Copy full SHA for 1f6eb9c

• Variables no longer used. Replaced by map file

  Show description for fe81ffa
  reyesj2

  committed
  fe81ffa

  Copy full SHA for fe81ffa

• update so-kafka-clusterid

  Show description for 746128e
  reyesj2

  committed
  746128e

  Copy full SHA for 746128e

• add kafkapass soc annotation

  Show description for 4ac04a1
  reyesj2

  committed
  4ac04a1

  Copy full SHA for 4ac04a1

Commits on Apr 19, 2024

• logging changes. ensure salt master has pillarWatch engine

  m0duspwnens

  committed
  6c5e057

  Copy full SHA for 6c5e057

Commits on Apr 22, 2024

• Setup kafka reactor for managing kafka controllers globally

  Show description for 25d63f7
  reyesj2

  committed
  25d63f7

  Copy full SHA for 25d63f7

• Update kafka process_x_roles annotation

  Show description for 5a401af
  reyesj2

  committed
  5a401af

  Copy full SHA for 5a401af

• Merge remote-tracking branch 'remotes/origin/kaffytaffy' into reyesj2/kafka

  reyesj2

  committed
  685b80e

  Copy full SHA for 685b80e

Commits on Apr 23, 2024

• Update kafka managed node pillar template to include its process.role

  Show description for aa0c589
  reyesj2

  committed
  aa0c589

  Copy full SHA for aa0c589

• Update kafka cert permissions

  Show description for 36573d6
  reyesj2

  committed
  36573d6

  Copy full SHA for 36573d6

Commits on Apr 29, 2024

• Set kafka.nodes state to run first to populate kafka.nodes pillar

  Show description for 29c964c
  reyesj2

  committed
  29c964c

  Copy full SHA for 29c964c

• Split kafka defaults between broker / controller

  Show description for 086ebe1
  reyesj2

  committed
  086ebe1

  Copy full SHA for 086ebe1

• Remove salt reactor for Kafka

  Show description for 529c8d7
  reyesj2

  committed
  529c8d7

  Copy full SHA for 529c8d7

• Use SOC UI to configure list of KRaft (Kafka) controllers for cluster

  Show description for fd9a914
  reyesj2

  committed
  fd9a914

  Copy full SHA for fd9a914

• Rename kafkapass -> kafka_pass

  Show description for 11055b1
  reyesj2

  committed
  11055b1

  Copy full SHA for 11055b1

• Add missing configuration for nodes running Kafka broker role only

  Show description for 529bc01
  reyesj2

  committed
  529bc01

  Copy full SHA for 529bc01

• Add Kafka connectivity between manager - > receiver nodes.

  Show description for a6e8b25
  reyesj2

  committed
  a6e8b25

  Copy full SHA for a6e8b25

Commits on Apr 30, 2024

• Add Kafka output to elastic-fleet setup. Includes separating topics by event.module with fallback to default-logs if no event.module is specified or doesn't match processors

  Show description for 9c83a52
  reyesj2

  committed
  9c83a52

  Copy full SHA for 9c83a52

• Add id to grid-kafka fleet output policy

  Show description for fcc4050
  reyesj2

  committed
  fcc4050

  Copy full SHA for fcc4050

• Setup elastic fleet rollover from logstash -> kafka output policy

  Show description for bb49944
  reyesj2

  committed
  bb49944

  Copy full SHA for bb49944

Commits on May 1, 2024

• Dynamically create Kafka topics based on event.module from elastic agent logs eg. zeek-topic. Depends on Kafka brokers having auto.create.topics.enable set to true

  Show description for cef9bb1
  reyesj2

  committed
  cef9bb1

  Copy full SHA for cef9bb1

• Update soup for Kafka

  Show description for eb12496
  reyesj2

  committed
  eb12496

  Copy full SHA for eb12496

• soup typo

  Show description for 63f3e23
  reyesj2

  committed
  63f3e23

  Copy full SHA for 63f3e23

• Make kafka configuration changes prior to 2.4.70 upgrade

  Show description for 6b60e85
  reyesj2

  committed
  6b60e85

  Copy full SHA for 6b60e85

• Remove check for existing value since Kafka pillar is made empty on upgrade

  Show description for 84abfa6
  reyesj2

  committed
  84abfa6

  Copy full SHA for 84abfa6

• Write out Kafka pillar path

  Show description for de0af58
  reyesj2

  committed
  de0af58

  Copy full SHA for de0af58

• Reconfigure logstash Kafka input

  Show description for 3efdb4e
  reyesj2

  committed
  3efdb4e

  Copy full SHA for 3efdb4e

• Generate different Kafka certs for different SO nodetypes

  Show description for e164d15
  reyesj2

  committed
  e164d15

  Copy full SHA for e164d15

• Add kafka state to standalone highstate

  Show description for 58ebbfb
  reyesj2

  committed
  58ebbfb

  Copy full SHA for 58ebbfb

• Create new Kafka output policy using salt

  Show description for 47ced60
  reyesj2

  committed
  47ced60

  Copy full SHA for 47ced60

Commits on May 2, 2024

• Automatically switch between Kafka output policy and logstash output policy when globals.pipeline changes

  Show description for de9f642
  reyesj2

  committed
  de9f642

  Copy full SHA for de9f642

• Setup Kafka to use PKCS12 and remove need for converting to JKS

  Show description for f663ef8
  reyesj2

  committed
  f663ef8

  Copy full SHA for f663ef8

• Reconfigure Kafka defaults

  Show description for 093cbc5
  reyesj2

  committed
  093cbc5

  Copy full SHA for 093cbc5

• Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka

  reyesj2

  committed
  e960ae6

  Copy full SHA for e960ae6

Commits on May 8, 2024

• Add basic read-only metric collection from Kafka

  Show description for dff609d
  reyesj2

  committed
  dff609d

  Copy full SHA for dff609d

• Logstash consumer threads should match topic partition count

  Show description for eca2a4a
  reyesj2

  committed
  eca2a4a

  Copy full SHA for eca2a4a

• merge 2.4/dev

  Show description for 2ad87bf
  reyesj2

  committed
  2ad87bf

  Copy full SHA for 2ad87bf

Commits on May 22, 2024

• Set default replication factor back to Kafka default

  Show description for 91f8b1f
  reyesj2

  committed
  91f8b1f

  Copy full SHA for 91f8b1f

• Logstash should be disabled when Kafka is enabled except when a minion override exists OR node is a standalone

  Show description for b1beb61
  reyesj2

  committed
  b1beb61

  Copy full SHA for b1beb61

• Small changes needed for using new Kafka docker image + added Kafka logging output to /opt/so/log/kafka/

  Show description for 382cd24
  reyesj2

  committed
  382cd24

  Copy full SHA for 382cd24

Commits on May 27, 2024

• Remove redis.sh telegraf script when Kafka is global pipeline

  Show description for fcb6a47
  reyesj2

  committed
  fcb6a47

  Copy full SHA for fcb6a47

Commits on May 28, 2024

• Add jolokia metrics for influxdb dashboard

  Show description for 15a0b95
  reyesj2

  committed
  15a0b95

  Copy full SHA for 15a0b95

• Add basic Kafka metrics to 'Security Onion Performance' influxdb dashboard

  Show description for 47efcfd
  reyesj2

  committed
  47efcfd

  Copy full SHA for 47efcfd

• Remove unneeded jolokia aggregate metrics to reduce data ingested to influx

  Show description for 1c1a1a1
  reyesj2

  committed
  1c1a1a1

  Copy full SHA for 1c1a1a1

• only show specific fields when hovering over Kafka influxdb panels

  Show description for 0d7c331
  reyesj2

  committed
  0d7c331

  Copy full SHA for 0d7c331

• Correct dashboard name

  Show description for 77b5aa4
  reyesj2

  committed
  77b5aa4

  Copy full SHA for 77b5aa4

• Revert "Remove unneeded jolokia aggregate metrics to reduce data ingested to influx"

  Show description for 5909707
  reyesj2

  committed
  5909707

  Copy full SHA for 5909707

Commits on May 29, 2024

• elastic agent should be able to communicate over 9092 for sending logs to kafka brokers

  Show description for 876d860
  reyesj2

  committed
  876d860

  Copy full SHA for 876d860

• Update some annotations and defaults

  Show description for d9ec556
  reyesj2

  committed
  d9ec556

  Copy full SHA for d9ec556

• WIP: Manage Kafka nodes pillar role value

  Show description for 386be4e
  reyesj2

  committed
  386be4e

  Copy full SHA for 386be4e

Commits on May 30, 2024

• Update pillarWatch config for global.pipeline

  Show description for 949cea9
  reyesj2

  committed
  949cea9

  Copy full SHA for 949cea9

• Merge remote-tracking branch 'origin/2.4/dev' into reyesj2/kafka

  Show description for 1fd5165
  reyesj2

  committed
  1fd5165

  Copy full SHA for 1fd5165

• revert version for soup test before 2.4.80 pipeline unpaused

  Show description for 48713a4
  reyesj2

  committed
  48713a4

  Copy full SHA for 48713a4

• Remove bad config

  Show description for dbb99d0
  reyesj2

  committed
  dbb99d0

  Copy full SHA for dbb99d0

• Revert "revert version for soup test before 2.4.80 pipeline unpaused"

  Show description for 00b5a5c
  reyesj2

  committed
  00b5a5c

  Copy full SHA for 00b5a5c

Commits on May 31, 2024

• Update kafka nodes pillar before running highstate with pillarwatch engine. This allows configuring your Kafka controllers before cluster comes up for the first time

  Show description for e3ea477
  reyesj2

  committed
  e3ea477

  Copy full SHA for e3ea477

• Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka

  reyesj2

  committed
  75bdc92

  Copy full SHA for 75bdc92

• Move soup kafka needfuls to up_to_2.4.80

  Show description for 1a832fa
  reyesj2

  committed
  1a832fa

  Copy full SHA for 1a832fa

Commits on Jun 2, 2024

• Remove so-kafka-clusterid script, created during soup

  Show description for 2e85a28
  reyesj2

  committed
  2e85a28

  Copy full SHA for 2e85a28

Commits on Jun 4, 2024

• update receiver pillar access

  Show description for d9c58d9
  reyesj2

  committed
  d9c58d9

  Copy full SHA for d9c58d9

• update license

  Show description for fb1d4fd
  reyesj2

  committed
  fb1d4fd

  Copy full SHA for fb1d4fd

• create kafka.id from kafka {partition}-{offset}-{timestamp} for tracking event

  Show description for 3b0339a
  reyesj2

  committed
  3b0339a

  Copy full SHA for 3b0339a

Commits on Jun 6, 2024

• Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka

  reyesj2

  committed
  4581a46

  Copy full SHA for 4581a46

• Remove unused kafka reactor

  Show description for c472326
  reyesj2

  committed
  c472326

  Copy full SHA for c472326

Commits on Jun 10, 2024

• Update Kafka controller(s) via SOC UI

  Show description for 284c1be
  reyesj2

  committed
  284c1be

  Copy full SHA for 284c1be

• merge 2.4/dev

  Show description for 824f852
  reyesj2

  committed
  824f852

  Copy full SHA for 824f852

Commits on Jun 11, 2024

• kafka.id field should only be present when metadata for kafka exists

  Show description for 08557ae
  reyesj2

  committed
  08557ae

  Copy full SHA for 08557ae

• Added Kafka reset to SOC UI. Incase of changing an active broker to a controller topics may become unavailable. Resolving this would require manual intervention. This option allows running a reset …

  Show description for ca7b89c
  reyesj2

  committed
  ca7b89c

  Copy full SHA for ca7b89c

• remove dash(-) from kafka.id

  Show description for a81e4c3
  reyesj2

  committed
  a81e4c3

  Copy full SHA for a81e4c3

• remove redundant 'kafka_' from annotations & defaults

  Show description for 628893f
  reyesj2

  committed
  628893f

  Copy full SHA for 628893f

Commits on Jun 12, 2024

• Update Kafka self reset & add initial Kafka wrapper scripts to build out

  Show description for b7eebad
  reyesj2

  committed
  b7eebad

  Copy full SHA for b7eebad

• Revert a whiptail menu change. Kafka is not an install option

  Show description for dd40962
  reyesj2

  committed
  dd40962

  Copy full SHA for dd40962

• Revert a so-setup change. Kafka is not an installable option

  Show description for b56d497
  reyesj2

  committed
  b56d497

  Copy full SHA for b56d497

• Renamed Kafka pillar

  Show description for 83412b8
  reyesj2

  committed
  83412b8

  Copy full SHA for 83412b8

• Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names

  Show description for e8106be
  reyesj2

  committed
  e8106be

  Copy full SHA for e8106be

• Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka

  reyesj2

  committed
  2bc53f9

  Copy full SHA for 2bc53f9

• Use kafka:password for kafka certs

  Show description for f372b09
  reyesj2

  committed
  f372b09

  Copy full SHA for f372b09

• telegraft should only include jolokia config when Kafka is set as the global.pipeline

  Show description for 9c31622
  reyesj2

  committed
  9c31622

  Copy full SHA for 9c31622

Commits on Jun 13, 2024

• wrong location for global.pipeline check

  Show description for 80b1d51
  reyesj2

  committed
  80b1d51

  Copy full SHA for 80b1d51