[Barracuda CloudGen] Add initial Barracuda CloudGen Firewall integration #3796

legoguy1000 · 2022-07-21T17:46:08Z

What does this PR do?

Add initial Barracuda CloudGen Firewall integration for receiving Firewall Insight logs as described at https://campus.barracuda.com/product/cloudgenfirewall/doc/96025953/how-to-enable-filebeat-stream-to-a-logstash-pipeline. Elastic Agent starts a server to receive data sent over the Lumberjack protocol by CloudGen firewall. (This is the same protocol used between Beats and Logstash.)

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Requires [Filebeat] Add lumberjack input beats#32175
Closes Barracuda Cloudgen Firewall #3773

Screenshots

Logs

Real sample from CloudGen 8.3:

{
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "@timestamp": "2022-09-23T17:01:55.033Z",
  "beat": {
    "hostname": "2dc323d01e3b11ed861d0242a",
    "name": "2dc323d01e3b11ed861d0242a",
    "version": "6.2.4"
  },
  "input_type": "log",
  "message": "{\"ip_addr\":\"10.11.0.4\",\"model\":\"vfc4\",\"firmware\":\"GWAY-8.3.1-0086\",\"serial\":\"1995477\",\"hostname\":\"2dc323d01e3b11ed861d0242a\",\"box\":\"2dc323d01e3b11ed861d0242a\",\"box_description\":\"CGF-Skout\",\"geo_country\":\"na\",\"geo_position\":\"0 0 0 N; 0 0 0 E\",\"geo_latitude\":37.9273,\"geo_longitude\":-76.8545,\"brs_type\":\"version\",\"brs_index\":\"version\",\"brs_version\":1663952515,\"version\":1}",
  "offset": 371,
  "policy": "snapshot",
  "product": "ngfw",
  "prospector": {
    "type": "log"
  },
  "serial": 1995477,
  "sn": "7cc9edc95b97a5657cf7d27419009fc5",
  "source": "/var/phion/run/brsd/version"
}

2022-09-23-barracuda-cloudgen-firewall-insights.ndjson.txt

elasticmachine · 2022-07-21T17:50:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-21T16:04:51.012+0000
Duration: 15 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7
Skipped	0
Total	7

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

legoguy1000 · 2022-08-21T00:00:27Z

Tried using the lumberjack input today with the agent and I guess it needs to be added to this list first https://github.com/elastic/elastic-agent/blob/main/internal/spec/filebeat.yml#L24???

andrewkroh · 2022-08-21T00:37:49Z

Good call. That spec needs updated.

legoguy1000 · 2022-08-26T00:01:46Z

@andrewkroh Using the 8.5.0-SNAPSHOT image, the lumberjack input is working. I am running into 2 issues though.

I am currently using a filebeat container to simulate the lumberjack source but there appears to be a race condition with Filebeat and the Agent setup/policy... Unlike with elastic/stream which waits for the SIGHUP, Filebeat doesn't wait and I can't get events to generate unless I manually restart the Filebeat container and "republish" the events.
I'm getting some publishing errors

[elastic_agent.filebeat][warn] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.August, 25, 23, 56, 33, 95198956, time.UTC), Meta:{"input_id":"lumberjack-a08f22ef-5d2d-4ffc-8fb8-3d5cb2aa7f91","raw_index":"logs-barracuda_cloudgen_firewall.log-ep","stream_id":"lumberjack-barracuda_cloudgen_firewall.log-a08f22ef-5d2d-4ffc-8fb8-3d5cb2aa7f91"}, Fields:{"agent":{"ephemeral_id":"6ddcb0a4-6f36-4b37-8395-aaa73a6ba832","id":"4d7d11a6-0d0f-4694-8c64-787b073cfc45","name":"docker-fleet-agent","type":"filebeat","version":"8.5.0"},"data_stream":{"dataset":"barracuda_cloudgen_firewall.log","namespace":"ep","type":"logs"},"ecs":{"version":"8.0.0"},"elastic_agent":{"id":"4d7d11a6-0d0f-4694-8c64-787b073cfc45","snapshot":true,"version":"8.5.0"},"event":{"dataset":"barracuda_cloudgen_firewall.log"},"input":{"type":"lumberjack"},"lumberjack":{"@metadata":{"beat":"filebeat","type":"_doc","version":"8.3.3"},"@timestamp":"2022-08-25T23:56:32.088Z","agent":{"ephemeral_id":"47e01448-f2e5-4f53-b71f-ee50576d7028","id":"17ee087a-c1dd-4d73-90c6-c8f9c81c1e02","name":"8761db0f577a","type":"filebeat","version":"8.3.3"},"ecs":{"version":"8.0.0"},"host":{"name":"8761db0f577a"},"input":{"type":"filestream"},"log":{"file":{"path":"/sample_logs/web.log"},"offset":467},"message":"{\"timestamp\":1526377804000,\"traffic_type\":0,\"action\":0,\"source_ip\":\"192.168.42.105\",\"source_port\":\"50159\",\"destination_ip\":\"89.160.20.114\",\"destination_port\":\"443\",\"method\":\"GET\",\"status_code\":\"200\",\"user_agent\":\"mozilla/5.0 (windows nt 6.1) applewebkit/537.36 (khtml, like gecko) chrome/66.0.3359.139 safari/537.36\",\"content_type\":\"\",\"name\":\"https://clientservices.googleapis.com/chrome-variations/seed?osname=win\u0026channel=stable\u0026milestone=66\",\"size\":0,\"domain\":\"clientservices.googleapis.com\",\"category\":[],\"user\":\"192.168.42.105\",\"user_type\":0,\"fw_rule\":\"LAN-2-INTERNET\",\"app_rule\":\"\u003cApp\u003e:\u003cpass-no-match\u003e\"}","product":"ngfw","sn":"4f94abdf7a8c465fa2cd76f680ecafd1","type":"ngfw-wf"},"source":{"address":"192.168.16.5:53796"},"tags":["barracuda_cloudgen_firewall-log","forwarded"]}, Private:(*lumberjack.batchACKTracker)(0xc000011008), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [@timestamp] of type [date] in document with id '9Qxv14IBNgppDiFepmQJ'. Preview of field's value: '+50338-12-27T18:26:40.000Z'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [+50338-12-27T18:26:40.000Z] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}, dropping event!

andrewkroh · 2022-08-26T00:28:48Z

We can add in a lumberjack output to elastic/stream to ease testing. I've opened elastic/stream#39 to track this.

I'm getting some publishing errors

The event that is logged doesn't show that odd year value so I wonder if it's something happening on the ingest node pipeline side that's causing the data corruption.

legoguy1000 · 2022-08-26T00:30:54Z

I think I may have found it. The Web data uses Epoch MS, not Seconds. Also I may have a way to wait for the agent to be ready.

legoguy1000 · 2022-08-26T00:40:29Z

Error 1 is resolved. My plan for the race condition seems to kinda work but I think there is a bigger problem as I'm getting multiple (5+) log records of the lumberjack input starting and stopping and I think thats causing some of the issues.

elasticmachine · 2022-09-07T00:43:29Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

epixa · 2022-09-07T01:08:25Z

Can you add this integration to the CODEOWNERS file in this PR?

andrewkroh · 2022-09-13T23:47:42Z

I added the Lumberjack output to elastic/stream in elastic/stream#41. This is the config that I was using. I got blocked by an issue in 8.5.0-SNAPSHOT today where the API keys for Filebeat were invalid, so I couldn't test E2E. But I did confirm that stream was connected and communicating to Filebeat.

The JSON contained in the sample_logs will need adapted to look more like what Filebeat is be sending.

version: '2.3'
services:
  barracuda-cloudgen-lumberjack:
    image: docker.elastic.co/observability/stream:local # Requires next release.
    volumes:
      - ./sample_logs:/sample_logs:ro
    environment:
      - STREAM_PROTOCOL=lumberjack
      - STREAM_LUMBERJACK_PARSE_JSON=true
      - STREAM_ADDR=tcp://elastic-agent:5044
      - STREAM_DELAY=5s
      - STREAM_START_SIGNAL=SIGHUP
    command: log /sample_logs/*.log

I also found a bug in the input that would cause it Filebeat to panic if multiple clients were streaming data. elastic/beats#33071

legoguy1000 · 2022-09-14T00:23:11Z

Using ur example, the JSON in the sample logs file was sent in the message block to the agent? Is there a method to send additional fields? The CloudGen is sending additional fields separate from the message field, https://campus.barracuda.com/product/cloudgenfirewall/doc/96025108/how-to-enable-filebeat-stream-to-a-logstash-pipeline/, also shown in my POC using Filebeat.

andrewkroh · 2022-09-14T00:36:16Z

Using my example stream config, each JSON log line will be sent as a structured event. On receiving side, the lumberjack input will produce an event that contains lumberjack.* whose contents are everything from the JSON log line (as an object).

andrewkroh · 2022-09-15T16:43:51Z

The new container with lumberjack output is ready: docker.elastic.co/observability/stream:v0.8.0

andrewkroh · 2022-09-15T16:53:58Z

/test

elasticmachine · 2022-09-15T17:17:35Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`4/4`)	💚 2.669
Classes	100.0% (`4/4`)	💚 2.669
Methods	100.0% (`33/33`)	💚 10.207
Lines	89.107% (`499/560`)	👎 -2.154
Conditionals	100.0% (`0/0`)	💚

legoguy1000 · 2022-09-19T18:29:16Z

@andrewkroh I think ready to try tests. I sent a message in slack with an error that the stream container seems to have

{"level":"info","ts":"2022-09-19T18:01:29.890Z","caller":"command/root.go:138","msg":"Delaying connection.","delay":5}
{"level":"debug","ts":"2022-09-19T18:01:34.891Z","caller":"output/util.go:28","msg":"Connecting...","address":"tcp://elastic-agent:5044"}
{"level":"info","ts":"2022-09-19T18:01:34.894Z","caller":"output/util.go:41","msg":"Connected","address":"tcp://elastic-agent:5044"}
{"level":"debug","ts":"2022-09-19T18:01:34.895Z","caller":"command/log.go:81","msg":"Sending log line.","address":"tcp://elastic-agent:5044","log":"/sample_logs/firewall.log","line_number":1}
Error: EOF

It appears to hang on the sending log line....

This is to be consistent with other integrations. Also disable lumberjack v1 by default.

@timestamp

If a failure occurs while @timestamp was deleted then indexing would fail and data would be lost.

[git-generate] cd packages/barracuda_cloudgen_firewall/data_stream/log/fields yq 'sort_by(.name)' ecs.yml yq -i 'sort_by(.name)' ecs.yml

Relates elastic#4227

I didn't see where it was used. And if we were going to keep it then it should be using 'external: ecs'.

andrewkroh · 2022-09-21T13:54:10Z

/test

legoguy1000 · 2022-09-21T13:57:22Z

packages/barracuda_cloudgen_firewall/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+  - rename:
+      field: source.address
+      target_field: labels.origin_address
+      ignore_missing: true
+  - rename:
+      field: tls.client.subject
+      target_field: labels.origin_client_subject
+      ignore_missing: true
+  - remove:
+      field:
+        - source
+        - tls
+      ignore_missing: true


Separate from this PR would it make sense to update the lumberjack input to use different fields than source and tls in the actual input code? It seems misleading by default to have those fields populated when its only relevant to the transport of receiving the logs, not really whats in the log its self? Just a thought.

I was thinking to propose some official ECS fields for this data like perhaps log.source.ip, log.source.port, and log.source.x509.*.

The syslog and TCP/UDP uses log.source.address but its not an official ECS field, i think its a legacy but works for exactly what is wanted.

elasticmachine · 2022-09-21T14:08:01Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Add sample event to readme. Remove reference links. I'm not sure if they are supported in kibana.

andrewkroh · 2022-09-21T14:19:42Z

/test

andrewkroh

I think this is good to go for a technical preview. Things to follow up on:

Propose standardized fields for log origin metadata.
Test with a real CloudGen firewall.
Document TLS requirements if any (I think the CloudGen UI requires TLS enabled on the server.)

andrewkroh · 2022-09-21T15:54:22Z

Great work @legoguy1000 getting to the finishing line. Thank you!

andrewkroh · 2022-09-21T16:04:36Z

/test

andrewkroh added Team:Security-External Integrations Integration:barracuda_cloudgen_firewall Barracuda CloudGen Firewall Logs New Integration labels Aug 15, 2022

andrewkroh mentioned this pull request Aug 21, 2022

Add Filebeat lumberjack input to spec elastic/elastic-agent#959

Merged

6 tasks

andrewkroh mentioned this pull request Aug 26, 2022

Add lumberjack output elastic/stream#39

Closed

legoguy1000 marked this pull request as ready for review September 7, 2022 00:43

legoguy1000 added 4 commits September 15, 2022 16:49

elastic#3773: Add initial Barracuda Cloudgen Firewall integration

cdf259a

Add sample data, inital framework for system tests

ed7bf0d

Resolve pipeline issues

8c25b62

update codeowners

1e14290

legoguy1000 force-pushed the 3773-barracuda-cloudgen branch from 644bbeb to 1e14290 Compare September 15, 2022 16:50

legoguy1000 added 3 commits September 16, 2022 20:05

test

7f62b11

update test config

db1c514

update tests

48c51ac

andrewkroh added 19 commits September 21, 2022 08:09

change "Cloudgen" to "CloudGen"

05acc85

Separate listen_address and listen_port

a36247f

This is to be consistent with other integrations. Also disable lumberjack v1 by default.

Match ECS version in pipeline to build.yml

4d44319

Remove message when event.original already exists

6226b95

Use triple braces in mustache templates to avoid escapes

9a78e00

Regenerate golden files with ecs.version 8.4.0

6956267

Use labels to hold origin metadata

f8fa120

Refactor pipeline so that @timestamp is always present

71b4e40

If a failure occurs while @timestamp was deleted then indexing would fail and data would be lost.

elastic-package format

5302349

Define labels in fields

d7d2bc4

Sort ecs.yml fields

8b5a6bf

[git-generate] cd packages/barracuda_cloudgen_firewall/data_stream/log/fields yq 'sort_by(.name)' ecs.yml yq -i 'sort_by(.name)' ecs.yml

Use ECS definition of source.geo.location

c5157c3

Relates elastic#4227

Remove duplicate @timestamp and tags field declarations

54bf2ed

Remove container.id

848d36e

I didn't see where it was used. And if we were going to keep it then it should be using 'external: ecs'.

Fix duration to ns conversion

4dd436c

Update sample event

2aea11a

Add tag and description to all scripts

2a1fed0

Don't use null-safe operator on 'ctx' - ctx?.

9bb842e

Normalize indentation in YAML

4453f19

andrewkroh changed the title ~~[Barracuda Cloudgen] Add initial Barracuda Cloudgen Firewall integration~~ [Barracuda CloudGen] Add initial Barracuda CloudGen Firewall integration Sep 21, 2022

legoguy1000 commented Sep 21, 2022

View reviewed changes

Add sample event to readme

c9a1ee2

Add sample event to readme. Remove reference links. I'm not sure if they are supported in kibana.

Tailor manifest to descriptions in CloudGen docs

1283c14

andrewkroh approved these changes Sep 21, 2022

View reviewed changes

andrewkroh merged commit 8cded63 into elastic:main Sep 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Barracuda CloudGen] Add initial Barracuda CloudGen Firewall integration #3796

[Barracuda CloudGen] Add initial Barracuda CloudGen Firewall integration #3796

legoguy1000 commented Jul 21, 2022 •

edited by andrewkroh

Loading

elasticmachine commented Jul 21, 2022 •

edited

Loading

Build stats

Test stats 🧪

legoguy1000 commented Aug 21, 2022

andrewkroh commented Aug 21, 2022

legoguy1000 commented Aug 26, 2022

andrewkroh commented Aug 26, 2022

legoguy1000 commented Aug 26, 2022

legoguy1000 commented Aug 26, 2022

elasticmachine commented Sep 7, 2022

epixa commented Sep 7, 2022

andrewkroh commented Sep 13, 2022 •

edited

Loading

legoguy1000 commented Sep 14, 2022

andrewkroh commented Sep 14, 2022

andrewkroh commented Sep 15, 2022

andrewkroh commented Sep 15, 2022

elasticmachine commented Sep 15, 2022 •

edited

Loading

legoguy1000 commented Sep 19, 2022

andrewkroh commented Sep 21, 2022

legoguy1000 Sep 21, 2022

andrewkroh Sep 21, 2022

legoguy1000 Sep 21, 2022

elasticmachine commented Sep 21, 2022

andrewkroh commented Sep 21, 2022

andrewkroh left a comment

andrewkroh commented Sep 21, 2022

andrewkroh commented Sep 21, 2022

[Barracuda CloudGen] Add initial Barracuda CloudGen Firewall integration #3796

[Barracuda CloudGen] Add initial Barracuda CloudGen Firewall integration #3796

Conversation

legoguy1000 commented Jul 21, 2022 • edited by andrewkroh Loading

What does this PR do?

Checklist

Related issues

Screenshots

Logs

elasticmachine commented Jul 21, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

legoguy1000 commented Aug 21, 2022

andrewkroh commented Aug 21, 2022

legoguy1000 commented Aug 26, 2022

andrewkroh commented Aug 26, 2022

legoguy1000 commented Aug 26, 2022

legoguy1000 commented Aug 26, 2022

elasticmachine commented Sep 7, 2022

epixa commented Sep 7, 2022

andrewkroh commented Sep 13, 2022 • edited Loading

legoguy1000 commented Sep 14, 2022

andrewkroh commented Sep 14, 2022

andrewkroh commented Sep 15, 2022

andrewkroh commented Sep 15, 2022

elasticmachine commented Sep 15, 2022 • edited Loading

🌐 Coverage report

legoguy1000 commented Sep 19, 2022

andrewkroh commented Sep 21, 2022

legoguy1000 Sep 21, 2022

Choose a reason for hiding this comment

andrewkroh Sep 21, 2022

Choose a reason for hiding this comment

legoguy1000 Sep 21, 2022

Choose a reason for hiding this comment

elasticmachine commented Sep 21, 2022

🚀 Benchmarks report

andrewkroh commented Sep 21, 2022

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Sep 21, 2022

andrewkroh commented Sep 21, 2022

legoguy1000 commented Jul 21, 2022 •

edited by andrewkroh

Loading

elasticmachine commented Jul 21, 2022 •

edited

Loading

andrewkroh commented Sep 13, 2022 •

edited

Loading

elasticmachine commented Sep 15, 2022 •

edited

Loading