Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
7bcdf53
Shorewall module: Initial commit
mdelapenya May 16, 2019
5385157
Run 'make create-fileset MODULE=shorewall FILESET=log' to initialise …
mdelapenya May 16, 2019
a0c8545
Define GROK expression to parse Shorewall
mdelapenya May 16, 2019
4d22a58
Update default file to parse
mdelapenya May 16, 2019
98904d9
Add shorewall log example for tests
mdelapenya May 16, 2019
e4282d2
Add expected JSON after running tests
mdelapenya May 16, 2019
0359a79
Use grok expression instead of its grok word
mdelapenya May 17, 2019
1754755
Convert one field (nf_src_port) to integer
mdelapenya May 17, 2019
41c8c0c
Update expected test output after running tests
mdelapenya May 17, 2019
4fc87ec
Convert all integer fields
mdelapenya May 17, 2019
c565d6d
Convert timestamp field into Elasticsearch's @timestamp identifier
mdelapenya May 17, 2019
fd1fd04
Remove old timestamp field in favour of ES's defaults @timestamp
mdelapenya May 17, 2019
9b4518f
Update expected log for timestamp
mdelapenya May 17, 2019
624078e
Use sneakcase for fields
mdelapenya May 17, 2019
2bc9ca4
Use Elastic Common Schema (ECS) for existing fields
mdelapenya May 17, 2019
3a899dd
Update action fields to denote they are shorewall specifics
mdelapenya May 17, 2019
037c84d
Move field documentation to the proper location: the fileset
mdelapenya May 18, 2019
e55ad2c
Remove ECS fields from docs, are they are not needed
mdelapenya May 18, 2019
329ca15
Use proper shared field definition for shorewall
mdelapenya May 18, 2019
ff07455
Use Elastic Common Schema (ECS) for more fields
mdelapenya May 18, 2019
5c01192
Represent shorewall specific fields using dot notation for children e…
mdelapenya May 18, 2019
0359d43
Update expected log after tests run
mdelapenya May 18, 2019
8e5cc48
Use proper type for LEN field
mdelapenya May 18, 2019
007b0cb
Run "make update" to autogenerate docs for fields
mdelapenya May 20, 2019
e0deca0
Run "make update" from OSS directory to autogenerate docs for fields,…
mdelapenya May 20, 2019
db763af
Move specific log fields to the parent level
mdelapenya May 20, 2019
0535c39
Add more log examples from a real firewall
mdelapenya May 20, 2019
63dbeda
Improve patterns covering the new file
mdelapenya May 20, 2019
cea3267
List new fields into the docs
mdelapenya May 20, 2019
b13e4a6
Update expected log for tests
mdelapenya May 20, 2019
c7cb5c9
Document fields with "make update"
mdelapenya May 20, 2019
b178dcf
Remove path for Darwin and Windows
mdelapenya May 20, 2019
5f84a65
Use Shorewall zone fields
mdelapenya May 21, 2019
146485d
Set UTC timezone when configuring @timestamp
mdelapenya May 21, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions filebeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ grouped in the following categories:
* <<exported-fields-rabbitmq>>
* <<exported-fields-redis>>
* <<exported-fields-santa>>
* <<exported-fields-shorewall>>
* <<exported-fields-suricata>>
* <<exported-fields-system>>
* <<exported-fields-traefik>>
Expand Down Expand Up @@ -12484,6 +12485,148 @@ type: keyword

Hash of process executable.

--

[[exported-fields-shorewall]]
== shorewall fields

Module for parsing Shorewall log files.



[float]
== shorewall fields

Fields from Shorewall logs.



[float]
== network fields

Shorewal log files



*`shorewall.network.in`*::
+
--
type: keywork

Name of the input network interface


--

*`shorewall.network.out`*::
+
--
type: keyword

Name of the output network interface


--

[float]
== action fields

Shorewal network log files



*`shorewall.action.one`*::
+
--
type: keywork

Specifies the action to be taken if the connection request matches the rule. target must be one of the following values (ACCEPT, ACCEPT+, ACCEPT!, ADD, AUDIT, A_ACCEPT, A_ACCEPT+, A_ACCEPT!, A_DROP, A_DROP!, A_REJECT, A_REJECT!, ?COMMENT, CONMARK, CONTINUE, CONTINUE!, COUNT, DEL, DNAT, DNAT-, DROP, DROP!, HELPER, INLINE, IPTABLES, IP6TABLES, LOG, MACRO, MARK, NFLOG, NFQUEUE, NONAT, QUEUE, QUEUE!, REJECT, REJECT!, REDIRECT, REDIRECT-, TARPIT, ULOG. See http://shorewall.net/manpages/shorewall-rules.html


--

*`shorewall.action.two`*::
+
--
type: keyword

Action two


--

*`shorewall.frame_type`*::
+
--
type: keyword

This field is part of the MAC address in the log. It indicates whether the ethernet frame carried an IPv4 datagram or not.


--

*`shorewall.precedence`*::
+
--
type: keyword

Type Of Service, and it's increasingly being replaced by DS and ECN.


--

*`shorewall.res`*::
+
--
type: keyword

Reserved bits. The ECN flags "CWR" and "ECNE" will show up in the two least significant bits of this field.


--

*`shorewall.time1`*::
+
--
type: integer

--

*`shorewall.time2`*::
+
--
type: integer

--

*`shorewall.ttl`*::
+
--
type: integer

The remaining Time To Live.


--

*`shorewall.urgp`*::
+
--
type: integer

The Urgent Pointer allows for urgent, "out of band" data transfer. Unfortunately not all protocol implementations agree, so this facility is hardly ever used.


--

*`shorewall.window`*::
+
--
type: integer

The TCP Receive Window size. This may be scaled by bit-shifting left by a number of bits specified in the "Window Scale" TCP option. If the host supports ECN, then the TCP Receive Window size will also be controlled by that.


--

[[exported-fields-suricata]]
Expand Down
60 changes: 60 additions & 0 deletions filebeat/docs/modules/shorewall.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
////
This file is generated! See scripts/docs_collector.py
////

[[filebeat-module-shorewall]]
:modulename: shorewall
:has-dashboards: true

== shorewall module

This is the shorewall module.

include::../include/what-happens.asciidoc[]

[float]
=== Compatibility

TODO: document with what versions of the software is this tested


include::../include/running-modules.asciidoc[]

[float]
=== Example dashboard

This module comes with a sample dashboard. For example:

TODO: include an image of a sample dashboard. If you do not include a dashboard,
remove this section and set `:has-dashboards: false` at the top of this file.

include::../include/configuring-intro.asciidoc[]

TODO: provide an example configuration

:fileset_ex: {fileset}

include::../include/config-option-intro.asciidoc[]

TODO: document the variables from each fileset. If you're describing a variable
that's common to other modules, you can reuse shared descriptions by including
the relevant file. For example:

[float]
==== `{fileset}` log fileset settings

include::../include/var-paths.asciidoc[]

:has-dashboards!:

:fileset_ex!:

:modulename!:


[float]
=== Fields

For a description of each field in the module, see the
<<exported-fields-shorewall,exported fields>> section.

2 changes: 2 additions & 0 deletions filebeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ This file is generated! See scripts/docs_collector.py
* <<filebeat-module-rabbitmq>>
* <<filebeat-module-redis>>
* <<filebeat-module-santa>>
* <<filebeat-module-shorewall>>
* <<filebeat-module-suricata>>
* <<filebeat-module-system>>
* <<filebeat-module-traefik>>
Expand Down Expand Up @@ -60,6 +61,7 @@ include::modules/postgresql.asciidoc[]
include::modules/rabbitmq.asciidoc[]
include::modules/redis.asciidoc[]
include::modules/santa.asciidoc[]
include::modules/shorewall.asciidoc[]
include::modules/suricata.asciidoc[]
include::modules/system.asciidoc[]
include::modules/traefik.asciidoc[]
Expand Down
10 changes: 10 additions & 0 deletions x-pack/filebeat/filebeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,16 @@ filebeat.modules:
# Filebeat will choose the the default path.
#var.paths:

#------------------------------ Shorewall Module ------------------------------
- module: shorewall
# All logs
{fileset}:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

#------------------------------- Suricata Module -------------------------------
- module: suricata
# All logs
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions x-pack/filebeat/module/shorewall/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- module: shorewall
# All logs
{fileset}:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
47 changes: 47 additions & 0 deletions x-pack/filebeat/module/shorewall/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
:modulename: shorewall
:has-dashboards: true

== shorewall module

This is the shorewall module.

include::../include/what-happens.asciidoc[]

[float]
=== Compatibility

TODO: document with what versions of the software is this tested
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TODO should be completed or removed



include::../include/running-modules.asciidoc[]

[float]
=== Example dashboard

This module comes with a sample dashboard. For example:

TODO: include an image of a sample dashboard. If you do not include a dashboard,
remove this section and set `:has-dashboards: false` at the top of this file.

include::../include/configuring-intro.asciidoc[]

TODO: provide an example configuration

:fileset_ex: {fileset}

include::../include/config-option-intro.asciidoc[]

TODO: document the variables from each fileset. If you're describing a variable
that's common to other modules, you can reuse shared descriptions by including
the relevant file. For example:

[float]
==== `{fileset}` log fileset settings

include::../include/var-paths.asciidoc[]

:has-dashboards!:

:fileset_ex!:

:modulename!:
81 changes: 81 additions & 0 deletions x-pack/filebeat/module/shorewall/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
- key: shorewall
title: "shorewall"
description: >
Module for parsing Shorewall log files.
fields:
- name: shorewall
type: group
description: >
Fields from Shorewall logs.
fields:
- name: network
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@webmat Potentially useful for ECS?

type: group
description: >
Shorewal log files
fields:
- name: in
type: keywork
description: >
Name of the input network interface
- name: out
type: keyword
description: >
Name of the output network interface
- name: frame_type
type: keyword
description: >
This field is part of the MAC address in the log. It indicates whether
the ethernet frame carried an IPv4 datagram or not.
- name: packet_action
type: keywork
description: >
Specifies the action to be taken if the connection request matches
the rule. target must be one of the following values (ACCEPT, ACCEPT+,
ACCEPT!, ADD, AUDIT, A_ACCEPT, A_ACCEPT+, A_ACCEPT!, A_DROP, A_DROP!,
A_REJECT, A_REJECT!, ?COMMENT, CONMARK, CONTINUE, CONTINUE!, COUNT, DEL,
DNAT, DNAT-, DROP, DROP!, HELPER, INLINE, IPTABLES, IP6TABLES, LOG, MACRO,
MARK, NFLOG, NFQUEUE, NONAT, QUEUE, QUEUE!, REJECT, REJECT!, REDIRECT,
REDIRECT-, TARPIT, ULOG. See http://shorewall.net/manpages/shorewall-rules.html
- name: precedence
type: keyword
description: >
Type Of Service, and it's increasingly being replaced by DS and ECN.
- name: res
type: keyword
description: >
Reserved bits. The ECN flags "CWR" and "ECNE" will show up in the two
least significant bits of this field.
- name: time1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is time1 and time2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm gonna change this, as that [x.y] is the timestamp coming from kernel

type: integer
- name: time2
type: integer
- name: ttl
type: integer
description: >
The remaining Time To Live.
- name: urgp
type: integer
description: >
The Urgent Pointer allows for urgent, "out of band" data transfer.
Unfortunately not all protocol implementations agree, so this facility
is hardly ever used.
- name: window
type: integer
description: >
The TCP Receive Window size. This may be scaled by bit-shifting left
by a number of bits specified in the "Window Scale" TCP option.
If the host supports ECN, then the TCP Receive Window size will also
be controlled by that.
- name: zone
type: group
description: >
Shorewal Zone fields
fields:
- name: device
type: keyword
description: >
Name of the device for the zone
- name: name
type: keywork
description: >
Name of the zone
Loading