Skip to content

network_traffic: expose configuration options to users #3157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
efd6 merged 6 commits into from
May 11, 2022
Merged

network_traffic: expose configuration options to users #3157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3b2d504
make file names consistent where possible
efd6 Apr 21, 2022
1bf369c
update changelog and manifest
efd6 Apr 28, 2022
d72435e
attach comment of preferred default to correct yaml node
efd6 May 3, 2022
29514bd
map ports as multi text
efd6 May 10, 2022
d168b9a
prepare tests for multi ports
efd6 May 11, 2022
875c0f8
generate config files
efd6 May 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/network_traffic/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.10.0"
changes:
- description: Add configuration options for each protocol.
type: enhancement
link: https://github.com/elastic/integrations/pull/3157
- version: "0.9.0"
changes:
- description: Update to ECS 8.2
Expand Down
47 changes: 39 additions & 8 deletions packages/network_traffic/data_stream/amqp/agent/stream/amqp.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,43 @@
type: amqp
ports: [{{port}}]
{{#if processes}}
procs:
enabled: true
monitored:
{{#each processes}}
- cmdline_grep: {{this}}
{{/each}}
{{#if port}}
ports:
{{#each port as |p|}}
- {{p}}
{{/each}}
{{/if}}
{{#if max_body_length}}
max_body_length: {{max_body_length}}
{{/if}}
{{#if parse_headers}}
parse_headers: {{parse_headers}}
{{/if}}
{{#if parse_arguments}}
parse_arguments: {{parse_arguments}}
{{/if}}
{{#if hide_connection_information}}
hide_connection_information: {{hide_connection_information}}
{{/if}}
{{#if send_request}}
send_request: {{send_request}}
{{/if}}
{{#if send_response}}
send_response: {{send_response}}
{{/if}}
{{#if keep_null}}
keep_null: {{keep_null}}
{{/if}}
{{#if transaction_timeout}}
transaction_timeout: {{transaction_timeout}}
{{/if}}
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{#if tags}}
tags:
{{#each tags as |tag|}}
- {{tag}}
{{/each}}
{{/if}}
{{#if interface}}
Copy link
Member

@P1llus P1llus Apr 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add support for custom processors, similar to here:
https://github.com/elastic/integrations/blob/main/packages/tcp/data_stream/generic/agent/stream/tcp.yml.hbs#L38

interface:
Expand Down
96 changes: 91 additions & 5 deletions packages/network_traffic/data_stream/amqp/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,101 @@ streams:
- input: packet
vars:
- name: port
type: integer
# currently the Kibana UI doesn't support multi inputs
# that are numeric, you get "Error: r.toLowerCase is not a function"
# multi: true
title: Port
# so map this as text
type: text
multi: true
title: Ports
required: true
Copy link
Member

@P1llus P1llus Apr 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it does not support multi, it might be wise that we change this to type yaml, instead of text or integer, to allow for array support?

Copy link
Contributor Author

@efd6 efd6 Apr 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will that get around the kibana issue?

Copy link
Member

@andrewkroh andrewkroh May 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using mutli: true with type: text I can still pass numbers through the policy (tested this with tags). One option could be to change the type to text and forego the number validation until this is fixed. And speaking of fixes, I cannot find an open issue in elastic/kibana for supporting multi + integer, can you please open one, @efd6?

Copy link
Contributor Author

@efd6 efd6 May 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

elastic/kibana#131360

Copy link
Contributor Author

@efd6 efd6 May 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried multi: true with type: text and I am unable to get this to work; the tests fail with {"statusCode":400,"error":"Bad Request","message":"Package policy is invalid: inputs.packet.streams.network_traffic.http.vars.port: Invalid format"}.

Copy link
Member

@andrewkroh andrewkroh May 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With tags the handlebar template uses a for loop to construct the value. Maybe try that format if you haven't already.

Copy link
Contributor Author

@efd6 efd6 May 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that was one of the many already tried.

Copy link
Member

@andrewkroh andrewkroh May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using multi: true with type: text and for loop in the Handlebar template is working for me on 7.17.0.

Screen Shot 2022-05-10 at 15 22 39

Generated policy:

  - id: packet-network-9c3f6353-0533-4ad9-828f-fd99cf93f3f1
    name: network_traffic-1
    revision: 1
    type: packet
    use_output: default
    meta:
      package:
        name: network_traffic
        version: 0.10.0
    data_stream:
      namespace: default
    streams:
      - id: packet-network_traffic.dhcpv4-9c3f6353-0533-4ad9-828f-fd99cf93f3f1
        type: dhcpv4
        data_stream:
          dataset: network_traffic.dhcpv4
          type: logs
        ports:
          - 67
          - 68

diff --git a/packages/network_traffic/data_stream/dhcpv4/agent/stream/dhcpv4.yml.hbs b/packages/network_traffic/data_stream/dhcpv4/agent/stream/dhcpv4.yml.hbs
index 3856fc4d9..dba4a3a7b 100644
--- a/packages/network_traffic/data_stream/dhcpv4/agent/stream/dhcpv4.yml.hbs
+++ b/packages/network_traffic/data_stream/dhcpv4/agent/stream/dhcpv4.yml.hbs
@@ -1,5 +1,12 @@
 type: dhcpv4
-ports: [{{port}}]
+
+{{#if port}}
+ports:
+{{#each port as |p|}}
+  - {{p}}
+{{/each}}
+{{/if}}
+
 {{#if keep_null}}
 keep_null: {{keep_null}}
 {{/if}}
diff --git a/packages/network_traffic/data_stream/dhcpv4/manifest.yml b/packages/network_traffic/data_stream/dhcpv4/manifest.yml
index 8e5ff7b20..646b0cc19 100644
--- a/packages/network_traffic/data_stream/dhcpv4/manifest.yml
+++ b/packages/network_traffic/data_stream/dhcpv4/manifest.yml
@@ -5,14 +5,14 @@ streams:
   - input: packet
     vars:
       - name: port
-        type: integer
+        type: text
         # currently the Kibana UI doesn't support multi inputs
         # that are numeric, you get "Error: r.toLowerCase is not a function"
-        # multi: true
-        title: Port
+        multi: true
+        title: Ports
         required: true
         show_user: true
-        default: 67 # default: [67, 68]
+        default: [67, 68]
       - name: keep_null
         type: bool
         title: Keep Null

show_user: true
default: 5672
# default: [5672]
default: [5672]
- name: max_body_length
type: integer
title: Max Body Length
description: |-
Truncate messages that are published and avoid huge messages being
indexed.
Default: 1000
show_user: false
multi: false
required: false
- name: parse_headers
type: bool
title: Parse Headers
description: |-
Hide the header fields in header frames.
Default: false
show_user: false
multi: false
required: false
- name: parse_arguments
type: bool
title: Parse Arguments
description: |-
Hide the additional arguments of method frames.
Default: false
show_user: false
multi: false
required: false
- name: hide_connection_information
type: bool
title: Hide Connection Information
description: |-
Hide all methods relative to connection negotiation between server and
client.
Default: true
show_user: false
multi: false
required: false
- name: send_request
type: bool
title: Send Request
description: |-
If this option is enabled, the raw message of the request (`request` field)
is sent to Elasticsearch. The default is false.
show_user: false
multi: false
required: false
- name: send_response
type: bool
title: Send Response
description: |-
If this option is enabled, the raw message of the response (`response`
field) is sent to Elasticsearch. The default is false.
show_user: false
multi: false
required: false
- name: keep_null
type: bool
title: Keep Null
description: Set to true to publish fields with null values in events.
show_user: false
multi: false
required: false
- name: transaction_timeout
type: text
title: Transaction Timeout
description: |-
Transaction timeout. Expired transactions will no longer be correlated to
incoming responses, but sent to Elasticsearch immediately.
show_user: false
multi: false
required: false
- name: processors
type: yaml
title: Processors
description: Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
show_user: false
multi: false
required: false
- name: tags
type: text
title: Tags
description: Tags to include in the published event.
show_user: false
multi: true
required: false
title: AMQP
description: Capture AMQP Traffic
template_path: amqp.yml.hbs
47 changes: 39 additions & 8 deletions packages/network_traffic/data_stream/cassandra/agent/stream/cassandra.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,43 @@
type: cassandra
ports: [{{port}}]
{{#if processes}}
procs:
enabled: true
monitored:
{{#each processes}}
- cmdline_grep: {{this}}
{{/each}}
{{#if port}}
ports:
{{#each port as |p|}}
- {{p}}
{{/each}}
{{/if}}
{{#if send_request}}
send_request: {{send_request}}
{{/if}}
{{#if send_request_header}}
send_request_header: {{send_request_header}}
{{/if}}
{{#if send_response}}
send_response: {{send_response}}
{{/if}}
{{#if send_response_header}}
send_response_header: {{send_response_header}}
{{/if}}
{{#if keep_null}}
keep_null: {{keep_null}}
{{/if}}
{{#if compressor}}
compressor: {{compressor}}
{{/if}}
{{#if ignored_ops}}
ignored_ops:
{{#each ignored_ops as |ignored_op|}}
- {{ignored_op}}
{{/each}}
{{/if}}
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{#if tags}}
tags:
{{#each tags as |tag|}}
- {{tag}}
{{/each}}
{{/if}}
{{#if interface}}
interface:
Expand Down
85 changes: 79 additions & 6 deletions packages/network_traffic/data_stream/cassandra/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,88 @@ streams:
- input: packet
vars:
- name: port
type: integer
title: Port
required: true
# currently the Kibana UI doesn't support multi inputs
# that are numeric, you get "Error: r.toLowerCase is not a function"
# multi: true
# so map this as text
type: text
multi: true
title: Ports
required: true
show_user: true
default: 9042
# default: [9042]
default: [9042]
- name: send_request
type: bool
title: Send Request
description: |-
If this option is enabled, the raw message of the request (`cassandra_request` field)
is included in published events. The default is true.
show_user: false
multi: false
required: false
- name: send_request_header
type: bool
title: Send Request Header
description: |-
If this option is enabled, the raw message of the response (`cassandra_request.request_headers` field)
is included in published events. The default is true. enable `send_request` first before enable this option.
show_user: false
multi: false
required: false
- name: send_response
type: bool
title: Send Response
description: |-
If this option is enabled, the raw message of the response (`cassandra_response` field)
is included in published events. The default is true.
show_user: false
multi: false
required: false
- name: send_response_header
type: bool
title: Send Response Header
description: |-
If this option is enabled, the raw message of the response (`cassandra_response.response_headers` field)
is included in published events. The default is true. enable `send_response` first before enable this option.
show_user: false
multi: false
required: false
- name: keep_null
type: bool
title: Keep Null
description: Set to true to publish fields with null values in events.
show_user: false
multi: false
required: false
- name: compressor
type: text
title: Compressor
description: |-
Configures the default compression algorithm being used to uncompress compressed frames by name. Currently only `snappy` is can be configured.
By default no compressor is configured.
show_user: false
multi: false
required: false
- name: ignored_ops
type: text
title: Ignored Ops
description: This option indicates which Operator/Operators will be ignored.
show_user: false
multi: true
required: false
- name: processors
type: yaml
title: Processors
description: Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
show_user: false
multi: false
required: false
- name: tags
type: text
title: Tags
description: Tags to include in the published event.
show_user: false
multi: true
required: false
title: Cassandra
description: Capture Cassandra Traffic
template_path: cassandra.yml.hbs
18 changes: 0 additions & 18 deletions packages/network_traffic/data_stream/dhcpv4/agent/stream/dhcp.yml.hbs
View file
Open in desktop

This file was deleted.

28 changes: 28 additions & 0 deletions packages/network_traffic/data_stream/dhcpv4/agent/stream/dhcpv4.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
type: dhcpv4
{{#if port}}
ports:
{{#each port as |p|}}
- {{p}}
{{/each}}
{{/if}}
{{#if keep_null}}
keep_null: {{keep_null}}
{{/if}}
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{#if tags}}
tags:
{{#each tags as |tag|}}
- {{tag}}
{{/each}}
{{/if}}
{{#if interface}}
interface:
{{#if (contains ".pcap" interface)}}
file: {{interface}}
{{else}}
device: {{interface}}
{{/if}}
{{/if}}
Loading