diff --git a/modules/network-observability-con_filter-network-flows-at-ingestion.adoc b/modules/network-observability-con_filter-network-flows-at-ingestion.adoc index 1e4dab01d7ec..1176ac8e24cd 100644 --- a/modules/network-observability-con_filter-network-flows-at-ingestion.adoc +++ b/modules/network-observability-con_filter-network-flows-at-ingestion.adoc @@ -78,4 +78,6 @@ spec: sampling: 10 <2> ---- <1> Sends matching flows to a specific output, such as Loki, Prometheus, or an external system. When omitted, sends to all configured outputs. -<2> Optional. Applies a sampling ratio to limit the number of matching flows to be stored or exported. For example, `sampling: 10` means 1/10 of the flows are kept. \ No newline at end of file +<2> Optional. Applies a sampling ratio to limit the number of matching flows to be stored or exported. For example, `sampling: 10` means 1/10 of the flows are kept. + + diff --git a/modules/network-observability-flowcollector-api-specifications.adoc b/modules/network-observability-flowcollector-api-specifications.adoc index 0d2a0cde0ac2..d1af330342e2 100644 --- a/modules/network-observability-flowcollector-api-specifications.adoc +++ b/modules/network-observability-flowcollector-api-specifications.adoc @@ -154,7 +154,7 @@ is set to `eBPF`. | `type` | `string` -| `type` [deprecated *] selects the flows tracing agent. Previously, this field allowed to select between `eBPF` or `IPFIX`. +| `type` [deprecated (*)] selects the flows tracing agent. Previously, this field allowed to select between `eBPF` or `IPFIX`. Only `eBPF` is allowed now, so this field is deprecated and is planned for removal in a future version of the API. |=== @@ -180,7 +180,8 @@ Type:: | `object` | `advanced` allows setting some aspects of the internal configuration of the eBPF agent. This section is aimed mostly for debugging and fine-grained performance optimizations, -such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. +such as `GOGC` and `GOMAXPROCS` environment vars. Set these values at your own risk. You can also +override the default Linux capabilities from there. | `cacheActiveTimeout` | `string` @@ -205,25 +206,28 @@ Otherwise it is matched as a case-sensitive string. | List of additional features to enable. They are all disabled by default. Enabling additional features might have performance impacts. Possible values are: + - `PacketDrop`: Enable the packets drop flows logging feature. This feature requires mounting -the kernel debug filesystem, so the eBPF agent pods must run as privileged. -If the `spec.agent.ebpf.privileged` parameter is not set, an error is reported. + +the kernel debug filesystem, so the eBPF agent pods must run as privileged via `spec.agent.ebpf.privileged`. + - `DNSTracking`: Enable the DNS tracking feature. + - `FlowRTT`: Enable flow latency (sRTT) extraction in the eBPF agent from TCP traffic. + - `NetworkEvents`: Enable the network events monitoring feature, such as correlating flows and network policies. -This feature requires mounting the kernel debug filesystem, so the eBPF agent pods must run as privileged. -It requires using the OVN-Kubernetes network plugin with the Observability feature. + -IMPORTANT: This feature is available as a Technology Preview. +This feature requires mounting the kernel debug filesystem, so the eBPF agent pods must run as privileged via `spec.agent.ebpf.privileged`. +It requires using the OVN-Kubernetes network plugin with the Observability feature. +IMPORTANT: This feature is available as a Technology Preview. + - `PacketTranslation`: Enable enriching flows with packet translation information, such as Service NAT. + -- `EbpfManager`: Unsupported * . Use eBPF Manager to manage Network Observability eBPF programs. Pre-requisite: the eBPF Manager operator (or upstream bpfman operator) must be installed. + +- `EbpfManager`: [Unsupported (*)]. Use eBPF Manager to manage Network Observability eBPF programs. Pre-requisite: the eBPF Manager operator (or upstream bpfman operator) must be installed. + + +- `UDNMapping`: Enable interfaces mapping to User Defined Networks (UDN). + + +This feature requires mounting the kernel debug filesystem, so the eBPF agent pods must run as privileged via `spec.agent.ebpf.privileged`. +It requires using the OVN-Kubernetes network plugin with the Observability feature. + + +- `IPSec`, to track flows between nodes with IPsec encryption. + -- `UDNMapping`: Unsupported *. Enable interfaces mapping to User Defined Networks (UDN). + -This feature requires mounting the kernel debug filesystem, so the eBPF agent pods must run as privileged. -It requires using the OVN-Kubernetes network plugin with the Observability feature. | `flowFilter` | `object` @@ -255,7 +259,7 @@ Otherwise it is matched as a case-sensitive string. | `privileged` | `boolean` | Privileged mode for the eBPF Agent container. When ignored or set to `false`, the operator sets -granular capabilities (BPF, PERFMON, NET_ADMIN, SYS_RESOURCE) to the container. +granular capabilities (BPF, PERFMON, NET_ADMIN) to the container. If for some reason these capabilities cannot be set, such as if an old kernel version not knowing CAP_BPF is in use, then you can turn on this mode for more global privileges. Some agent features require the privileged mode, such as packet drops tracking (see `features`) and SR-IOV support. @@ -267,7 +271,7 @@ For more information, see https://kubernetes.io/docs/concepts/configuration/mana | `sampling` | `integer` -| Sampling rate of the flow reporter. 100 means one flow on 100 is sent. 0 or 1 means all flows are sampled. +| Sampling ratio of the eBPF probe. 100 means one packet on 100 is sent. 0 or 1 means all packets are sampled. |=== == .spec.agent.ebpf.advanced @@ -276,7 +280,8 @@ Description:: -- `advanced` allows setting some aspects of the internal configuration of the eBPF agent. This section is aimed mostly for debugging and fine-grained performance optimizations, -such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. +such as `GOGC` and `GOMAXPROCS` environment vars. Set these values at your own risk. You can also +override the default Linux capabilities from there. -- Type:: @@ -289,6 +294,10 @@ Type:: |=== | Property | Type | Description +| `capOverride` +| `array (string)` +| Linux capabilities override, when not running as privileged. Default capabilities are BPF, PERFMON and NET_ADMIN. + | `env` | `object (string)` | `env` allows passing custom environment variables to underlying components. Useful for passing @@ -447,7 +456,7 @@ To change the default, you can define a rule that accepts everything: `{ action: | `sampling` | `integer` -| `sampling` sampling rate for the matched flows, overriding the global sampling defined at `spec.agent.ebpf.sampling`. +| `sampling` is the sampling ratio for the matched packets, overriding the global sampling defined at `spec.agent.ebpf.sampling`. | `sourcePorts` | `integer-or-string` @@ -481,7 +490,7 @@ Type:: Description:: + -- -`EBPFFlowFilterRules` defines the desired eBPF agent configuration regarding flow filtering rules. +`EBPFFlowFilterRule` defines the desired eBPF agent configuration regarding flow filtering rule. -- Type:: @@ -549,7 +558,7 @@ To filter two ports, use a "port1,port2" in string format. For example, `ports: | `sampling` | `integer` -| `sampling` sampling rate for the matched flows, overriding the global sampling defined at `spec.agent.ebpf.sampling`. +| `sampling` is the sampling ratio for the matched packets, overriding the global sampling defined at `spec.agent.ebpf.sampling`. | `sourcePorts` | `integer-or-string` @@ -661,7 +670,7 @@ If set to `true`, the `providedCaFile` field is ignored. | Select the type of TLS configuration: + - `Disabled` (default) to not configure TLS for the endpoint. -- `Provided` to manually provide cert file and a key file. Unsupported *. +- `Provided` to manually provide cert file and a key file. [Unsupported (*)]. - `Auto` to use {product-title} auto generated certificate using annotations. |=== @@ -791,7 +800,7 @@ Type:: | `object` | `advanced` allows setting some aspects of the internal configuration of the console plugin. This section is aimed mostly for debugging and fine-grained performance optimizations, -such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. +such as `GOGC` and `GOMAXPROCS` environment vars. Set these values at your own risk. | `autoscaler` | `object` @@ -833,7 +842,7 @@ Description:: -- `advanced` allows setting some aspects of the internal configuration of the console plugin. This section is aimed mostly for debugging and fine-grained performance optimizations, -such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. +such as `GOGC` and `GOMAXPROCS` environment vars. Set these values at your own risk. -- Type:: @@ -1165,7 +1174,7 @@ Required:: | `sasl` | `object` -| SASL authentication configuration. Unsupported *. +| SASL authentication configuration. [Unsupported (*)]. | `tls` | `object` @@ -1180,7 +1189,7 @@ Required:: Description:: + -- -SASL authentication configuration. Unsupported *. +SASL authentication configuration. [Unsupported (*)]. -- Type:: @@ -1676,7 +1685,7 @@ Required:: | `sasl` | `object` -| SASL authentication configuration. Unsupported *. +| SASL authentication configuration. [Unsupported (*)]. | `tls` | `object` @@ -1691,7 +1700,7 @@ Required:: Description:: + -- -SASL authentication configuration. Unsupported *. +SASL authentication configuration. [Unsupported (*)]. -- Type:: @@ -2077,7 +2086,7 @@ Type:: - `Forward` forwards the user token for authorization. + -- `Host` [deprecated *] - uses the local pod service account to authenticate to Loki. + +- `Host` [deprecated (*)] - uses the local pod service account to authenticate to Loki. + When using the Loki Operator, this must be set to `Forward`. @@ -2693,7 +2702,7 @@ This feature requires the "topology.kubernetes.io/zone" label to be set on nodes | `object` | `advanced` allows setting some aspects of the internal configuration of the flow processor. This section is aimed mostly for debugging and fine-grained performance optimizations, -such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. +such as `GOGC` and `GOMAXPROCS` environment vars. Set these values at your own risk. | `clusterName` | `string` @@ -2702,14 +2711,12 @@ such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. | `deduper` | `object` | `deduper` allows you to sample or drop flows identified as duplicates, in order to save on resource usage. -Unsupported *. | `filters` | `array` | `filters` lets you define custom filters to limit the amount of generated flows. These filters provide more flexibility than the eBPF Agent filters (in `spec.agent.ebpf.flowFilter`), such as allowing to filter by Kubernetes namespace, but with a lesser improvement in performance. -Unsupported *. | `imagePullPolicy` | `string` @@ -2743,9 +2750,9 @@ This setting is ignored when Kafka is disabled. - `Flows` to export regular network flows. This is the default. + -- `Conversations` to generate events for started conversations, ended conversations as well as periodic "tick" updates. + +- `Conversations` to generate events for started conversations, ended conversations as well as periodic "tick" updates. Note that in this mode, Prometheus metrics are not accurate on long-standing conversations. + -- `EndedConversations` to generate only ended conversations events. + +- `EndedConversations` to generate only ended conversations events. Note that in this mode, Prometheus metrics are not accurate on long-standing conversations. + - `All` to generate both network flows and all conversations events. It is not recommended due to the impact on resources footprint. + @@ -2775,7 +2782,7 @@ Description:: -- `advanced` allows setting some aspects of the internal configuration of the flow processor. This section is aimed mostly for debugging and fine-grained performance optimizations, -such as `GOGC` and `GOMAXPROCS` env vars. Set these values at your own risk. +such as `GOGC` and `GOMAXPROCS` environment vars. Set these values at your own risk. -- Type:: @@ -2803,7 +2810,7 @@ This delay is ignored when a FIN packet is collected for TCP flows (see `convers | `dropUnusedFields` | `boolean` -| `dropUnusedFields` [deprecated *] this setting is not used anymore. +| `dropUnusedFields` [deprecated (*)] this setting is not used anymore. | `enableKubeProbes` | `boolean` @@ -2910,7 +2917,8 @@ Description:: + -- Defines secondary networks to be checked for resources identification. -To guarantee a correct identification, indexed values must form an unique identifier across the cluster. If the same index is used by several resources, those resources might be incorrectly labeled. +To guarantee a correct identification, indexed values must form an unique identifier across the cluster. +If the same index is used by several resources, those resources might be incorrectly labeled. -- Type:: @@ -2955,7 +2963,6 @@ Description:: + -- `deduper` allows you to sample or drop flows identified as duplicates, in order to save on resource usage. -Unsupported *. -- Type:: @@ -2970,7 +2977,7 @@ Type:: | `mode` | `string` -| Set the Processor de-duplication mode. It comes in addition to the Agent-based deduplication because the Agent cannot de-duplicate same flows reported from different nodes. + +| Set the Processor de-duplication mode. It comes in addition to the Agent-based deduplication, since the Agent cannot de-duplicate same flows reported from different nodes. + - Use `Drop` to drop every flow considered as duplicates, allowing saving more on resource usage but potentially losing some information such as the network interfaces used from peer, or network events. + @@ -2981,7 +2988,7 @@ Type:: | `sampling` | `integer` -| `sampling` is the sampling rate when deduper `mode` is `Sample`. +| `sampling` is the sampling ratio when deduper `mode` is `Sample`. For example, a value of `50` means that 1 flow in 50 is sampled. |=== == .spec.processor.filters @@ -2991,7 +2998,6 @@ Description:: `filters` lets you define custom filters to limit the amount of generated flows. These filters provide more flexibility than the eBPF Agent filters (in `spec.agent.ebpf.flowFilter`), such as allowing to filter by Kubernetes namespace, but with a lesser improvement in performance. -Unsupported *. -- Type:: @@ -3017,64 +3023,17 @@ Type:: |=== | Property | Type | Description -| `allOf` -| `array` -| `filters` is a list of matches that must be all satisfied in order to remove a flow. - | `outputTarget` | `string` -| If specified, these filters only target a single output: `Loki`, `Metrics` or `Exporters`. By default, all outputs are targeted. - -| `sampling` -| `integer` -| `sampling` is an optional sampling rate to apply to this filter. - -|=== -== .spec.processor.filters[].allOf -Description:: -+ --- -`filters` is a list of matches that must be all satisfied in order to remove a flow. --- - -Type:: - `array` - - - - -== .spec.processor.filters[].allOf[] -Description:: -+ --- -`FLPSingleFilter` defines the desired configuration for a single FLP-based filter. --- - -Type:: - `object` - -Required:: - - `field` - - `matchType` - - - -[cols="1,1,1",options="header"] -|=== -| Property | Type | Description - -| `field` -| `string` -| Name of the field to filter on. -Refer to the documentation for the list of available fields: https://github.com/netobserv/network-observability-operator/blob/main/docs/flows-format.adoc. +| If specified, these filters target a single output: `Loki`, `Metrics` or `Exporters`. By default, all outputs are targeted. -| `matchType` +| `query` | `string` -| Type of matching to apply. +| A query that selects the network flows to keep. More information about this query language in https://github.com/netobserv/flowlogs-pipeline/blob/main/docs/filtering.md. -| `value` -| `string` -| Value to filter on. When `matchType` is `Equal` or `NotEqual`, you can use field injection with `$(SomeField)` to refer to any other field of the flow. +| `sampling` +| `integer` +| `sampling` is an optional sampling ratio to apply to this filter. For example, a value of `50` means that 1 matching flow in 50 is sampled. |=== == .spec.processor.kafkaConsumerAutoscaler @@ -3199,7 +3158,7 @@ If set to `true`, the `providedCaFile` field is ignored. | Select the type of TLS configuration: + - `Disabled` (default) to not configure TLS for the endpoint. -- `Provided` to manually provide cert file and a key file. Unsupported *. +- `Provided` to manually provide cert file and a key file. [Unsupported (*)]. - `Auto` to use {product-title} auto generated certificate using annotations. |=== @@ -3593,4 +3552,4 @@ If the namespace is different, the config map or the secret is copied so that it | `string` | Type for the certificate reference: `configmap` or `secret`. -|=== \ No newline at end of file +|=== diff --git a/modules/network-observability-flowmetric-api-specifications.adoc b/modules/network-observability-flowmetric-api-specifications.adoc index c9adb00fb33c..155f55dc3bea 100644 --- a/modules/network-observability-flowmetric-api-specifications.adoc +++ b/modules/network-observability-flowmetric-api-specifications.adoc @@ -103,13 +103,12 @@ When set to `Egress`, it is equivalent to adding the regular expression filter o | `filters` | `array` -| `filters` is a list of fields and values used to restrict which flows are taken into account. Oftentimes, these filters must -be used to eliminate duplicates: `Duplicate != "true"` and `FlowDirection = "0"`. +| `filters` is a list of fields and values used to restrict which flows are taken into account. Refer to the documentation for the list of available fields: https://docs.openshift.com/container-platform/latest/observability/network_observability/json-flows-format-reference.html. | `flatten` | `array (string)` -| `flatten` is a list of list-type fields that must be flattened, such as Interfaces and NetworkEvents. Flattened fields generate one metric per item in that field. +| `flatten` is a list of array-type fields that must be flattened, such as Interfaces or NetworkEvents. Flattened fields generate one metric per item in that field. For instance, when flattening `Interfaces` on a bytes counter, a flow having Interfaces [br-ex, ens5] increases one counter for `br-ex` and another for `ens5`. | `labels` @@ -131,9 +130,10 @@ Refer to the documentation for the list of available fields: https://docs.opensh | `type` | `string` -| Metric type: "Counter" or "Histogram". +| Metric type: "Counter", "Histogram" or "Gauge". Use "Counter" for any value that increases over time and on which you can compute a rate, such as Bytes or Packets. Use "Histogram" for any value that must be sampled independently, such as latencies. +Use "Gauge" for other values that don't necessitate accuracy over time (gauges are sampled only every N seconds when Prometheus fetches the metric). | `valueField` | `string` @@ -261,8 +261,7 @@ To learn more about `promQL`, refer to the Prometheus documentation: https://pro Description:: + -- -`filters` is a list of fields and values used to restrict which flows are taken into account. Oftentimes, these filters must -be used to eliminate duplicates: `Duplicate != "true"` and `FlowDirection = "0"`. +`filters` is a list of fields and values used to restrict which flows are taken into account. Refer to the documentation for the list of available fields: https://docs.openshift.com/container-platform/latest/observability/network_observability/json-flows-format-reference.html. -- diff --git a/modules/network-observability-flows-format.adoc b/modules/network-observability-flows-format.adoc index a81303aa2b6c..854796311138 100644 --- a/modules/network-observability-flows-format.adoc +++ b/modules/network-observability-flows-format.adoc @@ -155,16 +155,9 @@ The "Cardinality" column gives information about the implied metric cardinality | no | fine | n/a -| `Duplicate` -| boolean -| Indicates if this flow was also captured from another interface on the same host -| n/a -| no -| fine -| n/a | `Flags` | string[] -| List of TCP flags comprised in the flow, according to RFC-9293, with additional custom flags to represent the following per-packet combinations: + +| List of TCP flags comprised in the flow, as per RFC-9293, with additional custom flags to represent the following per-packet combinations: + - SYN_ACK + - FIN_ACK + - RST_ACK @@ -182,6 +175,13 @@ The "Cardinality" column gives information about the implied metric cardinality | yes | fine | host.direction +| `IPSecStatus` +| string +| Status of the IPsec encryption (on egress, given by the kernel xfrm_output function) or decryption (on ingress, via xfrm_input) +| `ipsec_status` +| no +| fine +| n/a | `IcmpCode` | number | ICMP code @@ -242,7 +242,7 @@ The "Cardinality" column gives information about the implied metric cardinality | `Packets` | number | Number of packets -| `pkt_drop_cause` +| n/a | no | avoid | packets @@ -423,35 +423,35 @@ The "Cardinality" column gives information about the implied metric cardinality | n/a | `XlatDstAddr` | string -| Packet translation destination address +| packet translation destination address | `xlat_dst_address` | no | avoid | n/a | `XlatDstPort` | number -| Packet translation destination port +| packet translation destination port | `xlat_dst_port` | no | careful | n/a | `XlatSrcAddr` | string -| Packet translation source address +| packet translation source address | `xlat_src_address` | no | avoid | n/a | `XlatSrcPort` | number -| Packet translation source port +| packet translation source port | `xlat_src_port` | no | careful | n/a | `ZoneId` | number -| Packet translation zone id +| packet translation zone id | `xlat_zone_id` | no | avoid @@ -470,4 +470,4 @@ The "Cardinality" column gives information about the implied metric cardinality | yes | fine | n/a -|=== \ No newline at end of file +|=== \ No newline at end of file