Merge branch 'master' into http_common

Author: urso

CHANGELOG.next.asciidoc

@@ -236,6 +236,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix ILM alias creation when write alias exists and initial index does not exist {pull}26143[26143]
 - Omit full index template from errors that occur while loading the template. {pull}25743[25743]
 - In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively.
+- Fix encoding errors when using the disk queue on nested data with multi-byte characters {pull}26484[26484]
 
 *Auditbeat*
 
@@ -819,6 +820,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update PanOS module to parse HIP Match logs. {issue}24350[24350] {pull}25686[25686]
 - Support MongoDB 4.4 in filebeat's MongoDB module. {issue}20501[20501] {pull}24774[24774]
 - Enhance GCP module to populate orchestrator.* fields for GKE / K8S logs {pull}25368[25368]
+- Add log_group_name_prefix config into aws-cloudwatch input. {pull}26187[26187]
 - Move Filebeat azure module to GA. {pull}26114[26114] {pull}26168[26168]
 - http_endpoint: Support multiple documents in a single request by POSTing an array or NDJSON format. {pull}25764[25764]
 - Make `filestream` input GA. {pull}26127[26127]
@@ -835,6 +837,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]
 - Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
 - Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
+- Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
 
 *Heartbeat*
 
@@ -843,6 +846,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Journalbeat*
 
+- Suppress too many bad message error logs when reading from corrupted journal for 5 seconds. {pull}26224[26224]
 
 *Metricbeat*
 

dev-tools/packaging/templates/docker/Dockerfile.tmpl

@@ -47,7 +47,7 @@ ENV NODE_PATH={{ $beatHome }}/.node
 RUN echo \
     $NODE_PATH \
     {{ $beatHome }}/.config \
-    {{ $beatHome }}/suites \
+    {{ $beatHome }}/.synthetics \
     {{ $beatHome }}/.npm \
     {{ $beatHome }}/.cache \
     | xargs -IDIR sh -c 'mkdir -p DIR && chmod 0770 DIR'

filebeat/docs/fields.asciidoc

@@ -152208,6 +152208,17 @@ example: Montreal
 
 --
 
+*`threatintel.indicator.geo.continent_name`*::
++
+--
+Name of the continent.
+
+type: keyword
+
+example: North America
+
+--
+
 *`threatintel.indicator.geo.country_iso_code`*::
 +
 --
@@ -153643,6 +153654,166 @@ type: keyword
 
 --
 
+[float]
+=== recordedfuture
+
+Fields for Recorded Future Threat Intel
+
+
+
+[float]
+=== entity
+
+Entity that represents a threat.
+
+
+
+*`threatintel.recordedfuture.entity.id`*::
++
+--
+Entity ID.
+
+
+type: keyword
+
+example: ip:192.0.2.13
+
+--
+
+*`threatintel.recordedfuture.entity.name`*::
++
+--
+Entity name. Value for the entity.
+
+
+type: keyword
+
+example: 192.0.2.13
+
+--
+
+*`threatintel.recordedfuture.entity.type`*::
++
+--
+Entity type.
+
+
+type: keyword
+
+example: IpAddress
+
+--
+
+*`threatintel.recordedfuture.intelCard`*::
++
+--
+Link to the Recorded Future Intelligence Card for to this indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.recordedfuture.ip_range`*::
++
+--
+Range of IPs for this indicator.
+
+
+type: ip_range
+
+example: 192.0.2.0/16
+
+--
+
+[float]
+=== risk
+
+Risk fields.
+
+
+
+*`threatintel.recordedfuture.risk.criticality`*::
++
+--
+Risk criticality (0-4).
+
+
+type: byte
+
+--
+
+*`threatintel.recordedfuture.risk.criticalityLabel`*::
++
+--
+Risk criticality label. One of None, Unusual, Suspicious, Malicious, Very Malicious.
+
+
+type: keyword
+
+--
+
+*`threatintel.recordedfuture.risk.evidenceDetails`*::
++
+--
+Risk's evidence details.
+
+
+type: flattened
+
+--
+
+*`threatintel.recordedfuture.risk.score`*::
++
+--
+Risk score (0-99).
+
+
+type: short
+
+--
+
+*`threatintel.recordedfuture.risk.riskString`*::
++
+--
+Number of Risk Rules observed as a factor of total number of rules.
+
+
+type: keyword
+
+example: 1/54
+
+--
+
+*`threatintel.recordedfuture.risk.riskSummary`*::
++
+--
+Risk summary.
+
+
+type: keyword
+
+example: 1 of 54 Risk Rules currently observed.
+
+--
+
+*`threatintel.recordedfuture.risk.riskSummary.text`*::
++
+--
+type: text
+
+--
+
+*`threatintel.recordedfuture.risk.rules`*::
++
+--
+Number of rules observed.
+
+
+type: long
+
+--
+
 [[exported-fields-tomcat]]
 == Apache Tomcat fields
 

filebeat/docs/index.asciidoc

@@ -16,7 +16,6 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :beat_default_index_prefix: {beatname_lc}
 :beat_kib_app: {kib} Logs
 :has_ml_jobs: yes
-:has_central_config:
 :has_solutions:
 :ignores_max_retries:
 :has_docker_label_ex:

filebeat/docs/modules/threatintel.asciidoc

@@ -29,6 +29,7 @@ The available filesets are:
 * <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
 * <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
 * <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
+* <<recordedfuture,recordedfuture>>: Supports gathering threat intel attributes from Recorded Future.
 
 include::../include/gs-link.asciidoc[]
 
@@ -223,7 +224,7 @@ How often the API is polled for updated information.
 
 *`var.first_interval`*::
 
-How far back to search when retrieving events the first time the beat starts up.
+How far back to search when retrieving events the first time {beatname_uc} starts up.
 After the first interval has passed the module itself will use the timestamp
 from the last response as the filter when retrieving new events.
 
@@ -297,7 +298,7 @@ How often the API is polled for updated information.
 
 *`var.first_interval`*::
 
-How far back to search when retrieving events the first time the beat starts up.
+How far back to search when retrieving events the first time the {beatname_uc} starts up.
 After the first interval has passed the module itself will use the timestamp
 from the last response as the filter when retrieving new events.
 
@@ -409,7 +410,7 @@ Anomali Threat Intel is mapped to the following ECS fields.
 
 To configure the ThreatStream integration you first need to define an output
 in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
-It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
+It will deliver indicators via HTTP or HTTPS to a {beatname_uc} instance running as
 a server.
 
 Configure an Integrator output with the following settings:
@@ -419,12 +420,12 @@ Configure an Integrator output with the following settings:
   Adjust the paths to the python executable and the directory where the Elastic SDK
   has been unpacked.
 * Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
-    - `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
+    - `url`: Use the host and port where {beatname_uc} will be running, and `http` or `https` accordingly.
     - `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
         this field.
-    - `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
+    - `secret`: A shared secret string to authenticate messages between the SDK and {beatname_uc}.
 
-Then configure the `anomalithreatstream` fileset in Filebeat accordingly:
+Then configure the `anomalithreatstream` fileset in {beatname_uc} accordingly:
 [source,yaml]
 ----
 - module: threatintel
@@ -449,11 +450,11 @@ Port number to use for the HTTP server.
 
 *`var.secret`*::
 
-Shared secret between the SDK and Filebeat, used to authenticate messages.
+Shared secret between the SDK and {beatname_uc}, used to authenticate messages.
 
 *`var.ssl_certificate`*::
 
-Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
+Path to the public SSL certificate for the HTTPS server. If unset, {beatname_uc}
 will use unsecure HTTP connections.
 
 *`var.ssl_key`*::
@@ -488,6 +489,94 @@ Anomali ThreatStream fields are mapped to the following ECS fields:
 [[a]]
 [small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
 
+[[recordedfuture]]
+[float]
+==== `recordedfuture` fileset settings
+
+The `recordedfuture` fileset fetches intelligence from the Recorded Future Connect API.
+It supports `domain`, `hash`, `ip` and `url` data types.
+
+To enable it you need to define the URL to fetch data from. You can construct this URL
+using the https://api.recordedfuture.com/index.html[Recorded Future API Explorer.] The URL
+must point to the `/search` endpoint and contain a suitable `limit`
+(how many records to return from a single request) and `fields` parameters.
+The `entity` and `timestamps` fields are required.
+
+Sample configuration:
+[source,yaml]
+----
+- module: threatintel
+  recordedfuture:
+    enabled: true
+    var.input: httpjson
+    var.interval: 5m
+    var.first_interval: 168h
+    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
+    var.api_token: "<RF_TOKEN>"
+----
+
+To fetch threat intelligence from multiple data types, you must define more than
+one instance of the module:
+[source,yaml]
+----
+- module: threatintel
+  recordedfuture:
+    enabled: true
+    var.input: httpjson
+    var.interval: 5m
+    var.first_interval: 168h
+    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
+    var.api_token: "<RF_TOKEN>"
+- module: threatintel
+  recordedfuture:
+    enabled: true
+    var.input: httpjson
+    var.interval: 1m
+    var.first_interval: 168h
+    var.url: "https://api.recordedfuture.com/v2/hash/search?limit=200&fields=entity,fileHashes,timestamps,risk,intelCard,location&metadata=false"
+    var.api_token: "<RF_TOKEN>"
+----
+
+*`var.url`*::
+
+The URL of the API endpoint to connect with.
+
+*`var.api_token`*::
+
+The API token used to access Recorded Future API.
+
+*`var.interval`*::
+
+How often the API is polled for updated information.
+
+*`var.first_interval`*::
+
+How far back to search when retrieving events the first time {beatname_uc} starts up.
+After the first interval has passed the module itself will use the timestamp
+from the last response as the filter when retrieving new events.
+
+*`var.proxy_url`*::
+
+Optional URL to use as HTTP proxy.
+
+
+Recorded Future fields are mapped to the following ECS fields:
+
+[options="header"]
+|=============================================================
+| Recorded Future fields | ECS Fields
+| entity.name            | threatintel.indicator.{url,ip,domain,file.hash}
+| entity.type            | threatintel.indicator.type
+| fileHashes             | threatintel.indicator.file.hash
+| intelCard              | event.reference
+| location.asn           | threatintel.indicator.as.number
+| location.location      | threatintel.indicator.geo
+| location.organization  | threatintel.indicator.as.organization.name
+| risk.score             | event.risk_score
+| timestamps.firstSeen   | threatintel.indicator.first_seen
+| timestamps.lastSeen    | threatintel.indicator.last_seen
+|=============================================================
+
 :has-dashboards!:
 
 [float]