[RFC] Threat Intel - Stage 1

rfcs/text/0008-threat-intel.md

@@ -1,7 +1,7 @@
 # 0008: Cyber Threat Intelligence Fields
 <!-- Leave this ID at 0000. The ECS team will assign a unique, contiguous RFC number upon merging the initial stage of this RFC. -->
 
-- Stage: **1 (proposal)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
+- Stage: **2 (proposal)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
 - Date: **2020-11-09** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->
 
 Elastic Security Solution will be adding the capability to ingest, process and utilize threat intelligence information for increasing detection coverage and helping analysts making quicker investigation decisions. Threat intelligence can be collected from a number of sources with a variety of structured and semi-structured data representations. This makes threat intelligence an ideal candidate for ECS mappings. Threat intelligence data will require ECS mappings to normalize it and make it usable in our security solution. This RFC is focused on identifying new field sets and values that need to be created for threat intelligence data. Existing ECS field reuse will be prioritized where possible. If new fields are required we will utilize [STIX Cyber Observable data model](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr) as guidance.
@@ -23,30 +23,36 @@ Stage 1: Describe at a high level how this change affects fields. Which fieldset
 
 ### Proposed New Fields for Threat fieldset
 
-  * threat.time_first_seen
-    _The date and time when intelligence souce first reported sighting this indicator._
-  * threat.time_last_seen
-    _The date and time when intelligence source last reported sighting this indicator._
-  * threat.sightings
-    _Number of times this indicator was observed conducting threat activity._
-  * threat.type
-    _Type of indicator as reprsented by Cyber Observable in STIX 2.0_
-  * threat.description
-    _Describes the type of action conducted by the threat._
-  * threat.tlp
-    _Traffic Light Protocol, which dictates sharing policies_
-  * threat.classification
-    _Describes type of threat delivery (Hacktool etc.) and family name.
-  * threat.scanner_stats
-    _Count of Anti virus/EDR that successfully detected malicious file or URL. Sources like VirusTotal, Reversing Labs often provide these statistics._
-
-### Proposed New Values
-
-  * event.kind:enrichment _Propose adding this value to capture outcome of this event. It could also appy to other types of contextual data such as directory services, IPAM data, asset lists._
-  * event.category:threat _Proposed threat.type field would be a subcategory for this value of event.category_
-  * event.type:indicator _Proposed value represents type of threat information. In future this could be extended to other STIX 2.0 Standard Data Objects like Actor, Infrastucture etc._
-
-### Existing ECS Fields For Nested Use in Threat.*
+Field | Type | Example | Description
+--- | --- | --- | ---
+threat.ioc.time_first_seen | date | 2020-12-01 | The date and time when intelligence souce first reported sighting this indicator
+threat.ioc.time_last_seen | date | 2020-12-02| The date and time when intelligence source last reported sighting this indicator.
+threat.ioc.sightings | long | 20 | Number of times this indicator was observed conducting threat activity
+threat.ioc.type | keyword | IPV4 | Type of indicator as reprsented by Cyber Observable in STIX 2.0
+threat.ioc.description | text | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat
+threat.marking.tlp | keyword | RED | Data markings represent restrictions, permissions, and other guidance for how data can be used and shared. Examples could be TLP (White, Green, Amber, Red).
+threat.ioc.scanner_stats | long | 4 | Count of Anti virus/EDR that successfully detected malicious file or URL. Sources like VirusTotal, Reversing Labs often provide these statistics.
+
+### Proposed New Values for Event Fieldset
+
+Field | New Value | Description
+--- | --- | ---
+event.kind | enrichment | Propose adding this value to capture the type of information this event contains and how it should be used. Threat intelligence will be used to enrich source events and signals. Enrichment could also appy to other types of contextual data sources (not just threat intelligence) such as directory services, IPAM data, asset lists.
+event.category | threat | Propose adding this value to represent a new category of event data
+event.type | indicator | Propose adding this value to be used as a sub-bucket of `event.category` to represent type of threat information. In future this could be extended to other STIX 2.0 Standard Data Objects like Actor, Infrastucture etc.
+
+### Using existing Event Fieldset
+Field | Type | Example | Description
+--- | ---| --- | ---
+event.reference | keyword | https://feodotracker.abuse.ch/ | URL to the intelligence source
+event.dataset | keyword | Feodo tracker | name of specific dataset from the intelligence source. Intelligence sources often provide multiple datasets - IP blocklist, File hash blocklist etc.
+event.provider | keyword | Abuse.ch | name of intelligence provider
+event.severity | long | 7 | severity provided by threat intelligence source
+event.risk_score | float | 10 | risk score provided by threat intelligence source
+event.original | keyword | 2020-10-29 19:16:38,181.120.29.49,80,2020-11-02,Heodo | raw intelligence event
+
+
+### Using existing ECS Fields to store IOC information
 
   * file.*
   * file.hash.*
@@ -75,7 +81,13 @@ Stage 3: Add or update all remaining field definitions. The list should now be e
 Stage 1: Describe at a high-level how these field changes will be used in practice. Real world examples are encouraged. The goal here is to understand how people would leverage these fields to gain insights or solve problems. ~1-3 paragraphs.
 -->
 
-The additions to threat.* fields and nested use will be used to represent data collected threat intelligence sources. A new rule type Indicator match will be introduced in 7.10 and ECS threat fields will enable a new category of detection alerts that matches incoming log and event data against threat intelligence sources and generates an alert. Additionally in the future we will be able to enrich events and alerts with context from threat intelligence to assist analysts in their investigative workflows.
+The additions described above will be used to enable cyber threat intelligence capabilities in Elastic Security solution. A new rule type Indicator match will be introduced in 7.10 and the propoosed ECS updates will enable a new category of detection alerts that match incoming log and event data against threat intelligence sources. Additionally in the future we will also develop enrichment flows that add context from threat intelligence to alerts and events to assist analysts in their investigative workflows.
+
+There are two primary uses for these fields.
+
+1. *Storing threat intelligence as an event document in threat index(s).* Threat intelligence data will collected from multiple sources stored in threat indices. The ECS fields proposed here will be used to structure the documents collected from various sources. 
+
+2. *Adding threat intelligence match/enrichment to another document which could be in a source event index or signals index.* The Indicator Match Rule will be used to generate signals when a match occurs between a source event and threat intelligence document. The ECS fields proposed here will be used to add the enrichment and threat intel context in the signal document. 
 
 ## Source data
 
@@ -97,7 +109,8 @@ Some examples of commercial intelligence include:
   * [Domain Tools](https://www.domaintools.com/products/api-integration/)
 
 #### Abuse.ch Feodo Tracker
-
+This dataset from Abuse.ch provides a list of bonet C&C servers associated with the Feodo malware family (Dridex, Emotet).
+[Abuse.ch Feodo Tracker](https://feodotracker.abuse.ch/downloads/ipblocklist.csv)
 ```
 # Firstseen,DstIP,DstPort,LastOnline,Malware
 2020-10-29 19:16:38,181.120.29.49,80,2020-11-02,Heodo
@@ -114,6 +127,92 @@ Some examples of commercial intelligence include:
 Stage 2: Included a real world example source document. Ideally this example comes from the source(s) identified in stage 1. If not, it should replace them. The goal here is to validate the utility of these field changes in the context of a real world example. Format with the source name as a ### header and the example document in a GitHub code block with json formatting.
 -->
 
+#### Botvrij.eu
+
+Freely available source of IOC's which includes Network IOCs, File Details, Email and Registry Key
+[https://botvrij.eu/data/](https://botvrij.eu/data/)
+
+```
+cc2477cf4d596a88b349257cba3ef356 # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+573ff02981a5c70ae6b2594b45aa7caa # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+c961a3e3bd646ed0732e867310333978 # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+2a98e06c3310309c58fb149a8dc7392c # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+f21c21c2fceac5118ebf088653275b4f # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+0ae37532a7bbce03e7686eee49441c41 # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+974b6559a6b45067b465050e5002214b # md5 - AZORult spreads as a fake ProtonVPN installer (191)
+7966c2c546b71e800397a67f942858d0 # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194)
+5909983db4d9023e4098e56361c96a6f # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194)
+3e856162c36b532925c8226b4ed3481c # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194)
+659bd19b562059f3f0cc978e15624fd9 # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194)
+
+```
+#### AlienVault OTX
+
+Rest Endpoint: `/api/v1/indicators/export`
+
+Schema
+```
+{
+  "$schema": "http://json-schema.org/draft-04/schema",
+  "additionalProperties": false,
+  "required": ["count", "next", "results", "previous"],
+  "properties": {
+    "count": {"type": "integer"},
+    "next": {"type": ["string", "null"]},
+    "results": {
+        "type": "array", 
+        "items": {
+            "additionalProperties": false,
+            "required": ["indicator", "title", "content", "type", "id", "description"],
+            "properties": {
+                "indicator": {"type": "string"},
+                "title": {"type": ["string", "null"]},
+                "content": {"type": ["string", "null"]},
+                "type": {"type": "string"},
+                "id": {"type": "integer"},
+                "description": {"type": ["string", "null"]}
+            }
+        }
+    },
+    "previous": {"type": ["string", "null"]}
+  } 
+}
+```
+
+Example
+```
+{
+    "count": 3,
+    "next": null,
+    "results": [
+        {
+            "indicator": "rustybrooks.com",
+            "description": null,
+            "title": null,
+            "content": "",
+            "type": "domain",
+            "id": 1
+        },
+        {
+            "indicator": "roll20.com",
+            "description": null,
+            "title": null,
+            "content": "",
+            "type": "domain",
+            "id": 3
+        },
+        {
+            "indicator": "redacted.ch",
+            "description": null,
+            "title": null,
+            "content": "",
+            "type": "domain",
+            "id": 6
+        }
+    ],
+    "previous": null
+}
+```
 <!--
 Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2.
 -->
@@ -127,13 +226,20 @@ Stage 2: Identifies scope of impact of changes. Are breaking changes required? S
  * ECS project (e.g. docs, tooling)
 The goal here is to research and understand the impact of these changes on users in the community and development teams across Elastic. 2-5 sentences each.
 -->
+ * Ingestion mechanism: Primary ingestion mechanisms will be Filebeat modules and Ingest Packages. There will be no impact on ingestion mechanisms.
+ * Usage mechanism: The primary use of the proposed ECS fields and values is through Elastic Security solution. In 7.10 we released Indicator match rule to support the use of the proposed new fields and values.
 
 ## Concerns
 
 <!--
 Stage 1: Identify potential concerns, implementation challenges, or complexity. Spend some time on this. Play devil's advocate. Try to identify the sort of non-obvious challenges that tend to surface later. The goal here is to surface risks early, allow everyone the time to work through them, and ultimately document resolution for posterity's sake.
 -->
-There is a proposal to nest all IoC fields under `threat.ioc.*` instead of the current `threat.* structure.` This would make it consistent with taxonomy structure for `threat.tactic.*` and `threat.techinique.*` . This needs to be resovled in Stage 2 of the RFC process.
+
+1. Identified in Stage 1: There is a proposal to nest all IoC fields under `threat.ioc.*` instead of the current `threat.* structure.` This would make it consistent with taxonomy structure for `threat.tactic.*` and `threat.techinique.*` . This needs to be resovled in Stage 2 of the RFC process.
+ * Proposed resolution: Nest all IoC fields under `threat.ioc.*`
+2. How to use `event.module`
+ * Proposed resolution: pending
+3. How to best represent malware{name,family,type}. Current proposal is to use `threat.ioc.classification` to describe threat delivery (Hacktool etc.) and family name.
 
 <!--
 Stage 2: Document new concerns or resolutions to previously listed concerns. It's not critical that all concerns have resolutions at this point, but it would be helpful if resolutions were taking shape for the most significant concerns.
@@ -164,7 +270,10 @@ The following are the people that consulted on the contents of this RFC.
 
 ## References
 
-[Threat Intel Field Set Draft](https://docs.google.com/spreadsheets/d/1hS3tF-sGmwnKb7uUGLo3Rng_q6EFgwo6UCae8Sp4E-g/edit?usp=sharing)
+* [Threat Intel Field Set Draft](https://docs.google.com/spreadsheets/d/1hS3tF-sGmwnKb7uUGLo3Rng_q6EFgwo6UCae8Sp4E-g/edit?usp=sharing)
+* [Abuse.ch Feodo Tracker](https://feodotracker.abuse.ch/downloads/ipblocklist.csv)
+* [https://botvrij.eu/data/](https://botvrij.eu/data/)
+* [STIX Cyber Observable data model](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr)
 <!-- Insert any links appropriate to this RFC in this section. -->
 
 ### RFC Pull Requests
@@ -174,6 +283,7 @@ The following are the people that consulted on the contents of this RFC.
 * Stage 0: https://github.com/elastic/ecs/pull/986
 * Stage 1: https://github.com/elastic/ecs/pull/1037
   * Stage 1 correction: https://github.com/elastic/ecs/pull/1100
+* Stage 2: https://github.com/elastic/ecs/pull/1127
 
 <!--
 * Stage 1: https://github.com/elastic/ecs/pull/NNN