Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FireEye ETP Event Collector #28863

Merged
merged 23 commits into from
Aug 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Packs/FireEyeETP/.secrets-ignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
[email protected]
[email protected]
https://docs.trellix.com

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
category: Email
sectionOrder:
- Connect
- Collect
commonfields:
id: FireEye ETP Event Collector
version: -1
configuration:
- defaultvalue: https://etp.us.fireeye.com
display: Server URL (e.g., https://etp.us.fireeye.com)
name: url
required: true
type: 0
section: Connect
- displaypassword: API Key
additionalinfo: The API Key allows you to integrate with the FireEye ETP.
name: credentials
required: true
hiddenusername: true
type: 9
- defaultvalue: "1000"
display: Maximum number of Alerts to fetch.
name: alerts_max_fetch
type: 0
additionalinfo: The maximum number of Alert events to fetch from FireEye ETP.
section: Collect
- defaultvalue: "1000"
display: Maximum number of Email Trace to fetch.
name: email_trace_max_fetch
type: 0
additionalinfo: The maximum number of Email Trace events to fetch from FireEye ETP.
section: Collect
- defaultvalue: "1000"
display: Maximum number of Activity Log fetch.
name: activity_log_max_fetch
type: 0
additionalinfo: The maximum number of Activity Log events to fetch from FireEye ETP.
section: Collect
- display: Trust any certificate (not secure)
name: insecure
type: 8
section: Connect
advanced: true
- display: Use system proxy settings
name: proxy
type: 8
section: Connect
advanced: true
- display: Fetch outbound traffic
name: outbound_traffic
type: 8
section: Collect
advanced: true
additionalinfo: Outbound traffic will be fetched in addition to inbound traffic of the event type. When enabled, the maximum number of events to fetch corresponds to outbound and inbound separately.
- display: Hide sensitive details from email
name: hide_sensitive
type: 8
section: Collect
advanced: true
defaultvalue: 'true'
additionalinfo: Hide subject and attachments details from emails.
description: Use this integration to fetch email security incidents from FireEye ETP as XSIAM events.
display: FireEye ETP Event Collector
name: FireEye ETP Event Collector
script:
commands:
- arguments:
- description: The number of events to return.
name: limit
defaultValue: 10
- defaultValue: 3 days
description: The start time by which to filter events. Date format will be the same as in the first_fetch parameter.
name: since_time
- auto: PREDEFINED
defaultValue: "false"
description: Set this argument to True in order to create events, otherwise the command will only display them.
name: should_push_events
predefined:
- "true"
- "false"
required: true
description: Gets events from FireEye ETP. This command is used for developing/ debugging and is to be used with caution, as it can create events, leading to events duplication and API request limitation exceeding.
name: fireeye-etp-get-events
dockerimage: demisto/python3:3.10.12.68714
isfetchevents: true
script: ""
subtype: python3
type: python
fromversion: 8.2.0
marketplaces:
- marketplacev2
tests:
- No tests (auto formatted)
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
FireEye Email Threat Prevention Event Collector allows you to fetch events of type Alert, Email Trace ad Activity Log from your system.

# Configuring API keys
Follow these steps to configure API keys:
1. Log in to the Email Security — Cloud Web Portal or IAM console.
2. Click **My Settings** in the top navigation bar.
3. Click the **API Keys** tab in the IAM console.
4. Click **Create API Key**.
5. On the Manage API Key page, specify the following:
- API key name.
- Expiration time for the API key. The expiration time of API keys should be set as “100d” for 100 days, or “1y” for 1 year, for example.
- Products. Select both “Email Threat Prevention” and “Identity Access Management”.
6. Select all entitlements as shown below.
7. To download or copy an API key, click the download or copy icon in the bottom right corner.
8. Click **Create API Key**.

# Permissions
For any API access, the following entitlements are required:
- iam.users.browse
- iam.orgs.self.read
- etp.alerts.read (For accessing alerts APIs)
- etp.email_trace.read (For accessing trace APIs)

## Cloud service regions
Use the URLs for the region that hosts your Email Security — Cloud service:

- US Instance: https://etp.us.fireeye.com/
- EMEA Instance: https://etp.eu.fireeye.com
- APJ Instance: https://etp.ap.fireeye.com/
- USGOV Instance: https://etp.us.fireeyegov.com/
- CA Instance: https://etp.ca.fireeye.com/

For more information, see the [Official Product Documentation](https://docs.trellix.com/bundle/etp_api/page/UUID-98fd1a2c-382d-130b-00c5-b9be402fe660_1.html#idm44910884288000).
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
from datetime import datetime, timezone, timedelta
import json
import pytest
import FireEyeETPEventCollector
from freezegun import freeze_time


def util_load_json(path):
with open(path, encoding='utf-8') as f:
return json.loads(f.read())


LAST_RUN_MULTIPLE_EVENT = {'Last Run': {
"alerts": {
"last_fetch_last_ids": ['a', 'b'],
'last_fetch_timestamp': '2023-07-19T12:37:00.028000'
},
"email_trace": {
"last_fetch_last_ids": [],
'last_fetch_timestamp': '2023-07-19T12:20:00.020000'
},
"activity_log": {
"last_fetch_last_ids": [],
'last_fetch_timestamp': '2023-07-19T12:20:00.020000'
}

}}
LAST_RUN_ONE_EVENT = {'Last Run': {
"alerts": {
"last_fetch_last_ids": ['a', 'b'],
'last_fetch_timestamp': '2023-07-19T12:37:00.028000'
},
}}

LAST_RUN_EMPTY: dict = {}
LAST_RUN_DICT_CASES = [
(LAST_RUN_MULTIPLE_EVENT, # case when multiple events exists.
[
FireEyeETPEventCollector.EventType('alerts', 25, outbound=False),
FireEyeETPEventCollector.EventType('email_trace', 25, outbound=False),
FireEyeETPEventCollector.EventType('activity_log', 25, outbound=False),
],
LAST_RUN_MULTIPLE_EVENT), # expected
(LAST_RUN_ONE_EVENT, # case when only one event exists
[FireEyeETPEventCollector.EventType('alerts', 25, outbound=False)],
LAST_RUN_ONE_EVENT) # expected

]


@ pytest.mark.parametrize('last_run_dict, event_types_to_run, expected', LAST_RUN_DICT_CASES)
def test_last_run(last_run_dict, event_types_to_run, expected):
"""
Given: mocked last run dictionary and events to fetch
When: trying to fetch events
Then: validate last run creation and save.
"""
last_run = FireEyeETPEventCollector.get_last_run_from_dict(last_run_dict, event_types_to_run)
assert len(last_run.event_types) == len(expected.get('Last Run', {}))
assert {e.name for e in event_types_to_run} - set(last_run.__dict__.keys()) == set()
new_dict = last_run.to_demisto_last_run()
assert new_dict['Last Run'].keys() == expected['Last Run'].keys()


def mock_client():
return FireEyeETPEventCollector.Client(
base_url='test.com',
verify_certificate=False,
proxy=False,
api_key='api-key',
outbound_traffic=False,
hide_sensitive=True
)


@ freeze_time("2023-07-18 11:34:30")
@ pytest.mark.parametrize('hide_sensitive, alert_expected, trace_expected, activity_expected', (
pytest.param(True,
'formatted_response_hidden_true', 'formatted_response_hidden_true',
'formatted_response', id="Hide sensitive"),
pytest.param(False,
'formatted_response_hidden_false', 'formatted_response_hidden_false',
'formatted_response', id="Do not hide sensitive")
))
def test_fetch_alerts(mocker, hide_sensitive, alert_expected, trace_expected, activity_expected):
"""
Given: mocked client, mocked responses and expected event structure,
When: fetching incidents
Then: Testing the formatted events are as required.
"""
client = mock_client()
client.hide_sensitive = hide_sensitive
mocked_alert_data = util_load_json('test_data/alerts.json')
mocked_trace_data = util_load_json('test_data/email_trace.json')
mocked_activity_data = util_load_json('test_data/activity_log.json')
event_types_to_run = [
FireEyeETPEventCollector.EventType('alerts', 25, outbound=False),
FireEyeETPEventCollector.EventType('email_trace', 1000, outbound=False),
FireEyeETPEventCollector.EventType('activity_log', 25, outbound=False)
]
collector = FireEyeETPEventCollector.EventCollector(client, event_types_to_run)
mocker.patch.object(FireEyeETPEventCollector.Client, 'get_alerts', side_effect=[
mocked_alert_data['ok_response_single_data'], {'data': []}])
mocker.patch.object(FireEyeETPEventCollector.Client, 'get_email_trace', side_effect=[
mocked_trace_data['ok_response_single_data'], {'data': []}])
mocker.patch.object(FireEyeETPEventCollector.Client, 'get_activity_log', side_effect=[
mocked_activity_data['ok_response'], {'data': []}])
next_run, events = collector.fetch_command(
demisto_last_run=LAST_RUN_MULTIPLE_EVENT,
first_fetch=datetime.now(),
)
assert events[0] == mocked_alert_data[alert_expected]
assert events[1] == mocked_trace_data[trace_expected]
assert events[2] == mocked_activity_data[activity_expected]


FAKE_ISO_DATE_CASES = [
("2023-08-01T14:15:26.123456+0000Z", # 6 digit milliseconds + tz+ Z
datetime(2023, 8, 1, 14, 15, 26, 123456, tzinfo=timezone.utc)),
("2023-08-01T14:15:26+0000Z", # No milliseconds + tz+ Z
datetime(2023, 8, 1, 14, 15, 26, tzinfo=timezone.utc)),
("2023-08-01T14:15:26+0000", # 6 digit milliseconds + tz , No Z
datetime(2023, 8, 1, 14, 15, 26, tzinfo=timezone.utc)),
("2023-08-01 14:15:26+0000Z", None), # Invalid format, missing 'T', expecting ValueError
("2023-08-01T14:15:26Z", # No milliseconds + tz+ Z
datetime(2023, 8, 1, 14, 15, 26, tzinfo=timezone.utc)),
("2023-08-01T14:15:26.123Z", # 3 digit milliseconds + Z
datetime(2023, 8, 1, 14, 15, 26, 123, tzinfo=timezone.utc)),
("2023-08-01T14:15:26.123+0000Z", # 3 digit milliseconds + tz+ Z
datetime(2023, 8, 1, 14, 15, 26, 123, tzinfo=timezone.utc))
]


@pytest.mark.parametrize("input_str, expected_dt", FAKE_ISO_DATE_CASES)
def test_parse_special_iso_format(input_str, expected_dt):
"""
Given: date string in differents formats
When: trying to convert from response to datetime
Then: make sure parsing is correct.
"""
if expected_dt is None:
with pytest.raises(ValueError):
FireEyeETPEventCollector.parse_special_iso_format(input_str)
else:
assert FireEyeETPEventCollector.parse_special_iso_format(input_str) == expected_dt


class TestLastRun:
@pytest.fixture
def last_run(self):
"""
Given: event_typess
When: trying to create last run
Then: make sure last run created with the events.
"""
# Create a LastRun instance with dummy event types
event_types = [FireEyeETPEventCollector.EventType('alerts', 25, outbound=False),
FireEyeETPEventCollector.EventType('email_trace', 25, outbound=False),
FireEyeETPEventCollector.EventType('activity_log', 25, outbound=False)
]
return FireEyeETPEventCollector.LastRun(event_types=event_types)

def test_to_demisto_last_run_empty(self, last_run):
# Test to_demisto_last_run method when there are no event types.
last_run.event_types = []
assert last_run.to_demisto_last_run() == {}


@ freeze_time("2023-07-30 11:34:30")
def test_get_command(mocker):
mocked_alert_data = util_load_json('test_data/alerts.json')
mocked_trace_data = util_load_json('test_data/email_trace.json')
mocked_activity_data = util_load_json('test_data/activity_log.json')
event_types_to_run = [
FireEyeETPEventCollector.EventType('alerts', 25, outbound=False),
FireEyeETPEventCollector.EventType('email_trace', 1000, outbound=False),
FireEyeETPEventCollector.EventType('activity_log', 25, outbound=False)
]
collector = FireEyeETPEventCollector.EventCollector(mock_client(), event_types_to_run)
mocker.patch.object(FireEyeETPEventCollector.Client, 'get_alerts', side_effect=[
mocked_alert_data['ok_response_single_data'], {'data': []}])
mocker.patch.object(FireEyeETPEventCollector.Client, 'get_email_trace', side_effect=[
mocked_trace_data['ok_response_single_data'], {'data': []}])
mocker.patch.object(FireEyeETPEventCollector.Client, 'get_activity_log', side_effect=[
mocked_activity_data['ok_response'], {'data': []}])
next_run, events = collector.get_events_command(
start_time=datetime.now() - timedelta(days=20)
)
assert events.readable_output


PAGINATION_CASES = [
('activity_log', 'test_data/activity_log.json', 'get_activity_log', 3), # 2 calls of activity log (4 events, one dup)
('alerts', 'test_data/alerts.json', 'get_alerts', 3), # 2 calls of alerts (4 events, one dup)
('email_trace', 'test_data/email_trace.json', 'get_email_trace', 3) # 2 calls of trace (4 events, one dup)
]


@ freeze_time("2023-08-02 11:34:30")
@pytest.mark.parametrize("event_name, res_mock_path, func_to_mock, expected_res", PAGINATION_CASES)
def test_pagination(mocker, event_name, res_mock_path, func_to_mock, expected_res):
"""
Given: a Mocked response of calls to API
When: Running fetch on activity log type
Then: Validate we fetch correct number of results, meaning:
1. No dups
2. All events arrived
"""
collector = FireEyeETPEventCollector.EventCollector(
mock_client(),
[FireEyeETPEventCollector.EventType(event_name, 4, outbound=False)]
)

mocked_data = util_load_json(res_mock_path)
mocker.patch.object(FireEyeETPEventCollector.Client, func_to_mock,
side_effect=mocked_data['paging_response'] + [{'data': []}])

# using timedelta with milliseconds due to a freeze_time issue.
events, md = collector.get_events_command(
start_time=datetime.now() - timedelta(days=2, milliseconds=1)
)
assert len(events)
Loading