Skip to content

Commit

Permalink
Adding additional documentation to project
Browse files Browse the repository at this point in the history
  • Loading branch information
Josh Rickard committed Aug 28, 2020
1 parent 24b7ff3 commit e74cb21
Show file tree
Hide file tree
Showing 4 changed files with 65 additions and 63 deletions.
20 changes: 20 additions & 0 deletions .env-example
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
ELK_VERSION=7.9.0
ELASTIC_USERNAME="elastic"
ELASTIC_PASSWORD="some_password"

# Configuration Variables
ELASTICSEARCH_HEAP="2g"
LOGSTASH_HEAP="1g"
PACKETBEAT_HEAP="256m"
FILEBEAT_HEAP="256m"
XPACK_ENCRYPTION_KEY="somesuperlongstringlikethisoneMQBbtsynu4bV2uxLy"

# Self signed TLS certificates
CA_PASSWORD="some password"
CA_DN="CN=Elastic Certificate Tool Autogenerated CA"
CA_DAYS=3650
ELASTIC_DIR=/usr/share/elasticsearch
LOGSTASH_DIR=/usr/share/logstash
KIBANA_DIR=/usr/share/kibana
PACKETBEAT_DIR=/usr/share/packetbeat
FILEBEAT_DIR=/usr/share/filebeat
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).


## [1.0.0] - 2020-08-28

- Initial release of elk-tls-docker project
- Added WiKi documentation
12 changes: 12 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Contributing

Swimlane welcomes contributions to our projects. Please follow these instructions on how to submit your contribution!

- Fork this repository
- Add your changes to your fork
- Add tests for your changes
- Review your changes, running any linting and tests to be sure they pass
- Create a pull request in this repository with your changes
- Be sure to outline what you are accomplishing with your changes and follow any template if applicable

Please be patient with your request! We value your contribution and will try to find time to review your pull request when we can.
84 changes: 21 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Setting up TLS ELK Stack

This docker-compose project will assist with setting up and creating a ELK stack using generated TLS certificates for communications. Basically, you get HTTPS for all services.
This docker-compose project will assist with setting up and creating a ELK stack using generated TLS certificates for communications. In general you get HTTPS for all services.

> Currently we encrypt communications between services but I plan on adding TLS authentication as an option in the near future.
> Please checkout our [WiKi](https://github.com/swimlane/elk-tls-docker/wiki) for detailed explanation of the project structure, configuration settings, and more.
## Environment Details

This project was built so that we could test and use built-in features undert Kibana SIEM, like detections, signals, cases, etc.
This project was built so that you can test and use built-in features under Kibana SIEM, like detections, signals, cases, and other features.

This docker-compose project will create the following Elastic containers based on version 7.8.1:

Expand All @@ -14,11 +16,11 @@ This docker-compose project will create the following Elastic containers based o
* Packetbeat
* Filebeat

> Please checkout our [WiKi](https://github.com/swimlane/elk-tls-docker/wiki) for detailed explanation of the project structure, configuration settings, and more.
## Setup

In order to use this project, you must first include the following in a [.env](.env) file:
In order to use this project, you must first include the following in a file named `.env`. I have provided an example environment variable file here [.env-example](https://github.com/swimlane/elk-tls-docker/.env-example).

> Copy or create your own `.env` from the provided example or from the code block below
```text
ELK_VERSION=7.8.1
Expand All @@ -45,6 +47,8 @@ FILEBEAT_DIR=/usr/share/filebeat

> Note: You may need to change the size of the HEAP variables in the above configuration file based on your system requirements. The settings present are for a machine with 8GB of memory
You can find more documentation about these settings in our [WiKi](https://github.com/swimlane/elk-tls-docker/wiki/Environment-Variables)

### Keystore

Before we build or create our containers we first need to create our keystore and certificates. You can do this using the [docker-compose.setup.yml](docker-compose.setup.yml) yaml file. If you run into issues you can see the associated documentation in our [WiKi Page about Certificates](https://github.com/swimlane/elk-tls-docker/wiki/Certificates) or create an issue in this repository.
Expand All @@ -53,7 +57,7 @@ Before we build or create our containers we first need to create our keystore an
docker-compose -f docker-compose.setup.yml run --rm certs
```

Once you run this yaml file, you should have all necessary certificates/keys. Next (recommended) you can set passwords for all accounts within ELK but this is optional and it it will use the default password defined in your `.env` file under the `ELASTIC_PASSWORD` value.
Once you run this yaml file, you should have all necessary certificates/keys. You can set passwords for all accounts within ELK but this is optional and it it will use the default password defined in your `.env` file under the `ELASTIC_PASSWORD` value.

### Setting Passwords (optional)

Expand All @@ -65,8 +69,6 @@ Let's run our ELK stack now:
docker-compose up -d
```

#### Set Passwords

You will need to set passwords for all accounts. I recommend in a tesitng environment to create a single password and use this across all accounts - it makes it easier when troublehshooting.

We need to access the elasticsearch container and generate our passwords:
Expand All @@ -77,6 +79,8 @@ docker-compose exec elasticsearch bash
# Set passwords for all accounts when prompted
```

For more information about generating passwords and certificates, please see our documentation in our [WiKi](https://github.com/swimlane/elk-tls-docker/wiki/Certificates)

## Running ELK Stack

Now, that you have your keys/certs and passwords set we can then just restart the containers by running:
Expand All @@ -87,69 +91,23 @@ docker-compose up -d

You should be able to login into the ELK stack and be on your way.

Below is a map of common services and their IPs/Ports created by this docker-compose project:

|Service |Protocol | IP Address | Port(s) |
|-------------|------------|------------|-----------|
|Elasticsearch|https |0.0.0.0 |9200, 9300 |
|Logstash |udp |0.0.0.0 |12201, 5000|
|Logstash |tcp |0.0.0.0 |5000 |
|Logstash |http |0.0.0.0 |5044, 9600 |
|Kibana |https |0.0.0.0 |5601 |
|Packetbeat |http |0.0.0.0 |Will capture local traffic|
|Filebeat |tcp |0.0.0.0 |9000 |


## Additional Notes
You can find additioanl information about the environments that are created on your [Environment Details](https://github.com/swimlane/elk-tls-docker/wiki/Environment-Details) WiKi page.

This section provides some traditional situations that may arise and how to handle them:
## Common Issues

* When trying to access Kibana for the first time, it may take several minutes for you to receive a response. If Kibana is working, you should see `server not up yet` when access the Kibana UI.
* Since we are generating self-signed certificates you may get a browser warning (especially Chrome) that the site is unsafe. If you select `continue` you may not be able to continue until you type `thisisunsafe` while the tab is selected. There is not place to enter this information, you actually just type it in and it will trust this certificate for this browser session. Please see this [great blog post here](https://medium.com/@dblazeski/chrome-bypass-net-err-cert-invalid-for-development-daefae43eb12).
Please see our WiKi documentation for the most [Common Issues](https://github.com/swimlane/elk-tls-docker/wiki/Common-Issues) I have seen through testing and usage of this project.

## Enabling features

This section talks about enabling features within Kibana and the Security stack. In order to use all the features (mostly dashboards and reports) within Kibana SIEM you may have to modify your index mappings.

### Create & Modify Index Mappings

As an example you can use the provided Python script [modify_index_mappings.py](bin/modify_index_mappings.py) to create a new (or modify an existing) elasticsearch index and modify its mappings to support features/UI components within the Elastic Kibana SIEM feature set.

### Loading pre-packaged rules

To access or load elastic's pre-packaged signals (detection rules) you can run the following after creating the default space above.

```python
import requests
from requests.auth import HTTPBasicAuth

_HOST = 'https://0.0.0.0:5601'
_USERNAME = 'elastic'
_PASSWORD = 'some_password'

headers = {
'kbn-xsrf': 'elk-tls-docker',
'Content-Type': 'application/json'
}

ENDPOINT = '/api/detection_engine/rules/prepackaged'

response = requests.put(
_HOST + ENDPOINT,
headers=headers,
auth=HTTPBasicAuth(_USERNAME, _PASSWORD),
verify=False)
print(response.json())
```
This project provides a few (continually adding as needed & requested) helper scripts that assist with enabling specific features within Elastic Kibana SIEM featureset as well as adding test data to your Elasticsearch instance.

## Adding Data to Kibana
Please see our [Enabling Features](https://github.com/swimlane/elk-tls-docker/wiki/Enabling-Features) page in our [Wiki](https://github.com/swimlane/elk-tls-docker/wiki)

Now we need to add some data. You can do this the traditional way by just configuring all your systems to point to Logstash or Elasticsearch or Kibana or you can use a few shortcuts :)
## Road Map

I am providing two different shortcuts. The first is using a project I wrote called `soc-faker` and the other is sending data directly to the `filebeat` service. You can find them both below:
Below are a list of features that are being planned for future releases:

* [Generate Fake Elasticsearch Documents using soc-faker](bin/send_document_to_elasticsearch.py)
* [Send raw strings to filebeat using sockets](bin/send_data_to_filebeat.py)
* Currently we encrypt communications between services but I plan on adding TLS authentication as an option in the near future.

## Built With

Expand Down

0 comments on commit e74cb21

Please sign in to comment.