Adding production yaml and new documentation

.env-example

@@ -17,4 +17,16 @@ ELASTIC_DIR=/usr/share/elasticsearch
 LOGSTASH_DIR=/usr/share/logstash
 KIBANA_DIR=/usr/share/kibana
 PACKETBEAT_DIR=/usr/share/packetbeat
-FILEBEAT_DIR=/usr/share/filebeat
+FILEBEAT_DIR=/usr/share/filebeat
+
+# Letsencrypt certificates
+## Setting STAGING to true means it will generate self-signed certificates
+## Setting STAGING to false means it will generate letsencrypt certificates
+# STAGING=false
+STAGING=true
+
+# swag Configuration
+DOMAIN=mydomain.com
+SUBDOMAIN=kibana
+[email protected]
+TIMEZONE=America/Chicago

README.md

@@ -1,6 +1,6 @@
 # Setting up TLS ELK Stack
 
-This docker-compose project will assist with setting up and creating a ELK stack using generated TLS certificates for communications.  In general you get HTTPS for all services.  
+This docker-compose project will assist with setting up and creating a ELK stack using either self-signed TLS certificates or using LetsEncrypt certificates for communications.  In general you get HTTPS for all services.
 
 > Please checkout our [WiKi](https://github.com/swimlane/elk-tls-docker/wiki) for detailed explanation of the project structure, configuration settings, and more.
 
@@ -42,6 +42,18 @@ LOGSTASH_DIR=/usr/share/logstash
 KIBANA_DIR=/usr/share/kibana
 PACKETBEAT_DIR=/usr/share/packetbeat
 FILEBEAT_DIR=/usr/share/filebeat
+
+# Letsencrypt certificates
+## Setting STAGING to true means it will generate self-signed certificates
+## Setting STAGING to false means it will generate letsencrypt certificates
+# STAGING=false
+STAGING=true
+
+# swag Configuration
+DOMAIN=mydomain.com
+SUBDOMAIN=kibana
+[email protected]
+TIMEZONE=America/Chicago
 ```
 
 > Note: You may need to change the size of the HEAP variables in the above configuration file based on your system requirements.  The settings present are for a machine with 8GB of memory
@@ -54,40 +66,40 @@ You can find more documentation about these settings in our [WiKi](https://githu
 
 Before we build or create our containers we first need to create our keystore and certificates.  You can do this using the [docker-compose.setup.yml](docker-compose.setup.yml) yaml file.  If you run into issues you can see the associated documentation in our [WiKi Page about Certificates](https://github.com/swimlane/elk-tls-docker/wiki/Certificates) or create an issue in this repository.
 
+
+#### Creating Keystore for self-signed certificates
+
+By default creation of self-signed certificates is used and makes the most sense when testing out this project.  To do so you simply run the following command first:
+
 ```bash
 docker-compose -f docker-compose.setup.yml run --rm certs
 ```
 
-Once you run this yaml file, you should have all necessary certificates/keys.  You can set passwords for all accounts within ELK but this is optional and it it will use the default password defined in your `.env` file under the `ELASTIC_PASSWORD` value.
+Please see our documentation about [Setup using self-signed certificates](https://github.com/swimlane/elk-tls-docker/wiki/Setup%20using%20self-signed%20certificates).
 
-### Setting Passwords (optional)
+#### Creating Keystore & Certificates for production
 
-The first thing you will is set passwords for all accounts within ELK.
+If you are wanting to deploy this project in a production like environment, please see our documentation [Setup using Letsencrypt](https://github.com/swimlane/elk-tls-docker/wiki/Setup%20using%20Letsencrypt).
 
-Let's run our ELK stack now:
-
-```
-docker-compose up -d
-```
 
-You will need to set passwords for all accounts.  I recommend in a tesitng environment to create a single password and use this across all accounts - it makes it easier when troublehshooting.
+## Running a development environment
 
-We need to access the elasticsearch container and generate our passwords:
+Now, that you have your keys/certs and [passwords](https://github.com/swimlane/elk-tls-docker/wiki/Setting%20Passwords) set we can then just restart the containers by running:
 
 ```
-docker-compose exec elasticsearch bash
-> bin/elasticsearch-setup-passwords interactive -u "https://0.0.0.0:9200"
-# Set passwords for all accounts when prompted
+docker-compose up -d
 ```
 
-For more information about generating passwords and certificates, please see our documentation in our [WiKi](https://github.com/swimlane/elk-tls-docker/wiki/Certificates)
+You should be able to login into the ELK stack and be on your way.
+
+You can find additioanl information about the environments that are created on your [Environment Details](https://github.com/swimlane/elk-tls-docker/wiki/Environment-Details) WiKi page.
 
-## Running ELK Stack
+## Running a production environment
 
-Now, that you have your keys/certs and passwords set we can then just restart the containers by running:
+Now, that you have your keys/certs and [passwords](https://github.com/swimlane/elk-tls-docker/wiki/Setting%20Passwords) set we can then just restart the containers by running:
 
 ```
-docker-compose up -d
+docker-compose -f docker-compose.production.yml -f docker-compose.override.yml up -d
 ```
 
 You should be able to login into the ELK stack and be on your way.
@@ -108,11 +120,8 @@ Please see our [Enabling Features](https://github.com/swimlane/elk-tls-docker/wi
 
 Below are a list of features that are being planned for future releases:
 
-* Currently we encrypt communications between services but I plan on adding TLS authentication as an option in the near future.
-
-## Built With
-
-* [carcass](https://github.com/MSAdministrator/carcass) - Python packaging template
+* Adding additional services from Elastic
+* Adding certificate authentication for external usage
 
 ## Contributing
 

bin/create_new_superuser.py

@@ -0,0 +1,30 @@
+from socfaker import SocFaker
+import requests, json
+from requests.auth import HTTPBasicAuth
+
+
+_HOST = 'https://0.0.0.0:9200'
+ENDPOINT = '/_security/user/soc'
+
+_USERNAME = 'elastic'
+_PASSWORD = 'some_password'
+
+headers = {
+    'kbn-xsrf': 'elk-tls-docker',
+    'Content-Type': 'application/json'
+}
+
+_BODY = {
+    'email': '[email protected]',
+    'full_name': 'SOC',
+    'password': 'some_password',
+    'roles': ['superuser']
+}
+
+response = requests.post(
+    _HOST + ENDPOINT, 
+    headers=headers,
+    data=json.dumps(_BODY), 
+    auth=HTTPBasicAuth(_USERNAME, _PASSWORD), 
+    verify=False)
+print(response.json())

docker-compose.override.yml

@@ -0,0 +1,30 @@
+version: '3.5'
+
+services:
+    swag:
+      image: linuxserver/swag
+      container_name: swag
+      cap_add:
+        - NET_ADMIN
+      environment:
+        - PUID=2000
+        - PGID=2000
+        - TZ=${TIMEZONE}
+        - URL=${DOMAIN}
+        - SUBDOMAINS=${SUBDOMAIN}
+        - VALIDATION=http
+        - EMAIL=${EMAIL}
+        - STAGING=false
+      volumes:
+        - ./swag/kibana.subdomain.conf:/config/nginx/proxy-confs/kibana.subdomain.conf
+        - ./swag:/config
+      ports:
+        - 443:443
+        - 80:80
+      restart: unless-stopped
+      networks:
+        - elk
+
+networks:
+  elk:
+    driver: bridge

docker-compose.production.yml

@@ -0,0 +1,198 @@
+version: '3.5'
+
+# will contain all elasticsearch data.
+volumes:
+  data:
+
+secrets:
+  elasticsearch.keystore:
+    file: ./secrets/elasticsearch.keystore
+  ca.crt:
+    file: ./secrets/ca/ca.crt
+  ca.key:
+    file: ./secrets/ca/ca.key
+  elasticsearch.key:
+    file: ./secrets/elasticsearch/elasticsearch.key
+  elasticsearch.cert:
+    file: ./secrets/elasticsearch/elasticsearch.crt
+  logstash.key:
+    file: ./secrets/logstash/logstash.key
+  logstash.pkcs8:
+    file: ./secrets/logstash/logstash.pkcs8.key
+  logstash.cert:
+    file: ./secrets/logstash/logstash.crt
+  kibana.key:
+    file: ./secrets/kibana/kibana.key
+  kibana.cert:
+    file: ./secrets/kibana/kibana.crt
+  filebeat.key:
+    file: ./secrets/filebeat/filebeat.key
+  filebeat.cert:
+    file: ./secrets/filebeat/filebeat.crt
+
+services:
+  elasticsearch:
+    container_name: elasticsearch
+    hostname: elasticsearch
+    build:
+      context: elasticsearch/
+      args:
+        ELK_VERSION: ${ELK_VERSION}
+    restart: unless-stopped
+    environment:
+      CONFIG_DIR: ${ELASTIC_DIR}/config
+      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
+      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
+      ES_JAVA_OPTS: -Xmx${ELASTICSEARCH_HEAP} -Xms${ELASTICSEARCH_HEAP}
+      bootstrap.memory_lock: "true"
+      discovery.type: single-node
+    volumes:
+      - data:${ELASTIC_DIR}
+      - ./elasticsearch/config/elasticsearch.yml:${ELASTIC_DIR}/config/elasticsearch.yml:ro
+    secrets:
+      - source: elasticsearch.keystore
+        target: ${ELASTIC_DIR}/config/elasticsearch.keystore
+      - source: ca.crt
+        target: ${ELASTIC_DIR}/config/ca.crt
+      - source: elasticsearch.cert
+        target: ${ELASTIC_DIR}/config/elasticsearch.crt
+      - source: elasticsearch.key
+        target: ${ELASTIC_DIR}/config/elasticsearch.key
+    expose:
+      - "9200:9200"
+      - "9300:9300"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 200000
+        hard: 200000
+    networks:
+      - elk
+
+  logstash:
+    container_name: logstash
+    hostname: logstash
+    build:
+      context: logstash/
+      args:
+        ELK_VERSION: $ELK_VERSION
+    restart: unless-stopped
+    volumes:
+      - ./logstash/config/logstash.yml:${LOGSTASH_DIR}/config/logstash.yml:ro
+      - ./logstash/pipeline/logstash.conf:${LOGSTASH_DIR}/config/logstash.conf:ro
+    environment:
+      CONFIG_DIR: ${LOGSTASH_DIR}/config
+      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
+      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
+      LS_JAVA_OPTS: "-Xmx${LOGSTASH_HEAP} -Xms${LOGSTASH_HEAP}"
+    secrets:
+      - source: ca.crt
+        target: ${LOGSTASH_DIR}/config/ca.crt
+      - source: logstash.cert
+        target: ${LOGSTASH_DIR}/config/logstash.crt
+      - source: logstash.key
+        target: ${LOGSTASH_DIR}/config/logstash.key
+      - source: logstash.pkcs8
+        target: ${LOGSTASH_DIR}/config/logstash.pkcs8.key
+    networks:
+      - elk
+    expose:
+     - "12201:12201/udp"
+     - "5044:5044"
+     - "9600:9600"
+     - "5000:5000/tcp"
+     - "5000:5000/udp"
+    depends_on: 
+      - elasticsearch
+
+  kibana:
+    container_name: kibana
+    hostname: kibana
+    build:
+      context: kibana/
+      args:
+        ELK_VERSION: $ELK_VERSION
+    restart: unless-stopped
+    volumes:
+      - ./kibana/config/kibana.yml:${KIBANA_DIR}/config/kibana.yml:ro
+    environment:
+      CONFIG_DIR: ${KIBANA_DIR}/config
+      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
+      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
+      ENCRYPTION_KEY: ${XPACK_ENCRYPTION_KEY}
+    secrets:
+      - source: ca.crt
+        target: ${KIBANA_DIR}/config/ca.crt
+      - source: kibana.cert
+        target: ${KIBANA_DIR}/config/kibana.crt
+      - source: kibana.key
+        target: ${KIBANA_DIR}/config/kibana.key
+    expose:
+      - "5601:5601"
+    networks:
+      - elk
+    depends_on: 
+      - elasticsearch
+
+  packetbeat:
+    container_name: packetbeat
+    hostname: packetbeat
+    user: root
+    build:
+      context: packetbeat/
+      args:
+        ELK_VERSION: $ELK_VERSION
+    restart: unless-stopped
+    cap_add: 
+      - NET_ADMIN
+      - NET_RAW
+    command: packetbeat -e -strict.perms=false
+    volumes:
+      - ./packetbeat/config/packetbeat.yml:${PACKETBEAT_DIR}/packetbeat.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+    environment:
+      CONFIG_DIR: ${PACKETBEAT_DIR}/config
+      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
+      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
+      LS_JAVA_OPTS: "-Xmx${PACKETBEAT_HEAP} -Xms${PACKETBEAT_HEAP}"
+    secrets:
+      - source: ca.crt
+        target: ${PACKETBEAT_DIR}/config/ca.crt
+    networks:
+      - elk
+    depends_on: 
+      - logstash
+
+  filebeat:
+    container_name: filebeat
+    hostname: filebeat
+    build:
+      context: filebeat/
+      args:
+        ELK_VERSION: $ELK_VERSION
+    restart: unless-stopped
+    command: filebeat -e -strict.perms=false
+    volumes:
+      - ./filebeat/config/filebeat.yml:${FILEBEAT_DIR}/filebeat.yml:ro
+    environment:
+      CONFIG_DIR: ${FILEBEAT_DIR}/config
+      LS_JAVA_OPTS: "-Xmx${FILEBEAT_HEAP} -Xms${FILEBEAT_HEAP}"
+    secrets:
+      - source: ca.crt
+        target: ${FILEBEAT_DIR}/config/ca.crt
+      - source: filebeat.cert
+        target: ${FILEBEAT_DIR}/config/filebeat.crt
+      - source: filebeat.key
+        target: ${FILEBEAT_DIR}/config/filebeat.key
+    expose:
+      - "9000:9000" 
+    networks:
+      - elk
+    depends_on: 
+      - logstash
+
+networks:
+  elk:
+    driver: bridge