Containers do not start for standalone and universal forward example.

[djflux]

The example docker compose file here does properly start any splunk containers using version splunk/splunk:9.1.1 and splunk/universalforwarder:9.0.5:

https://splunk.github.io/docker-splunk/EXAMPLES.html#create-standalone-and-universal-forwarder

Here is my docker-compose.yml:

version: "3.6"

networks:
  splunknet:
    driver: bridge
    attachable: true

services:
  uf1:
    networks:
      splunknet:
        aliases:
          - uf1
    image: splunk/universalforwarder:9.0.5
    hostname: uf1
    container_name: uf1
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD="changeme"
    ports:
      - 8089

  so1:
    networks:
      splunknet:
        aliases:
          - so1
    image: splunk/splunk:9.1.1
    hostname: so1
    container_name: so1
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD="changeme"
    ports:
      - 8000
      - 8089

This is the log message from the uf1 container:

uf1  | TASK [splunk_common : Enable forwarding to ['so1']] ****************************
uf1  | failed: [localhost] (item=so1) => {
uf1  |     "ansible_loop_var": "item",
uf1  |     "changed": false,
uf1  |     "cmd": [
uf1  |         "/opt/splunkforwarder/bin/splunk",
uf1  |         "add",
uf1  |         "forward-server",
uf1  |         "so1:9997",
uf1  |         "-auth",
uf1  |         "admin:changeme",
uf1  |         "--accept-license",
uf1  |         "--answer-yes",
uf1  |         "--no-prompt"
uf1  |     ],
uf1  |     "delta": "0:00:00.410134",
uf1  |     "end": "2023-09-14 19:19:34.928385",
uf1  |     "failed_when_result": true,
uf1  |     "item": "so1",
uf1  |     "rc": 12,
uf1  |     "start": "2023-09-14 19:19:34.518251"
uf1  | }
uf1  | 
uf1  | STDOUT:
uf1  | 
uf1  | Warning: Attempting to revert the SPLUNK_HOME ownership
uf1  | Warning: Executing "chown -R splunk /opt/splunkforwarder"
uf1  | 
uf1  | 
uf1  | STDERR:
uf1  | 
uf1  | Cannot connect to remote instance.

Here is the relevant log message from so1:

so1  | TASK [splunk_standalone : Setup global HEC] ************************************
so1  | fatal: [localhost]: FAILED! => {
so1  |     "cache_control": "private",
so1  |     "changed": false,
so1  |     "connection": "Close",
so1  |     "content_length": "130",
so1  |     "content_type": "text/xml; charset=UTF-8",
so1  |     "date": "Thu, 14 Sep 2023 19:20:03 GMT",
so1  |     "elapsed": 0,
so1  |     "redirected": false,
so1  |     "server": "Splunkd",
so1  |     "status": 401,
so1  |     "url": "https://127.0.0.1:8089/services/data/inputs/http/http",
so1  |     "vary": "Cookie, Authorization",
so1  |     "warnings": [
so1  |         "Module did not set no_log for password"
so1  |     ],
so1  |     "www_authenticate": "Basic realm=\"/splunk\"",
so1  |     "x_content_type_options": "nosniff",
so1  |     "x_frame_options": "SAMEORIGIN"
so1  | }
so1  | 
so1  | MSG:
so1  | 
so1  | Status code was 401 and not [200]: HTTP Error 401: Unauthorized
so1  | 

I've tried this compose file on Oracle Linux 8.8 as well as Docker Desktop on Windows 10 with the same results.

Here are the docker and rpm versions and the environment on Linux:

[root@dkr01 /home/rechenberg/code/docker/splunk-example]# docker images
REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE
splunk/splunk               9.1.1     64805960ef7d   6 days ago     2.15GB
splunk/universalforwarder   9.0.5     432e3f0876b1   2 months ago   679MB

[root@dkr01 ~]# docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:33:07 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[root@dkr01 ~]# docker compose version
Docker Compose version v2.21.0

[root@dkr01 ~]# rpm -qa | grep -i docker
docker-ce-rootless-extras-24.0.6-1.el8.x86_64
docker-ce-cli-24.0.6-1.el8.x86_64
docker-ce-24.0.6-1.el8.x86_64
docker-compose-plugin-2.21.0-1.el8.x86_64
docker-buildx-plugin-0.11.2-1.el8.x86_64

[root@dkr01 ~]# rpm -qa | grep -i container
containerd.io-1.6.22-3.1.el8.x86_64
container-selinux-2.205.0-2.module+el8.8.0+21045+adcb6a64.noarch

[root@dkr01 ~]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

Let me know if any more information is needed. Thanks for the help.

Cheers,
Andy

[djflux]

Following up - I can use the exact example file and the following command and the so1 container will start but the uf1 container still does not start outputting the same message. This behavior occurs on both Linux and Windows.

Command:

UF_IMAGE=splunk/universalforwarder:9.0.5 SPLUNK_IMAGE=splunk/splunk:9.1.1 SPLUNK_PASSWORD=changeme docker compose up

uf1  | TASK [splunk_common : Enable forwarding to ['so1']] ****************************
uf1  | failed: [localhost] (item=so1) => {
uf1  |     "ansible_loop_var": "item",
uf1  |     "changed": false,
uf1  |     "cmd": [
uf1  |         "/opt/splunkforwarder/bin/splunk",
uf1  |         "add",
uf1  |         "forward-server",
uf1  |         "so1:9997",
uf1  |         "-auth",
uf1  |         "admin:changeme",
uf1  |         "--accept-license",
uf1  |         "--answer-yes",
uf1  |         "--no-prompt"
uf1  |     ],
uf1  |     "delta": "0:00:00.338556",
uf1  |     "end": "2023-09-14 20:42:23.579038",
uf1  |     "failed_when_result": true,
uf1  |     "item": "so1",
uf1  |     "rc": 12,
uf1  |     "start": "2023-09-14 20:42:23.240482"
uf1  | }
uf1  | 
uf1  | STDOUT:
uf1  | 
uf1  | Warning: Attempting to revert the SPLUNK_HOME ownership
uf1  | Warning: Executing "chown -R splunk /opt/splunkforwarder"
uf1  | 
uf1  | 
uf1  | STDERR:
uf1  | 
uf1  | Cannot connect to remote instance.
uf1  | 

[mnieva-cisco]

Did you ever figure this out? I have the same issue right now

[djflux]

Did you ever figure this out? I have the same issue right now

Nothing yet. I'll give the new UF a try when I get to the office on Tuesday.