Docker compose yaml with wrong identation

[geoleven]

The generated YAML inserts 4 uneeded space in fron of two of the Telegraph dependencies making the docker unable to parse it.

[Paraphraser]

I don't normally run telegraf but when I copy the template to services, then tack the service.yml onto docker-compose.yml and up the stack, I don't get any parsing errors. Something else must be going on. Can you add your docker-compose.yml to the case?

$ cd ~/IOTstack
$ cp -R .templates/telegraf services/
$ cat services/telegraf/service.yml >>docker-compose.yml
$ tail docker-compose.yml 
      - USER_GID=1000

  telegraf:
    container_name: telegraf
    image: telegraf
    volumes:
      - ./services/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
    depends_on:
      - influxdb
      - mosquitto

$ docker-compose up -d
Pulling telegraf (telegraf:)...
latest: Pulling from library/telegraf
7d9beb74056b: Pull complete
689ccbaccf1b: Pull complete
aadf7011b922: Pull complete
cf800bf0524b: Pull complete
94226d6ee49f: Pull complete
34cf17aae49e: Pull complete
71d64ad4635d: Pull complete
Digest: sha256:dabb719790739bcb15d5b9b2ecabccd44bab2f13f7eaaee28da1c52f399b67fd
Status: Downloaded newer image for telegraf:latest
pihole is up-to-date
nodered is up-to-date
influxdb is up-to-date
gitea is up-to-date
grafana is up-to-date
mosquitto is up-to-date
portainer is up-to-date
Creating telegraf ... done

$ DPS
NAMES               CREATED             STATUS
telegraf            27 seconds ago      Up 21 seconds
mosquitto           3 hours ago         Up 3 hours
nodered             3 hours ago         Up 3 hours (healthy)
grafana             3 hours ago         Up 3 hours
pihole              3 hours ago         Up 3 hours (healthy)
influxdb            3 hours ago         Up 3 hours
gitea               3 hours ago         Up 3 hours
portainer           3 hours ago         Up 3 hours

[geoleven]

Okay, so I run the build again (it is fully reproducible for me), selected to overwrite everything.

Same error as expected:

So from the generated file here is the error:

and I also include the whole unedited file just for sanity reasons.
docker-compose.zip

[Paraphraser]

Interesting. Let's see where it is coming from, starting from a clean clone of IOTstack:

$ git clone https://github.com/SensorsIot/IOTstack.git AbsolutelyCleanCloneOfIOTstack
Cloning into 'AbsolutelyCleanCloneOfIOTstack'...
remote: Enumerating objects: 120, done.
remote: Counting objects: 100% (120/120), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 3063 (delta 49), reused 104 (delta 42), pack-reused 2943
Receiving objects: 100% (3063/3063), 1.07 MiB | 564.00 KiB/s, done.
Resolving deltas: 100% (1812/1812), done.

$ grep -R "cadvisor-arm" AbsolutelyCleanCloneOfIOTstack
Binary file AbsolutelyCleanCloneOfIOTstack/.git/index matches
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/service_cadvisor-arm.yml:  cadvisor-arm:
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/service_cadvisor-arm.yml:    image: budry/cadvisor-arm:latest
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/build.sh:  "cadvisor-arm" "monitor full container stack " "ON" \

$ grep -R "node-exporter" AbsolutelyCleanCloneOfIOTstack
Binary file AbsolutelyCleanCloneOfIOTstack/.git/index matches
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/service_node-exporter.yml:  node-exporter:
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/service_node-exporter.yml:    image: prom/node-exporter:latest
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/build.sh:  "node-exporter" "monitor this computer " "ON" \
AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/config.yml:#       - node-exporter:9100

So, Prometheus is implicated and the likely candidate is its build.sh

$ cat AbsolutelyCleanCloneOfIOTstack/.templates/prometheus/build.sh 
#!/bin/bash

DOCKER_COMPOSE_PATH=./.tmp/docker-compose.tmp.yml
TEMPLATE_PATH=./.templates/prometheus

if [[ ! -f $DOCKER_COMPOSE_PATH ]]; then
  echo "[Prometheus] Warning: $DOCKER_COMPOSE_PATH does not exist."
fi

# Configure Prometheus Add-ons

option_selection=$(whiptail --title "Select Prometheus Options" --checklist --separate-output \
  "Use the [SPACEBAR] to select add-on containers from the list below." 20 78 12 -- \
  "node-exporter" "monitor this computer " "ON" \
  "cadvisor-arm" "monitor full container stack " "ON" \
  3>&1 1>&2 2>&3)

mapfile -t selected_options <<< "$option_selection"

# (cat $TEMPLATE_PATH/service.yml; echo) >> $DOCKER_COMPOSE_PATH

for option in "${selected_options[@]}"; do
  # insert add-on service
  (cat $TEMPLATE_PATH/service_${option}.yml; echo) >> $DOCKER_COMPOSE_PATH

  # include add-on in depends_on
  sed -i.bak -e "/depends_on:/a\\
    \\ \\ \\ \\ \\ \\ - ${option}" $DOCKER_COMPOSE_PATH
done

# clean up
rm ${DOCKER_COMPOSE_PATH}.bak

and there's the likely culprit - the sed inside the for loop with its series of hard-coded spaces (\\ ).

That also explains why I didn't hit the same error. I didn't do the install from the menu and didn't run build.sh to choose any options.

Using sed to do this type of work is context-dependent (presumably it worked for @llusdess in his/her environment) and, as we now see, somewhat error-prone (it doesn't work in yours).

The "first blush" solution is to re-jig build.sh to remove four of the spaces (\\ \\ \\ \\ ). You are welcome to do that and equally welcome to propose a pull request to get that change into the repo. But there's a question lurking in the back of my mind: will fixing it for you break it for everyone else (eg @llusdess)? Answering that depends on knowing who is actually running Prometheus, whether it worked out of the box for them and, if so, why they didn't hit this error when you did, or if they also hit this error, just fixed it without mentioning it, and moved on. All unanswerable.

My honest advise is to just hand-edit your docker-compose.yml and move on.

If you wish, I'm happy to expand on my own approach to IOTstack maintenance (hint: it doesn't involve worrying too much about whether what menu.sh produces is 100% correct).

[Slyke]

I'll see if I can fix this on the master branch. This is one of the reasons to switch to Python in the rewrite, as it has a nice yaml library, so that direct text manipulation isn't required.

If you find that you need to continue rebuilding IOTstack, consider adding these to the compose-override.yml file so that they are injected in automatically. See: https://sensorsiot.github.io/IOTstack/Custom/ for instructions. Note that you'll have to add the dependencies for all of the required depends_on for each service (as overriding overwrites the list).

[Paraphraser]

See also #142

[iliavivanov]

Workaround
sed -i 's/ \\/\\/g' ~/IOTstack/.templates/prometheus/build.sh

[Paraphraser]

No build script any more. Segmented into three containers. I'm pretty sure this is fixed.