Starting telegraph - Error

[DSew20]

I added telegraf via mensu.sh, but I gen an error when docker tries to start telegraf:

ERROR: for telegraf Cannot start service telegraf: OCI runtime create failed: container_linux.go:370: starting container
process caused: process_linux.go:459:container init caused: rootfs_linux.go:59: mounting "/home/pi/IOTstack/services/
telegraf/telegraf.conf" to rootfs at "/var/lib/docker/overlay2/700ca549a4bf04a8d
b3560194052c797c14aa54b31a9f87eb0d2942a85057cc7/merged/etc/telegraf/telegraf.conf"
caused: not a directory: unknown: Are you trying to mount a directory onto a
file (or vice-versa)? Check if the specified host path exists and is the expected type

The docker-compose.yml is as created by menu.sh:
telegraf:
container_name: telegraf
image: telegraf
restart: unless-stopped
volumes:
- ./services/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
depends_on:
- influxdb
- mosquitto
networks:
- iotstack_nw

Any ideas how to fix this?

[Paraphraser]

From the answer to a similar question on the Discord channel.

docker-compose seems to assume that anything mentioned in a "volumes" definition is a folder. When the left hand side of a mapping is missing at "up" time it automatically creates a folder (unless there's a read-only flag).

The basic problem is that your config file has gone walkabout. Unless you can think of a reason why that might have happened, put it down to gremlins.

The solution is:

1. down your stack:

   $ cd ~/IOTstack
   $ docker-compose down
   

2. fix the underlying problem:

   $ cd services
   $ rm -rf telegraf
   $ mkdir telegraf
   $ cd telegraf
   $ cp ~/IOTstack/.templates/telegraf/telegraf.conf .
   $ cp ~/IOTstack/.templates/telegraf/service.yml .
   

   If the rm complains about permissions, repeat that command (only!) with sudo.

   The rationale in these commands is to undo all potential damage caused by letting docker-compose create the left-hand-side path. In particular, things in the "services" directory should not be owned by root (unlike the volumes directory where they mostly are owned by root).

3. Double check that the service definition for telegraf in your docker-compose.yml looks the same as the service.yml in the above, or that you can explain any differences.

   The rationale is to make sure that any changes you may have made while you were trying to solve the problem are undone.

4. Check that the contents of telegraf.conf are how they need to be. The file copied from the template is the default. I don't run telegraf so I have no idea how much customisation normally needs to be done. Do you have a backup somewhere?

5. Bring up your stack.

   $ cd ~/IOTstack
   $ docker-compose up -d
   

[DSew20]

Thank you Paraphraser, that fixed the issue.

Kind regards

[DSew20]

Does anybody have any pointers re how to get telegraf working in the IOTstack? Is does not fail now when pulling, but does not seem to do anything. I have created a telegraf.conf, but not sure where this needs to sit to get telegraf to push data to influxdb.

Any tips / guides that can help?

[Paraphraser]

I don't use telegraf but I believe that it sends data direct to InfluxDB via UDP. If that's your understanding as well then the question becomes, "have you configured InfluxDB to expect UDP data?"

If no, then maybe look at this gist.

The thing to be aware of as you read that gist is that the contents of the influxdb.env file are now in the service definition:

~/IOTstack/services/service.yml

which becomes part of:

~/IOTstack/docker-compose.yml

When you see these lines in the gist:

INFLUXDB_UDP_DATABASE=udp
INFLUXDB_UDP_ENABLED=true
INFLUXDB_UDP_BIND_ADDRESS=0.0.0.0:8086

you have to do the mental translation to these lines in the service definition and compose file:

    - INFLUXDB_UDP_ENABLED=true
    - INFLUXDB_UDP_BIND_ADDRESS=0.0.0.0:8086
    - INFLUXDB_UDP_DATABASE=udp

To answer your other question, you need to look at the service definition for telegraf:

telegraf:
  container_name: telegraf
  image: telegraf
  restart: unless-stopped
  volumes:
    - ./services/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
  depends_on:
    - influxdb
    - mosquitto
  networks:
    - iotstack_nw

That "volumes" statement decomposes into:

• the external path is ./services/telegraf/telegraf.conf
• the internal path is /etc/telegraf/telegraf.conf
• it's read-only (ie the container can't modify it)

For the external path, the leading "." means "the folder that contains docker-compose.yml" so it's the absolute path:

/home/pi/IOTstack/services/telegraf/telegraf.conf

Let's say you have a replacement telegraf.conf in your home directory. This is how you would change it:

$ cd
$ cp telegraf.conf ~/IOTstack/services/telegraf/telegraf.conf

Causing the running container to notice the change will be one of the following:

1. If the container responds to a "restart" message then:

   $ cd ~/IOTstack
   $ docker-compose restart telegraf
   

2. If a "restart" doesn't seem to work, use brute force:

   $ cd ~/IOTstack
   $ docker-compose stop telegraf
   $ docker-compose rm -f telegraf
   $ docker-compose up -d telegraf
   

Sometimes containers respond appropriately to restart, sometimes not. There's no way to be sure. The brute force method clobbers the running container and removes it, then rebuilds it again.

Sometimes an "up" will work as well:

$ cd ~/IOTstack
$ docker-compose up -d telegraf

but it depends on whether docker-compose thinks there's been a material change. I'm not sure that a config file would count as a material change - but you could give that a whirl too.

Hope this helps.

[DSew20]

Thank you again Paraphraser. Your guide above is amazing, works brilliantly and is very clear.

For anybody else trying to get telegraf working, I found that I needed to use the brute force method (stop, remove, rebuild) but it then works. I also used a slightly modified telegraf.conf file with just influxdb outputs and recommended Raspberry Pi inputs.

The only issue I have now found is that telegraf monitors the container, not the host.

I have found some guides that suggest needing to add some volumes and environment variables to docker to allow the container to see the host. The recommended volumes and variables are below:

     -v /:/hostfs:ro
    -v /etc:/hostfs/etc:ro
-v /proc:/hostfs/proc:ro
-v /sys:/hostfs/sys:ro
-v /var:/hostfs/var:ro
-v /run:/hostfs/run:ro
-e HOST_ETC=/hostfs/etc
-e HOST_PROC=/hostfs/proc
-e HOST_SYS=/hostfs/sys
-e HOST_VAR=/hostfs/var
-e HOST_RUN=/hostfs/run
-e HOST_MOUNT_PREFIX=/hostfs

I added the following to docker-compose and seem to be getting some measures (hopefully for the host). But not sure is these are correct:

environment:
  HOST_PROC: /rootfs/proc
  HOST_SYS: /root/sys
  HOST_ETC: /root/etc
volumes:
- ./services/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
- /sys:/rootfs/sys
- /proc:/rootfs/proc
- /etc:/rootfs/etc

I'm really not familiar with linux, so not sure which of the sys variables I need to see the CPU, memory, network, disk etc for the host. The points above seem to get some. What is the correct nomenclature for such volumes and variables being added to the docker-compose.yml.

My only other issue is docker seems to sporadically knock-out my wifi and lan. If I take down the stack, reset the network and bring up the stack again it clears the error. But, then it drops a few hours later. There are separate guides for this that I am trying to follow.

[Paraphraser]

Syntactically, it would be closer to:

telegraf:
  container_name: telegraf
  image: telegraf
  restart: unless-stopped
  environment:
    - HOST_ETC=/hostfs/etc
    - HOST_PROC=/hostfs/proc
    - HOST_SYS=/hostfs/sys
    - HOST_VAR=/hostfs/var
    - HOST_RUN=/hostfs/run
    - HOST_MOUNT_PREFIX=/hostfs
  volumes:
    - ./services/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
    - /:/hostfs:ro
    - /etc:/hostfs/etc:ro
    - /proc:/hostfs/proc:ro
    - /sys:/hostfs/sys:ro
    - /var:/hostfs/var:ro
    - /run:/hostfs/run:ro
  depends_on:
    - influxdb
    - mosquitto
  networks:
    - iotstack_nw

I emphasise syntactically. What it does and whether that is appropriate, or even whether it actually works, are separate matters.

I have to say that I hate the look of:

    - /:/hostfs:ro

As I read that, it says "map the entire Raspberry Pi file system into /hostfs".

Whenever I see a pattern like that, I start to think that I'm looking at the answer to a question which should never have been asked.

Also, if you map / to /hostfs then /etc is implicitly mapped to /hostfs/etc. Ditto /proc through /run. Those lines would seem to be redundant. If they turn out to be necessary then that's something I can't explain.

Gives me the fidgets...

---

On your other questions:

not sure which of the sys variables I need to see the CPU, memory, network, disk etc for the host

No idea. Someone with more knowledge will have to answer that. I only track temperature and I don't do it with Telegraf (just a cron job, firing a script every 5 minutes which sticks the result of vcgencmd measure_temp into an MQTT payload and sends it on its way).

My only other issue is docker seems to sporadically knock-out my wifi and lan

See:

• Issue 219.
• This gist

Issue 219 boils down to:

$ sudo bash -c '[ $(egrep -c "^allowinterfaces eth0,wlan0" /etc/dhcpcd.conf) -eq 0 ] && echo "allowinterfaces eth0,wlan0" >> /etc/dhcpcd.conf'
$ sudo systemctl daemon-reload
$ sudo systemctl restart dhcpcd

I've done this on all of my RPis (3B+ and multiple 4Bs), no matter how they are set up (SD boot+run; SD boot + SSD run; SSD boot+run). I've also added the bash -c ... line to my standard build script so that it will always be there in future. The systemctl commands are only needed to activate it "now" and you can always just do the first line and reboot.

I'm still running the isc-dhcp-fix.sh script (from the gist reference). I'm not sure whether it's needed because it and the "Issue 219 fix" are playing in the same area. The "Issue 219 fix" has probably taken care of everything but it does no harm to keep the script running. I'll wait a while and see how many log entries I get (that aren't explained by restarts). If a few weeks go by and it never pops up in any log, I'll declare "problem solved", remove the script and withdraw the gist.

I'm considering making the bash -c ... part of a Pull Request for IOTstack so that it gets installed for everyone. What do you think of that plan?

[Finnitio]

I added telegraf via mensu.sh, but I gen an error when docker tries to start telegraf:

ERROR: for telegraf Cannot start service telegraf: OCI runtime create failed: container_linux.go:370: starting container
process caused: process_linux.go:459:container init caused: rootfs_linux.go:59: mounting "/home/pi/IOTstack/services/
telegraf/telegraf.conf" to rootfs at "/var/lib/docker/overlay2/700ca549a4bf04a8d
b3560194052c797c14aa54b31a9f87eb0d2942a85057cc7/merged/etc/telegraf/telegraf.conf"
caused: not a directory: unknown: Are you trying to mount a directory onto a
file (or vice-versa)? Check if the specified host path exists and is the expected type

The docker-compose.yml is as created by menu.sh:
telegraf:
container_name: telegraf
image: telegraf
restart: unless-stopped
volumes:

• ./services/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
  depends_on:
• influxdb
• mosquitto
  networks:
• iotstack_nw

Any ideas how to fix this?

I had the same problem after doing a clean install on a newly set up raspberry pi os. The troubleshoot in Post from @Paraphraser fixed it. So thank you @Paraphraser !

[Paraphraser]

Telegraf has been through several iterations since this so I'm pretty sure this is stale.