mosquito problems

[the1gofer]

I created a password file, and had clients connected, but at some point the container restarted and now I'm getting the following error.

from docker container ps:
b244a6ac0577 iotstack_mosquitto "/docker-entrypoint.…" 6 minutes ago Restarting (13) 19 seconds ago mosquitto

from the logs:

2022-02-23T16:13:02.280538764Z 2022-02-23T16:13:02: mosquitto version 2.0.14 starting
2022-02-23T16:13:02.280594059Z 2022-02-23T16:13:02: Config loaded from /mosquitto/config/mosquitto.conf.
2022-02-23T16:13:02.280616355Z 2022-02-23T16:13:02: Error: Unable to open pwfile "enabled".
2022-02-23T16:13:02.280635799Z 2022-02-23T16:13:02: Error opening password file "enabled".

I found an article that said this was probably becuase the pwfile has the wrong owner,
-rw-r--r-- 1 1883 1883 364 Feb 21 09:00 pwfile

id -nu 1 says the owner is doemon.

any suggestions how to fix?

[ukkopahis]

Try renaming ./volumes/mosquitto/config to config.bak and restart the container. The container will recreate the missing config-folder. Look at what the file permissions and owners are in the container created "clean" configs. Fix your config.bak to match those, stop the container, delete "config" and rename your original "config.bak" back to "config".

[Paraphraser]

Inside the container, user ID 1883 is "mosquitto". The container expects to launch as root (the RPi's root), then the self-repair code runs which recreates any missing files/folders and unconditionally (and recursively) sets ownership to 1883:1883. Then the Mosquitto process (the broker) is launched which immediately downgrades its permissions to user 1883.

If you are able to create any file/folder in the persistent storage area (~/IOTstack/volumes/mosquitto), then launch ("up") or restart the Mosquitto container and that does not result in the file changing ownership to 1883:1883 then it implies something else is wrong. Perhaps there is a "user" statement in the compose fragment for Mosquitto? There should not be.

Otherwise, I'd be suspecting some problem with pwfile itself. If you delete (or rename) that file and restart the container, a new null pwfile should be created. Then you can start adding usernames and passwords to it as documented.

[Paraphraser]

config and pwfile folders and contents come from the template. The others are created by Mosquitto as it runs. Log may not be there if you let all logging go through docker.

[the1gofer]

ok so I've managed to recreate the issue.

I'm still learning quite bit about linux, containers, etc, so I ended up starting from a fresh image, resetting up IOTstack.

Is it possible it has to do with the fresh setup?

It seems to work once I get the first entry. Still no bash

Ok, I went in a deleted the test passwords, and tried to enter a real one, and am getting the error message again. So it has something to do the empty file.

If I remove the file, then restart the container it gets stuck

so it doens't like me messing with that file very much

[Paraphraser]

That's true. Mosquitto is very sensitive to any of its files being futzed with. That's got nothing to do with IOTstack. In historical terms, Mosquitto was always a problem. It is less of a problem with the self-repair code in place but the self-repair code can't fix everything.

If you really want to open a shell into the Mosquitto container, use "ash":

It's possible that "restart" isn't strong enough and that a recreate is required:

$ cd ~/IOTstack
$ docker-compose up -d --force-recreate mosquitto

[Paraphraser]

Actually, a restart is sufficient.

[the1gofer]

When you say restart you mean docker restart mosquito.

[the1gofer]

Maybe I missing something, but doesn't this make password authentication unusable? You can set it up and use it, but the moment the container restarts you have to reimage it fresh?

[ukkopahis]

but doesn't this make password authentication unusable? You can set it up and use it, but the moment the container restarts you have to reimage it fresh?

Shouldn't work like that. You probably changed something in config/ or pwfile/ that broke it, but didn't restart. Thus the service worked ok until you had do a restart much later.

But if you can report how to reproduce¹ the problem, we will fix it.

By restart. we mean: docker-compose restart mosquitto
At this time docker restart mosquitto also works, but it may not in the future². Better to make it a habit to use docker-compose always when applicable.

Footnotes

1. Report:
   a) your full docker-compose.yml
   b) starting from a clean slate: cd ~/IOTstack && docker-compose down mosquitto && sudo rm -rf volumes/mosquitto && docker-compose up -d mosquitto
   c) output of docker image history iotstack_mosquitto:latest
   d) every command executed and its output
   e) expected result vs actual result ↩

2. This change would be needed in order to support updates with "rollback points". This would remove static container_name:s from the services. The containers would henceforth be named automatically, e.g. iotstack_mosquitto_1 ↩