How to run aziot-edge in a docker container?

[mmoles-Growlink]

I would like to re-open issue #7258 . Egil was able to get the iot edge daemon running in a docker container, but the docker.sock was actually not being passed, so all of the containers that Edge Agent created was under the iot edge daemon container.

For devices imaged 3 years ago using Debian 9 Stretch and the latest IoT-Edge 1.1, releases for Debian 9 were dropped after 1.2.10.
For remote devices or critical devices, doing an OS upgrade is a lot more risky than updating a container unless you built your system around it. Considering the rapid development of iotEdge, it was a bit surprising to see how fast support was dropped for some of the OS. If we can get DinD working with iotedge, while the OS can still be unsupported, at least those devices can keep up for a little bit longer.

#7258
https://github.com/egil/IoT-Edge-Device-in-Docker

His example had it launching into /lib/systemd/systemd which wipes everything under var/run/ including the docker.sock file that would normally be mounted. So, instead i moved it to another location and had docker and edge target that. I am now running into this issue. I tried adding a bunch of users to the docker group to no avail.

As for files, i used the ones from https://github.com/egil/IoT-Edge-Device-in-Docker
dockefile I converted it to use FROM arm32v7/ubuntu:22.04.
docker-entrypoint.sh changed the aziot/config.toml to have moby runtime use uri = "unix:///iotedge/docker.sock"

mm@Me70231:~ $
mm@Me70231:~ $ sudo docker run -d --restart unless-stopped --privileged -it -v /var/run/docker.sock:/iotedge/docker.sock -v /dev:/dev -v /iotedge/storage:/iotedge/storage  -v /sys/fs/cgroup:/sys/fs/cgroup:rw  -e DOCKER_HOST='unix:///iotedge/docker.sock' -e connectionString='HostName=my-iothub.azure-devices.net;DeviceId=Me-70231;SharedAccessKey=XxxxxxxxxxxxxxxxxxX'  --hostname=edgedevice1 --name iot-edge-device --dns 8.8.8.8 --net=host --log-driver "json-file" --log-opt "max-file=10" --log-opt "max-size=500k" 192.168.1.71:5000/iotedgedocker:0.3.4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
mm@Me70231:~ $ docker exec -it iot-edge-device bash
root@edgedevice1:/#
root@edgedevice1:/#
root@edgedevice1:/#
root@edgedevice1:/# docker ps
CONTAINER ID   IMAGE                                   COMMAND                  CREATED         STATUS         PORTS     NAMES
72966824499d   192.168.1.71:5000/iotedgedocker:0.3.4   "/usr/local/bin/dock…"   9 seconds ago   Up 7 seconds             iot-edge-device
root@edgedevice1:/#


root@edgedevice1:/# iotedge check --verbose

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
    could not load cert with ID "aziot-edged-trust-bundle"

    Caused by:
        parameter "id" has an invalid value
        caused by: not found
        caused by: could not load cert with ID "aziot-edged-trust-bundle"

                   Caused by:
                       parameter "id" has an invalid value
                       caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    SocketError - SocketErrorCode (ConnectionRefused) : Connection refused /var/run/iotedge/mgmt.sock
    One or more errors occurred. (Got bad response: )
        caused by: SocketError - SocketErrorCode (ConnectionRefused) : Connection refused /var/run/iotedge/mgmt.sock
                   One or more errors occurred. (Got bad response: )
        caused by: docker returned exit status: 1, stderr = SocketError - SocketErrorCode (ConnectionRefused) : Connection refused /var/run/iotedge/mgmt.sock
                   One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
        caused by: Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
                   Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
                   You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
        caused by: Container engine is not configured to rotate module logs which may cause it run out of disk space.
                   Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
                   You can ignore this warning if you are setting log policy per module in the Edge deployment.
× production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeAgent container
        caused by: Could not check current state of edgeAgent container
        caused by: docker returned exit status: 1, stderr = Error: No such object: edgeAgent
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container
        caused by: Could not check current state of edgeHub container
        caused by: docker returned exit status: 1, stderr = Error: No such object: edgeHub
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the default network can connect to upstream MQTT port - OK
    skipping because of not required in this configuration
× container on the IoT Edge module network can connect to upstream AMQP port - Error
    Container on the azure-iot-edge network could not connect to my-iothub.azure-devices.net:5671
        caused by: Container on the azure-iot-edge network could not connect to my-iothub.azure-devices.net:5671
        caused by: docker returned exit status: 125, stderr = docker: Error response from daemon: network azure-iot-edge not found.
× container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error
    Container on the azure-iot-edge network could not connect to my-iothub.azure-devices.net:443
        caused by: Container on the azure-iot-edge network could not connect to my-iothub.azure-devices.net:443
        caused by: docker returned exit status: 125, stderr = docker: Error response from daemon: network azure-iot-edge not found.
√ container on the IoT Edge module network can connect to upstream MQTT port - OK
    skipping because of not required in this configuration
27 check(s) succeeded.
2 check(s) raised warnings.
6 check(s) raised errors.
2 check(s) were skipped due to errors from other checks.

root@edgedevice1:/# journalctl -u aziot-edged.service -f
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]: 2024-06-27T16:46:59Z [INFO] - Device is Me-70231 on my-iothub.azure-devices.net
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]: 2024-06-27T16:46:59Z [INFO] - Initializing module runtime...
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]: 2024-06-27T16:46:59Z [INFO] - Using runtime network id azure-iot-edge
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]: 2024-06-27T16:46:59Z [WARN] - container runtime error
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]: Caused by:
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]:     0: error trying to connect: Permission denied (os error 13)
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]:     1: Permission denied (os error 13)
Jun 27 16:46:59 edgedevice1 aziot-edged[1817]: 2024-06-27T16:46:59Z [ERR!] - Failed to initialize module runtime: runtime operation error: initialize module runtime
Jun 27 16:46:59 edgedevice1 systemd[1]: aziot-edged.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 16:46:59 edgedevice1 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.
Jun 27 16:47:04 edgedevice1 systemd[1]: aziot-edged.service: Scheduled restart job, restart counter is at 177.
Jun 27 16:47:04 edgedevice1 systemd[1]: Stopped Azure IoT Edge daemon.
Jun 27 16:47:05 edgedevice1 systemd[1]: Started Azure IoT Edge daemon.
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [INFO] - Starting Azure IoT Edge Daemon
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [INFO] - Version - 1.5.0
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [INFO] - Obtaining Edge device provisioning data...
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [INFO] - Device is Me-70231 on my-iothub.azure-devices.net
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [INFO] - Initializing module runtime...
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [INFO] - Using runtime network id azure-iot-edge
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [WARN] - container runtime error
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: Caused by:
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]:     0: error trying to connect: Permission denied (os error 13)
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]:     1: Permission denied (os error 13)
Jun 27 16:47:05 edgedevice1 aziot-edged[1825]: 2024-06-27T16:47:05Z [ERR!] - Failed to initialize module runtime: runtime operation error: initialize module runtime
Jun 27 16:47:05 edgedevice1 systemd[1]: aziot-edged.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 16:47:05 edgedevice1 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.
q
^C

[ryanwinter]

With OS support, the OS is removed when it is out of support from the provider as outlined here. For Debian Stretch, this support ran from 2017 to 2022.

Moving on to the main question, docker in docker support, this is unfortunately something we don't support so are unable to provide feedback on at this time. It does look like there have been updates to the linked repo and I can see you are engaged there, hopefully someone else in the community can provide some insight.

Let me know if you made any more progress here.

[ryanwinter]

Close issue, reactive if you have any more questions.