Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot resolve edgeAgent and edgeHub storage warning #2880

Closed
mahdighorbanpour opened this issue Apr 24, 2020 · 22 comments
Closed

Cannot resolve edgeAgent and edgeHub storage warning #2880

mahdighorbanpour opened this issue Apr 24, 2020 · 22 comments
Labels
customer-reported iotedge needinfo Needs more info from issue reporter / updates from PR submitter no-issue-activity

Comments

@mahdighorbanpour
Copy link

mahdighorbanpour commented Apr 24, 2020

Expected Behavior

iotedge check command should display storage settings resolved and mark ok (green).

Current Behavior

I still get thses warnings :

‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its C:\Windows\Temp\edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its C:\\Windows\\Temp\edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

  1. create a normal deployment with no custom modules
  2. define binds like example in the provided link. here is my deployment json file:
{
    "modulesContent": {
        "$edgeAgent": {
            "properties.desired": {
                "schemaVersion": "1.0",
                "runtime": {
                    "type": "docker",
                    "settings": {
                        "minDockerVersion": "v1.25",
                    }
                },
                "systemModules": {
                    "edgeAgent": {
                        "settings": {
                            "image": "mcr.microsoft.com/azureiotedge-agent:1.0.9",
                            "createOptions": "{\"HostConfig\": {\"Binds\":[\"C:\\\\temp\\\\edgeAgent:C:\\\\Windows\\\\Temp\\\\edgeAgent\"]}}"
                        },
                        "type": "docker",
                        "env": {
                            "experimentalfeatures__enabled": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableMetrics": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableGetLogs": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableUploadLogs": {
                                "value": "true"
                            },
                            "storageFolder": {
                                "value": "C:\\\\Windows\\\\Temp"
                            }
                        }
                    },
                    "edgeHub": {
                        "settings": {
                            "image": "mcr.microsoft.com/azureiotedge-hub:1.0.9",
                            "createOptions": "{\"HostConfig\":{\"Binds\":[\"C:\\\\temp\\\\edgeHub:C:\\\\Windows\\\\Temp\\\\edgeHub\"],\"PortBindings\":{\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
                        },
                        "type": "docker",
                        "status": "running",
                        "restartPolicy": "always",
                        "env": {
                            "storageFolder": {
                                "value": "C:\\\\Windows\\\\Temp"
                            }
                        }
                    }
                },
                "modules":{}
            }
        },
        "$edgeHub": {
            "properties.desired": {
                "routes": {},
                "schemaVersion": "1.0",
                "storeAndForwardConfiguration": {
                    "timeToLiveSecs": 7200
                }
            }
        }
    }
}
  1. after edge device received the deployment, run iotedge check

Context (Environment)

Windows 10

Output of iotedge check

√ config.yaml is well-formed - OK
‼ config.yaml has well-formed connection string - Warning
    Device not configured with manual provisioning, in this configuration 'iotedge check' is not able to discover the device's backing IoT Hub.
    To run connectivity checks in this configuration please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    If no hostname is provided, all hub connectivity tests will be skipped.
√ container engine is installed and functional - OK
√ Windows host version is supported - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: certificates - OK
√ production readiness: container engine - OK
√ production readiness: logs policy - OK
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its C:\Windows\Temp\edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its C:\\Windows\\Temp\edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with DPS endpoint - OK

13 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.
6 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

  • Host OS : Windows 10
  • Architecture : amd64
  • Container OS : Windows containers

Runtime Versions

  • iotedged : iotedge 1.0.9 (1f7cf6a)
  • Edge Agent [image tag (e.g. 1.0.0)]:
  • Edge Hub [image tag (e.g. 1.0.0)]:
  • Docker/Moby [run docker version]: 3.0.5

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

iotedged logs

<Paste here between the triple backticks>

edge-agent logs

2020-04-24 10:13:11.112 +00:00 Edge Agent Main()
<6> 2020-04-24 10:13:11.990 +00:00 [INF] - Initializing Edge Agent.
<6> 2020-04-24 10:13:12.693 +00:00 [INF] - Version - 1.0.9.29532150 (1f7cf6a21dfa365c8f21475553c544f4df02dd44)
<6> 2020-04-24 10:13:12.694 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2020-04-24 10:13:13.008 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"EnableUploadLogs":false,"EnableGetLogs":false,"EnableMetrics":false}
<6> 2020-04-24 10:13:13.745 +00:00 [INF] - Installing certificates [CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-24 10:13:15.687 +00:00 [INF] - Created persistent store at C:\Windows\TEMP\edgeAgent
<6> 2020-04-24 10:13:16.108 +00:00 [INF] - Started operation refresh twin config
<6> 2020-04-24 10:13:16.283 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2020-04-24 10:13:16.509 +00:00 [INF] - Registering request handler RestartModule
<6> 2020-04-24 10:13:19.151 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2020-04-24 10:13:19.524 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2020-04-24 10:13:19.764 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 46 and reported properties version 617.
<6> 2020-04-24 10:13:20.976 +00:00 [INF] - Plan execution started for deployment 46
<6> 2020-04-24 10:13:21.058 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2020-04-24 10:13:21.085 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2020-04-24 10:13:35.589 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2020-04-24 10:13:39.428 +00:00 [INF] - Plan execution ended for deployment 46
<6> 2020-04-24 10:13:40.379 +00:00 [INF] - Updated reported properties
<6> 2020-04-24 10:13:45.792 +00:00 [INF] - Updated reported properties

edge-hub logs

2020-04-24 10:13:41.172 +00:00 Edge Hub Main()
<6> 2020-04-24 10:13:45.495 +00:00 [INF] - Installing certificates [CN=iotedged workload ca:5/22/2020 8:39:54 AM],[CN=MyEdgeDeviceCA.ca:5/22/2020 8:39:54 AM],[CN=Azure IoT CA TestOnly Intermediate CA:5/22/2020 8:38:02 AM],[CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-24 10:13:45.693 +00:00 [INF] - Installing certificates [CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-24 10:13:49.147 +00:00 [INF] - Created persistent store at C:\\Windows\\Temp\edgeHub
<6> 2020-04-24 10:13:49.347 +00:00 [INF] - Initializing Edge Hub
<6> 2020-04-24 10:13:49.384 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2020-04-24 10:13:49.393 +00:00 [INF] - Version - 1.0.9.29532150 (1f7cf6a21dfa365c8f21475553c544f4df02dd44)
<6> 2020-04-24 10:13:49.393 +00:00 [INF] - OptimizeForPerformance=True
<6> 2020-04-24 10:13:49.403 +00:00 [INF] - Loaded server certificate with expiration date of "2020-05-22T08:39:54.0000000+00:00"
<6> 2020-04-24 10:13:49.623 +00:00 [INF] - Created new message store
<6> 2020-04-24 10:13:49.694 +00:00 [INF] - Created device scope identities cache
<6> 2020-04-24 10:13:49.733 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2020-04-24 10:13:50.113 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2020-04-24 10:13:50.315 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2020-04-24 10:13:50.716 +00:00 [INF] - Initialized storing twin manager
<6> 2020-04-24 10:13:50.765 +00:00 [INF] - Initializing configuration
<6> 2020-04-24 10:13:50.831 +00:00 [INF] - New device connection for device p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:50.867 +00:00 [INF] - Processing pending subscriptions for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:51.197 +00:00 [INF] - Attempting to connect to IoT Hub for client p2-az-220-hu/$edgeHub via AMQP...
<4> 2020-04-24 10:13:51.474 +00:00 [WRN] - Empty edge hub configuration received. Ignoring...
<6> 2020-04-24 10:13:53.577 +00:00 [INF] - Exiting disconnected state
<6> 2020-04-24 10:13:53.587 +00:00 [INF] - Received device connected callback
<6> 2020-04-24 10:13:53.624 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2020-04-24 10:13:53.634 +00:00 [INF] - Processing subscriptions for client p2-az-220-hu/$edgeHub on device connected.
<6> 2020-04-24 10:13:53.690 +00:00 [INF] - Entering connected state
<6> 2020-04-24 10:13:53.694 +00:00 [INF] - Cloud connection for p2-az-220-hu/$edgeHub is True
<6> 2020-04-24 10:13:53.713 +00:00 [INF] - Connection status for p2-az-220-hu/$edgeHub changed to ConnectionEstablished
<6> 2020-04-24 10:13:53.721 +00:00 [INF] - Client p2-az-220-hu/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2020-04-24 10:13:53.732 +00:00 [INF] - Created cloud proxy for client p2-az-220-hu/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2020-04-24 10:13:53.751 +00:00 [INF] - Initialized cloud proxy dbc04422-26bb-4e01-b710-7cfc7581628e for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:53.752 +00:00 [INF] - Created cloud connection for client p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:54.360 +00:00 [INF] - Processing pending subscriptions for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:54.695 +00:00 [INF] - Updated reported properties for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:54.737 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2020-04-24 10:13:54.780 +00:00 [INF] - No routes set in the edge hub
<6> 2020-04-24 10:13:54.781 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2020-04-24 10:13:54.781 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2020-04-24 10:13:54.805 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2020-04-24 10:13:54.806 +00:00 [INF] - Initialized edge hub configuration
<6> 2020-04-24 10:13:54.843 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2020-04-24 10:13:55.137 +00:00 [INF] - Scheduling server certificate renewal for "2020-05-22T08:37:24.0000865Z".
<6> 2020-04-24 10:13:55.139 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2020-04-24 10:13:55.148 +00:00 [INF] - Starting MQTT head
<6> 2020-04-24 10:13:55.251 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2020-04-24 10:13:55.476 +00:00 [INF] - Starting AMQP head
<6> 2020-04-24 10:13:55.536 +00:00 [INF] - Started MQTT head
<6> 2020-04-24 10:13:55.848 +00:00 [INF] - Updated reported properties for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:55.867 +00:00 [INF] - Started AMQP head
<6> 2020-04-24 10:13:55.876 +00:00 [INF] - Starting HTTP head
<6> 2020-04-24 10:13:56.745 +00:00 [INF] - User profile is available. Using '"C:\\Users\\ContainerUser\\AppData\\Local\\ASP.NET\\DataProtection-Keys"' as key repository and Windows DPAPI to encrypt keys at rest.
<6> 2020-04-24 10:13:56.938 +00:00 [INF] - Creating key {4e0ce94c-9512-4e55-855f-f43d5960e435} with creation date 2020-04-24 10:13:56Z, activation date 2020-04-24 10:13:56Z, and expiration date 2020-07-23 10:13:56Z.
<6> 2020-04-24 10:13:57.297 +00:00 [INF] - Writing data to file '"C:\\Users\\ContainerUser\\AppData\\Local\\ASP.NET\\DataProtection-Keys\\key-4e0ce94c-9512-4e55-855f-f43d5960e435.xml"'.
<4> 2020-04-24 10:13:58.666 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined in "UseKestrel()" instead.
<6> 2020-04-24 10:13:58.743 +00:00 [INF] - Started HTTP head
<6> 2020-04-24 10:18:54.874 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2020-04-24 10:23:54.867 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2020-04-24 10:28:08.382 +00:00 [INF] - Request starting HTTP/1.1 GET https://52.178.152.23/cgi-bin/luci
<6> 2020-04-24 10:28:08.710 +00:00 [INF] - Request finished in 347.0905ms 401 text/html
<6> 2020-04-24 10:28:11.314 +00:00 [INF] - Request starting HTTP/1.1 GET https://52.178.152.23/dana-na/auth/url_default/welcome.cgi
<6> 2020-04-24 10:28:11.314 +00:00 [INF] - Request finished in 0.363ms 401 text/html
<6> 2020-04-24 10:28:15.003 +00:00 [INF] - Request starting HTTP/1.1 GET https://52.178.152.23/remote/login?lang=en
<6> 2020-04-24 10:28:15.003 +00:00 [INF] - Request finished in 0.2092ms 401 text/html
<6> 2020-04-24 10:28:25.260 +00:00 [INF] - HttpExtensionConnectionAdapter authentication failed
<6> 2020-04-24 10:28:54.884 +00:00 [INF] - Entering periodic task to reauthenticate connected clients

Additional Information

I also tried docker inspect, on edgeHub the bind is listed and also the environment variable exists. but on edgeHub not affected.

@arsing
Copy link
Member

arsing commented Apr 24, 2020

                           "storageFolder": {
                               "value": "C:\\\\Windows\\\\Temp"
                           }

Both instances of this should be be:

                            "storageFolder": {
                                "value": "C:\\Windows\\Temp"
                            }

I also tried docker inspect, on edgeHub the bind is listed and also the environment variable exists. but on edgeHub not affected.

I assume you meant "but on edgeAgent not affected". For that, ensure you config.yaml also has storageFolder set in the agent.env section, like:

agent:
  name: 'edgeAgent'
  type: 'docker'
  env:
    'storageFolder': 'C:\Windows\temp'

Notice again that the \ is not doubled-up, because it's inside a YAML string quoted with '. Only YAML strings quoted with " need \ to be escaped as \\.

@arsing arsing added the needinfo Needs more info from issue reporter / updates from PR submitter label Apr 24, 2020
@mahdighorbanpour
Copy link
Author

I updated the deployment.json file as you said:

{
    "modulesContent": {
        "$edgeAgent": {
            "properties.desired": {
                "schemaVersion": "1.0",
                "runtime": {
                    "type": "docker",
                    "settings": {
                        "minDockerVersion": "v1.25" 
                    }
                },
                "systemModules": {
                    "edgeAgent": {
                        "settings": {
                            "image": "mcr.microsoft.com/azureiotedge-agent:1.0.9",
                            "createOptions": "{\"HostConfig\": {\"Binds\":[\"C:\\\\temp\\\\edgeAgent:C:\\\\Windows\\\\Temp\\\\edgeAgent\"]}}"
                        },
                        "type": "docker",
                        "env": {
                            "experimentalfeatures__enabled": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableMetrics": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableGetLogs": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableUploadLogs": {
                                "value": "true"
                            },
                            "storageFolder": {
                                "value": "C:\\Windows\\Temp"
                            }
                        }
                    },
                    "edgeHub": {
                        "settings": {
                            "image": "mcr.microsoft.com/azureiotedge-hub:1.0.9",
                            "createOptions": "{\"HostConfig\":{\"Binds\":[\"C:\\\\temp\\\\edgeHub:C:\\\\Windows\\\\Temp\\\\edgeHub\"],\"PortBindings\":{\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
                        },
                        "type": "docker",
                        "status": "running",
                        "restartPolicy": "always",
                        "env": {
                            "storageFolder": {
                                "value": "C:\\Windows\\Temp"
                            }
                        }
                    }
                },
                "modules":{}
            }
        },
        "$edgeHub": {
            "properties.desired": {
                "routes": {},
                "schemaVersion": "1.0",
                "storeAndForwardConfiguration": {
                    "timeToLiveSecs": 7200
                }
            }
        }
    }
}

Your assumption was correct. I meant edgeAgent , for that also I changed config.yaml. nd restarted the iotedge service. no changes happened. I still get the warnings. here is the docker inspect of edgeAgent:

 "HostConfig": {
            "Binds": [
                "C:\\ProgramData\\iotedge\\mgmt:C:\\ProgramData\\iotedge\\mgmt",
                "C:\\ProgramData\\iotedge\\workload:C:\\ProgramData\\iotedge\\workload"
            ],
			...
			 "Mounts": [
            {
                "Type": "bind",
                "Source": "c:\\programdata\\iotedge\\mgmt",
                "Destination": "c:\\programdata\\iotedge\\mgmt",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "c:\\programdata\\iotedge\\workload",
                "Destination": "c:\\programdata\\iotedge\\workload",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
		...
		"Env": [
                "IOTEDGE_APIVERSION=2019-11-05",
                "IOTEDGE_MANAGEMENTURI=unix:///C:/ProgramData/iotedge/mgmt/sock",
                "IOTEDGE_MODULEGENERATIONID=637219283304799162",
                "IOTEDGE_WORKLOADURI=unix:///C:/ProgramData/iotedge/workload/sock",
                "IOTEDGE_AUTHSCHEME=sasToken",
                "NetworkId=nat",
                "Mode=iotedged",
                "IOTEDGE_IOTHUBHOSTNAME=xxx.azure-devices.net",
                "IOTEDGE_DEVICEID=p2-az-220-hu",
                "EDGEDEVICEHOSTNAME=az-220-gw-hu",
                "IOTEDGE_MODULEID=$edgeAgent",
                "ASPNETCORE_URLS=http://+:80",
                "DOTNET_RUNNING_IN_CONTAINER=true"
            ],
			

and here is the docker inspect of edgHub

 "HostConfig": {
            "Binds": [
                "C:\\temp\\edgeHub:C:\\Windows\\Temp\\edgeHub",
                "C:\\ProgramData\\iotedge\\workload:C:\\ProgramData\\iotedge\\workload"
            ],
			...
			  "Mounts": [
            {
                "Type": "bind",
                "Source": "c:\\temp\\edgehub",
                "Destination": "c:\\windows\\temp\\edgehub",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "c:\\programdata\\iotedge\\workload",
                "Destination": "c:\\programdata\\iotedge\\workload",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
		...
		"Env": [
                "storageFolder=C:\\Windows\\Temp",
                "IOTEDGE_MODULEID=$edgeHub",
                "IOTEDGE_APIVERSION=2019-11-05",
                "IOTEDGE_IOTHUBHOSTNAME=xxx.azure-devices.net",
                "IOTEDGE_MODULEGENERATIONID=637219283304799162",
                "IOTEDGE_WORKLOADURI=unix:///C:/ProgramData/iotedge/workload/sock",
                "RuntimeLogLevel=Information",
                "IOTEDGE_AUTHSCHEME=sasToken",
                "EdgeDeviceHostName=az-220-gw-hu",
                "IOTEDGE_DEVICEID=p2-az-220-hu",
                "ASPNETCORE_URLS=http://+:80",
                "DOTNET_RUNNING_IN_CONTAINER=true"
            ],

@mahdighorbanpour
Copy link
Author

I noticed after a while I am getting this error :

2020-04-27 05:27:29.040 +00:00 Edge Agent Main()
<6> 2020-04-27 05:27:30.028 +00:00 [INF] - Initializing Edge Agent.
<6> 2020-04-27 05:27:30.803 +00:00 [INF] - Version - 1.0.9.29532150 (1f7cf6a21dfa365c8f21475553c544f4df02dd44)
<6> 2020-04-27 05:27:30.804 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2020-04-27 05:27:31.060 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"EnableUploadLogs":false,"EnableGetLogs":false,"EnableMetrics":false}
<6> 2020-04-27 05:27:31.841 +00:00 [INF] - Installing certificates [CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-27 05:27:34.127 +00:00 [INF] - Created persistent store at C:\Windows\TEMP\edgeAgent
<6> 2020-04-27 05:27:34.544 +00:00 [INF] - Started operation refresh twin config
<6> 2020-04-27 05:27:34.709 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2020-04-27 05:27:34.950 +00:00 [INF] - Registering request handler RestartModule
<6> 2020-04-27 05:27:37.420 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2020-04-27 05:27:37.796 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2020-04-27 05:27:38.103 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 47 and reported properties version 635.
<6> 2020-04-27 05:27:39.303 +00:00 [INF] - Plan execution started for deployment 47
<6> 2020-04-27 05:27:39.380 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2020-04-27 05:27:39.401 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2020-04-27 05:27:39.786 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2020-04-27 05:27:43.141 +00:00 [INF] - Plan execution ended for deployment 47
<6> 2020-04-27 05:27:44.057 +00:00 [INF] - Updated reported properties
<6> 2020-04-27 05:27:49.352 +00:00 [INF] - Updated reported properties
<6> 2020-04-27 06:27:34.627 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2020-04-27 06:27:34.633 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 47 and reported properties version 637.
<6> 2020-04-27 06:27:34.636 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<4> 2020-04-27 06:47:16.096 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout) in C:\agent\_work\4\s\edge-util\src\Microsoft.Azure.Devices.Edge.Util\TaskEx.cs:line 117
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Edgelet\versioning\ModuleManagementHttpClientVersioned.cs:line 127
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_11_05.ModuleManagementHttpClient.GetModules[T](CancellationToken cancellationToken) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Edgelet\version_2019_11_05\ModuleManagementHttpClient.cs:line 153
   at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerEnvironment.GetModulesAsync(CancellationToken token) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Docker\DockerEnvironment.cs:line 50
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.GetCurrentModuleSetAsync(CancellationToken token) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Core\Agent.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Core\Agent.cs:line 123
<6> 2020-04-27 06:47:16.696 +00:00 [INF] - Updated reported properties
<6> 2020-04-27 06:47:21.836 +00:00 [INF] - Updated reported properties
<6> 2020-04-27 07:27:34.346 +00:00 [INF] - Starting compaction of stores
<6> 2020-04-27 07:27:34.347 +00:00 [INF] - Starting compaction of store default
<6> 2020-04-27 07:27:34.355 +00:00 [INF] - Starting compaction of store deploymentConfig
<6> 2020-04-27 07:27:34.355 +00:00 [INF] - Starting compaction of store moduleState
<6> 2020-04-27 07:27:34.758 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2020-04-27 07:27:34.845 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 47 and reported properties version 639.
<6> 2020-04-27 07:27:34.846 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<4> 2020-04-27 08:11:40.328 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timed out
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout) in C:\agent\_work\4\s\edge-util\src\Microsoft.Azure.Devices.Edge.Util\TaskEx.cs:line 117
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Edgelet\versioning\ModuleManagementHttpClientVersioned.cs:line 127
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_11_05.ModuleManagementHttpClient.GetModules[T](CancellationToken cancellationToken) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Edgelet\version_2019_11_05\ModuleManagementHttpClient.cs:line 153
   at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerEnvironment.GetModulesAsync(CancellationToken token) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Docker\DockerEnvironment.cs:line 50
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.GetCurrentModuleSetAsync(CancellationToken token) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Core\Agent.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in C:\agent\_work\4\s\edge-agent\src\Microsoft.Azure.Devices.Edge.Agent.Core\Agent.cs:line 123
<6> 2020-04-27 08:11:40.450 +00:00 [INF] - Updated reported properties
<6> 2020-04-27 08:11:45.593 +00:00 [INF] - Updated reported properties

@yophilav
Copy link
Contributor

Hi Mahdi,

It is interesting that storageFolder doesn't seem to get set on the edgeAgent.
We have a known bug that an edgeAgent environment variable is not going updated unless the edgeAgent is redeployed. From your comment, it seemed like you might only restarted the iotedge service.

Could you please try docker rm -f edgeAgent and restart the iotedge service? This would ensure the edgeAgent is being redeployed.

Thank you!

@mahdighorbanpour
Copy link
Author

I created a new Azure windows 10 vm and installed a fresh iotEdge runtime. Retried with the same deployment package. and also tried removing edgeAgent container, but no luck!
When I navigate to c:\temp\edgeHub has already some files in it, but edgeAgent folder is still empty

@yophilav
Copy link
Contributor

yophilav commented May 4, 2020

Do you see storageFolder being set correctly on edgeAgent when you do docker inspect edgeAgent after docker rm -f edgeAgent?

@mahdighorbanpour
Copy link
Author

mahdighorbanpour commented May 5, 2020

Here is the docker inspect results:

docker inpect
    {
        "Id": "xxxx",
        "Created": "2020-05-05T11:41:48.3065884Z",
        "Path": "dotnet",
        "Args": [
            "Microsoft.Azure.Devices.Edge.Agent.Service.dll"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4244,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-05-05T11:41:59.7027334Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:cab1bd67d4d7d71b6eedcbfa5382e8e784881bef26aab1ed846974191381ef25",
        "ResolvConfPath": "",
        "HostnamePath": "",
        "HostsPath": "",
        "LogPath": "C:\\ProgramData\\iotedge-moby\\containers\\xxxx\\xxxx-json.log",
        "Name": "/edgeAgent",
        "RestartCount": 0,
        "Driver": "windowsfilter",
        "Platform": "windows",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "C:\\ProgramData\\iotedge\\mgmt:C:\\ProgramData\\iotedge\\mgmt",
                "C:\\ProgramData\\iotedge\\workload:C:\\ProgramData\\iotedge\\workload"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 0,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "process",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "dir": "C:\\ProgramData\\iotedge-moby\\windowsfilter\\38e843d5f4cc55c0b1ee6d724e495eb87dd1c87faa260f8d957b38d7a4930184"
            },
            "Name": "windowsfilter"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "c:\\programdata\\iotedge\\mgmt",
                "Destination": "c:\\programdata\\iotedge\\mgmt",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "c:\\programdata\\iotedge\\workload",
                "Destination": "c:\\programdata\\iotedge\\workload",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "38e843d5f4cc",
            "Domainname": "",
            "User": "ContainerAdministrator",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "IOTEDGE_AUTHSCHEME=sasToken",
                "IOTEDGE_WORKLOADURI=unix:///C:/ProgramData/iotedge/workload/sock",
                "EDGEDEVICEHOSTNAME=p7-win10-tst-gp",
                "IOTEDGE_MANAGEMENTURI=unix:///C:/ProgramData/iotedge/mgmt/sock",
                "IOTEDGE_MODULEGENERATIONID=637241971008587773",
                "NetworkId=nat",
                "Mode=iotedged",
                "IOTEDGE_APIVERSION=2019-11-05",
                "IOTEDGE_DEVICEID=xxxx",
                "IOTEDGE_IOTHUBHOSTNAME=xxx.azure-devices.net",
                "storageFolder=C:\\Windows\\temp",
                "IOTEDGE_MODULEID=$edgeAgent",
                "ASPNETCORE_URLS=http://+:80",
                "DOTNET_RUNNING_IN_CONTAINER=true"
            ],
            "Cmd": [
                "dotnet",
                "Microsoft.Azure.Devices.Edge.Agent.Service.dll"
            ],
            "ArgsEscaped": true,
            "Image": "mcr.microsoft.com/azureiotedge-agent:1.0.9",
            "Volumes": null,
            "WorkingDir": "C:\\app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "net.azure-devices.edge.owner": "Microsoft.Azure.Devices.Edge.Agent"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "38e843d5f4cc55c0b1ee6d724e495eb87dd1c87faa260f8d957b38d7a4930184",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "38e843d5f4cc55c0b1ee6d724e495eb87dd1c87faa260f8d957b38d7a4930184",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "nat": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "6f673444e5660d72b9f4a334ea994726c411bdd9e5ca749f2a565b0cbbb419cf",
                    "EndpointID": "eebbdeb86fd969d929a02abf4e91ab62ec8e615c236d2ce6f59716587ca212c8",
                    "Gateway": "172.23.192.1",
                    "IPAddress": "172.23.195.162",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "00:15:5d:e0:79:0e",
                    "DriverOpts": null
                }
            }
        }
    }

@thnk2wn
Copy link

thnk2wn commented May 6, 2020

I have the same problem on Linux. I went through Give modules access to a device's local storage and updated my deployment templates. Afterwards the edgeHub storage warning went away but not edgeAgent. They both have the same settings in deployment template but only edgeHub gets the env var set not edgeAgent.

"systemModules": {
    "edgeAgent": {
        "type": "docker",
        "settings": {
            "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
            "createOptions": {
                "HostConfig": {
                    "Binds": [
                        "/etc/iotedge/storage/:/iotedge/storage/"
                    ]
                }
            }
        },
        "env": {
            "storageFolder": {
                "value": "/iotedge/storage/"
            }
        }
    },
    "edgeHub": {
        "type": "docker",
        "status": "running",
        "restartPolicy": "always",
        "settings": {
            "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
            "createOptions": {
                "HostConfig": {
                    "Binds": [
                        "/etc/iotedge/storage/:/iotedge/storage/"
                    ],
                    "PortBindings": {
                        "5671/tcp": [
                            {
                                "HostPort": "5671"
                            }
                        ],
                        "8883/tcp": [
                            {
                                "HostPort": "8883"
                            }
                        ],
                        "443/tcp": [
                            {
                                "HostPort": "443"
                            }
                        ]
                    }
                }
            }
        },
        "env": {
            "storageFolder": {
                "value": "/iotedge/storage/"
            }
        }
    }
}

I also tried config.yaml tweaks, same behavior.

agent:
  name: "edgeAgent"
  type: "docker"
  env:
    storageFolder: "/etc/iotedge/storage"
  config:
    image: "mcr.microsoft.com/azureiotedge-agent:1.0"
    auth: {}

@SamanSattari
Copy link

Similar issue on Linux.. docker inspect edgeAgent shows that the env storage is not set. Removing doesn't help. Settings are the same as @thnk2wn
edgeHub can access the storage. Only edgeAgent seems not capable of picking up the settings.

@yophilav
Copy link
Contributor

@thnk2wn @SamanSattari

  1. Please set the storageFolder to a desired value
  2. Remove the edgeAgent docker using docker rm -f edgeAgent
  3. Just to be sure, you can restart iotedge using sudo systemctl stop iotedge && sudo systemctl start iotedge
  4. Use docker inspect edgeAgent to make sure the variable is set.

@SamanSattari
Copy link

@yophilav I have followed these steps.
After performing these steps, still the env value and hostConfigs in docker inspect do not have storageFolder and binds. However these values are reported to the IotHub.
Actually adding any other environment variable or hostConfig is not reflected when I do the inspect
Here is what I see for binds when I call inspect:

"Binds": [
                "/var/run/iotedge/mgmt.sock:/var/run/iotedge/mgmt.sock",
                "/var/run/iotedge/workload.sock:/var/run/iotedge/workload.sock"
            ],

and for env

Env": [
                "IOTEDGE_APIVERSION=2019-11-05",
                "IOTEDGE_AUTHSCHEME=sasToken",
                "IOTEDGE_MANAGEMENTURI=unix:///var/run/iotedge/mgmt.sock",
                "Mode=iotedged",
                "IOTEDGE_DEVICEID={my-iot-edge}",
                "EDGEDEVICEHOSTNAME={myVM}",
                "NetworkId=azure-iot-edge",
                "IOTEDGE_WORKLOADURI=unix:///var/run/iotedge/workload.sock",
                "IOTEDGE_IOTHUBHOSTNAME={myHostName}.azure-devices.net",
                "IOTEDGE_MODULEID=$edgeAgent",
                "IOTEDGE_MODULEGENERATIONID=637226471648120960",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ASPNETCORE_URLS=http://+:80",
                "DOTNET_RUNNING_IN_CONTAINER=true",
                "DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true",
                "DOTNET_VERSION=2.1.17"
            ]

and what I see in the IotHub:
image
image

@jefim
Copy link

jefim commented May 29, 2020

Hello @SamanSattari , did you get some resolution for this? I am facing exact same symptoms.

@SamanSattari
Copy link

@jefim no unfortunately..
I actually fixed my log policy by updating daeman.json.
And for the storage folder I haven't thought of any solution yet.

@jefim
Copy link

jefim commented Jun 4, 2020

@yophilav Do you have any other insights for this or is this somewhere under inspection? Any possible news? :)

@ssvillamor23
Copy link

Hi all,
I ?manage to fix temporarily by having one, i forgot if its the agent or the edgehub. One is version to 1.0.9.2and the other is 1.0.9. then modding the folder of your path accessible.

@github-actions
Copy link

This issue is being marked as stale because it has been open for 30 days with no activity.

@ralarcon
Copy link

Hi, I'm running on the exact same issue on linux machine... I've tried the proposed remediation steps without success.... any update on this??

@kpm-at-hfi
Copy link

Same... @veyalla perhaps this should be re-opened and closed when the docs are better or more complete?

@danwale
Copy link

danwale commented Sep 3, 2020

I've also hit this same issue on a Linux machine, following the steps of removing the edgeAgent docker image and having it re-provision did not resolve the issue. While Azure's IoT Hub reports back the storageFolder environment variable and the binds configuration for the edgeAgent modules createOptions those aren't reflected in docker inspect edgeAgent and when running iotedge check it still warns that the edge agents storage directory is not being persisted on the host filesystem.

@veyalla
Copy link
Contributor

veyalla commented Sep 3, 2020

I couldn’t repro with 1.0.10-rc2. We made some edgeAgent fixes in the 1.0.10 which may be taking care of this issue. Please attempt repro with 1.0.10-rc2, if possible.

@emsuhue
Copy link

emsuhue commented Sep 14, 2020

given that this problem persists on devices deployed with 1.0.9, could the documentation be changed to reflect this unexpected behaviour?

@mahdighorbanpour
Copy link
Author

I tried with newest release candidate version and seems the problem exists! this time I noticed actually the files are persisted in the destination directory but again warnings are there! both for edgeAgent and edgeHub

deployment manifest ```json { "modulesContent": { "$edgeAgent": { "properties.desired": { "schemaVersion": "1.0", "runtime": { "type": "docker", "settings": { "minDockerVersion": "v1.25", } }, "systemModules": { "edgeAgent": { "settings": { "image": "mcr.microsoft.com/azureiotedge-agent:1.0.10-rc2", "createOptions": "{\"HostConfig\": {\"Binds\":[\"C:\\\\ProgramData\\\\xxx:C:\\\\Windows\\\\Temp\"]}}" }, "type": "docker", "env": { "experimentalfeatures__enabled": { "value": "true" }, "experimentalfeatures__enableMetrics": { "value": "true" }, "experimentalfeatures__enableGetLogs": { "value": "true" }, "experimentalfeatures__enableUploadLogs": { "value": "true" }, "storageFolder": { "value": "C:\\Windows\\Temp" } } }, "edgeHub": { "settings": { "image": "mcr.microsoft.com/azureiotedge-hub:1.0.10-rc2", "createOptions": "{\"HostConfig\":{\"Binds\":[\"C:\\\\ProgramData\\\\xxx:C:\\\\Windows\\\\Temp\"],\"PortBindings\":{\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}" }, "type": "docker", "status": "running", "restartPolicy": "always", "env": { "storageFolder": { "value": "C:\\Windows\\Temp" } } } }, "modules": {} } }, "$edgeHub": { "properties.desired": { "routes": {}, "schemaVersion": "1.0", "storeAndForwardConfiguration": { "timeToLiveSecs": 7200 } } } } } ```
docker inpect edgeAgent ```json PS C:\Windows\system32> docker -H npipe:////./pipe/iotedge_moby_engine inspect edgeAgent [ { "Id": "fa2c580190ff95aa33c6f9a626fe6da647df13728abd8d145cd58171317ba54c", "Created": "2020-09-25T12:19:47.5405692Z", "Path": "dotnet", "Args": [ "Microsoft.Azure.Devices.Edge.Agent.Service.dll" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 2476, "ExitCode": 0, "Error": "", "StartedAt": "2020-09-25T12:19:48.3904994Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:437b35604dc741fe734da408064b5e51b3ed9a28ed9724736e40d8d0e3e28915", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "C:\\ProgramData\\iotedge-moby\\containers\\fa2c580190ff95aa33c6f9a626fe6da647df13728abd8d145cd58171317ba54c\\fa2c580190ff95aa33c6f9a626fe6da647df13728abd8d145cd58171317ba54c-json.log", "Name": "/edgeAgent", "RestartCount": 0, "Driver": "windowsfilter", "Platform": "windows", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "C:\\ProgramData\\xxx:C:\\Windows\\Temp", "C:\\ProgramData\\iotedge\\workload:C:\\ProgramData\\iotedge\\workload", "C:\\ProgramData\\iotedge\\mgmt:C:\\ProgramData\\iotedge\\mgmt" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": null, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 0, "ConsoleSize": [ 0, 0 ], "Isolation": "process", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": null, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "dir": "C:\\ProgramData\\iotedge-moby\\windowsfilter\\fa2c580190ff95aa33c6f9a626fe6da647df13728abd8d145cd58171317ba54c" }, "Name": "windowsfilter" }, "Mounts": [ { "Type": "bind", "Source": "c:\\programdata\\xxx", "Destination": "c:\\windows\\temp", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\programdata\\iotedge\\workload", "Destination": "c:\\programdata\\iotedge\\workload", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\programdata\\iotedge\\mgmt", "Destination": "c:\\programdata\\iotedge\\mgmt", "Mode": "", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "fa2c580190ff", "Domainname": "", "User": "ContainerAdministrator", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "EdgeDeviceHostName=desktop-uldmue5", "IOTEDGE_APIVERSION=2020-07-07", "IOTEDGE_AUTHSCHEME=sasToken", "IOTEDGE_DEVICEID=di-7966", "IOTEDGE_IOTHUBHOSTNAME=xxxIoTHubDev.azure-devices.net", "IOTEDGE_MANAGEMENTURI=unix:///C:/ProgramData/iotedge/mgmt/sock", "IOTEDGE_MODULEGENERATIONID=637366330356525195", "IOTEDGE_MODULEID=$edgeAgent", "IOTEDGE_WORKLOADURI=unix:///C:/ProgramData/iotedge/workload/sock", "Mode=iotedged", "NetworkId=nat", "RuntimeLogLevel=Information", "experimentalfeatures__enableGetLogs=true", "experimentalfeatures__enableMetrics=true", "experimentalfeatures__enableUploadLogs=true", "experimentalfeatures__enabled=true", "storageFolder=C:\\Windows\\Temp", "ASPNETCORE_URLS=http://+:80", "DOTNET_RUNNING_IN_CONTAINER=true" ], "Cmd": [ "dotnet", "Microsoft.Azure.Devices.Edge.Agent.Service.dll" ], "ArgsEscaped": true, "Image": "mcr.microsoft.com/azureiotedge-agent:1.0.10-rc2", "Volumes": null, "WorkingDir": "C:\\app", "Entrypoint": null, "OnBuild": null, "Labels": { "net.azure-devices.edge.owner": "Microsoft.Azure.Devices.Edge.Agent" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "fa2c580190ff95aa33c6f9a626fe6da647df13728abd8d145cd58171317ba54c", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "fa2c580190ff95aa33c6f9a626fe6da647df13728abd8d145cd58171317ba54c", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "nat": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "78af2a2fd584edac0bbfeb464a61bbdec9be713c2817410fb8ddbef78e41b212", "EndpointID": "69a32fccc83e65910212f066028a63c18ffdde4fdab3a579d8a65f1d7a44bda8", "Gateway": "172.17.224.1", "IPAddress": "172.17.230.228", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "00:15:5d:f7:f2:78", "DriverOpts": null } } } } ] ```
docker inpect edgeHub ```json PS C:\Windows\system32> docker -H npipe:////./pipe/iotedge_moby_engine inspect edgeHub [ { "Id": "56f81486a2c0372bfe0494e0a6c1e17c6d0a76bb545e546ecf605583390cb1c0", "Created": "2020-09-25T12:20:04.412838Z", "Path": "dotnet", "Args": [ "Microsoft.Azure.Devices.Edge.Hub.Service.dll" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 4860, "ExitCode": 0, "Error": "", "StartedAt": "2020-09-25T12:20:05.6047546Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:5202ad14db881a45329c85f9f3e3a3fd17fb14c55e3d11a8f0be26af029507ed", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "C:\\ProgramData\\iotedge-moby\\containers\\56f81486a2c0372bfe0494e0a6c1e17c6d0a76bb545e546ecf605583390cb1c0\\56f81486a2c0372bfe0494e0a6c1e17c6d0a76bb545e546ecf605583390cb1c0-json.log", "Name": "/edgeHub", "RestartCount": 0, "Driver": "windowsfilter", "Platform": "windows", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "C:\\ProgramData\\xxx:C:\\Windows\\Temp", "C:\\ProgramData\\iotedge\\workload:C:\\ProgramData\\iotedge\\workload" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": { "443/tcp": [ { "HostIp": "", "HostPort": "443" } ], "5671/tcp": [ { "HostIp": "", "HostPort": "5671" } ], "8883/tcp": [ { "HostIp": "", "HostPort": "8883" } ] }, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 0, "ConsoleSize": [ 0, 0 ], "Isolation": "process", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": null, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "dir": "C:\\ProgramData\\iotedge-moby\\windowsfilter\\56f81486a2c0372bfe0494e0a6c1e17c6d0a76bb545e546ecf605583390cb1c0" }, "Name": "windowsfilter" }, "Mounts": [ { "Type": "bind", "Source": "c:\\programdata\\iotedge\\workload", "Destination": "c:\\programdata\\iotedge\\workload", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\programdata\\xxx", "Destination": "c:\\windows\\temp", "Mode": "", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "56f81486a2c0", "Domainname": "", "User": "ContainerUser", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "ExposedPorts": { "443/tcp": {}, "5671/tcp": {}, "8883/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "EdgeDeviceHostName=desktop-uldmue5", "IOTEDGE_APIVERSION=2020-07-07", "IOTEDGE_AUTHSCHEME=sasToken", "IOTEDGE_DEVICEID=di-7966", "IOTEDGE_IOTHUBHOSTNAME=xxxIoTHubDev.azure-devices.net", "IOTEDGE_MODULEGENERATIONID=637366330356525195", "IOTEDGE_MODULEID=$edgeHub", "IOTEDGE_WORKLOADURI=unix:///C:/ProgramData/iotedge/workload/sock", "RuntimeLogLevel=Information", "storageFolder=C:\\Windows\\Temp", "ASPNETCORE_URLS=http://+:80", "DOTNET_RUNNING_IN_CONTAINER=true" ], "Cmd": [ "dotnet", "Microsoft.Azure.Devices.Edge.Hub.Service.dll" ], "ArgsEscaped": true, "Image": "mcr.microsoft.com/azureiotedge-hub:1.0.10-rc2", "Volumes": null, "WorkingDir": "C:\\app", "Entrypoint": null, "OnBuild": null, "Labels": { "net.azure-devices.edge.owner": "Microsoft.Azure.Devices.Edge.Agent" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "56f81486a2c0372bfe0494e0a6c1e17c6d0a76bb545e546ecf605583390cb1c0", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": { "443/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "443" } ], "5671/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "5671" } ], "8883/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "8883" } ] }, "SandboxKey": "56f81486a2c0372bfe0494e0a6c1e17c6d0a76bb545e546ecf605583390cb1c0", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "nat": { "IPAMConfig": null, "Links": null, "Aliases": [ "desktop-uldmue5" ], "NetworkID": "78af2a2fd584edac0bbfeb464a61bbdec9be713c2817410fb8ddbef78e41b212", "EndpointID": "4fe985ae129de4fa0c94b464e45266954bb1ff861c67f8fae5bd26df8790ed16", "Gateway": "172.17.224.1", "IPAddress": "172.17.226.201", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "00:15:5d:f7:fc:ee", "DriverOpts": null } } } } ] ```
iotedge logs edgeAgent ``` PS C:\Windows\system32> iotedge logs edgeAgent 2020-09-25 12:19:48.810 +00:00 Edge Agent Main() <6> 2020-09-25 14:19:49.197 +02:00 [INF] - Initializing Edge Agent. <6> 2020-09-25 14:19:49.416 +02:00 [INF] - Version - 1.0.10-rc2.34217022 (029016e) <6> 2020-09-25 14:19:49.416 +02:00 [INF] - █████╗ ███████╗██╗ ██╗██████╗ ███████╗ ██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝ ███████║ ███╔╝ ██║ ██║██████╔╝█████╗ ██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝ ██║ ██║███████╗╚██████╔╝██║ ██║███████╗ ╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝

██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗
██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝
██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗
██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝
██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗
╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝

<6> 2020-09-25 14:19:49.456 +02:00 [INF] - Experimental features configuration: {"Enabled":true,"DisableCloudSubscriptions":false}
<6> 2020-09-25 14:19:49.642 +02:00 [INF] - Installing certificates [CN=Root CA Cert - p2-xxx-iot:4/3/2030 10:00:24 AM] to CertificateAuthority
<6> 2020-09-25 14:19:49.826 +02:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics
<6> 2020-09-25 14:19:49.846 +02:00 [INF] - Updating performance metrics every 05m:00s
<6> 2020-09-25 14:19:49.852 +02:00 [INF] - Started operation Get system resources
<6> 2020-09-25 14:19:49.853 +02:00 [INF] - Collecting metadata metrics
<6> 2020-09-25 14:19:49.901 +02:00 [INF] - Set metadata metrics: 1.0.10-rc2.34217022 (029016e), {"Enabled":true,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"windows","Architecture":"x86_64","Version":"1.0.10~rc2 (029016e)","ServerVersion":"19.03.12+azure","KernelVersion":"10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)","OperatingSystem":"Windows 10 Pro Version 1809 (OS Build 17763.107)","NumCpus":2}, True
<6> 2020-09-25 14:19:49.920 +02:00 [INF] - Started operation Checkpoint Availability
<6> 2020-09-25 14:19:49.937 +02:00 [INF] - Started operation refresh twin config
<6> 2020-09-25 14:19:49.960 +02:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2020-09-25 14:19:50.348 +02:00 [INF] - Created persistent store at C:\Windows\Temp\edgeAgent
<6> 2020-09-25 14:19:50.393 +02:00 [INF] - Started operation Metrics Scrape
<6> 2020-09-25 14:19:50.394 +02:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2020-09-25 14:19:50.699 +02:00 [INF] - Registering request handler UploadModuleLogs
<6> 2020-09-25 14:19:50.699 +02:00 [INF] - Registering request handler GetModuleLogs
<6> 2020-09-25 14:19:50.699 +02:00 [INF] - Registering request handler UploadSupportBundle
<6> 2020-09-25 14:19:50.699 +02:00 [INF] - Registering request handler RestartModule
<6> 2020-09-25 14:19:50.853 +02:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2020-09-25 14:19:50.999 +02:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2020-09-25 14:19:51.052 +02:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 2 and reported properties version 1.
<6> 2020-09-25 14:19:51.479 +02:00 [INF] - Plan execution started for deployment 2
<6> 2020-09-25 14:19:51.499 +02:00 [INF] - Executing command: "Command Group: (\n [Create module edgeHub]\n [Start module edgeHub]\n)"
<6> 2020-09-25 14:19:51.503 +02:00 [INF] - Executing command: "Create module edgeHub"
<6> 2020-09-25 14:20:04.462 +02:00 [INF] - Executing command: "Start module edgeHub"
<6> 2020-09-25 14:20:05.663 +02:00 [INF] - Plan execution ended for deployment 2
<6> 2020-09-25 14:20:05.990 +02:00 [INF] - Updated reported properties
<6> 2020-09-25 14:20:11.204 +02:00 [INF] - Updated reported properties

</details>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-reported iotedge needinfo Needs more info from issue reporter / updates from PR submitter no-issue-activity
Projects
None yet
Development

No branches or pull requests