docker service logs command not responding

[anushkaee]

Expected behavior

We have created swarm cluster in azure using following template
https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fdownload.docker.com%2Fazure%2Fstable%2FDocker.tmpl

docker service logs -f should show service logs

Actual behavior

after scale up and scale down several times, docker service logs command stopped responding

Steps to reproduce the behavior

1. Create a service serv1 with replicas across multiple nodes
2. Run docker service logs -f serv1
3. Initially observe logs from multiple containers across different nodes
4. scale up and scale down several times
5. Run docker service logs -f serv1
6. command not responding

Information

docker-diagnose output

swarm-manager000003:~$ docker-diagnose
OK hostname=swarm-manager000001 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-manager000002 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-manager000003 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-worker000000 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-worker000001 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-worker000002 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-worker000003 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
OK hostname=swarm-worker000004 session=1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
Done requesting diagnostics.
Your diagnostics session ID is 1510318044-c5urt3zgyY9ulkooLzIoM8Vjv28fKqZg
Please provide this session ID to the maintainer debugging your issue.

docker version output

> ```
> swarm-manager000003:~$ docker version
> Client:
>  Version:      17.09.0-ce
>  API version:  1.32
>  Go version:   go1.8.3
>  Git commit:   afdb6d4
>  Built:        Tue Sep 26 22:39:28 2017
>  OS/Arch:      linux/amd64
> 
> Server:
>  Version:      17.09.0-ce
>  API version:  1.32 (minimum version 1.12)
>  Go version:   go1.8.3
>  Git commit:   afdb6d4
>  Built:        Tue Sep 26 22:45:38 2017
>  OS/Arch:      linux/amd64
>  Experimental: false
> ```

docker info output

Containers: 8
 Running: 6
 Paused: 0
 Stopped: 2
Images: 8
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: zbbpsttjfubkuumf9p0e214d0
 Is Manager: true
 ClusterID: wyn1lmhtgecbnb2r2rwhzjm5s
 Managers: 3
 Nodes: 8
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.0.0.9
 Manager Addresses:
  10.0.0.10:2377
  10.0.0.11:2377
  10.0.0.9:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.785GiB
Name: swarm-manager000003
ID: JRZS:L436:UFYH:KTKG:7T4K:4HP5:TGFI:TOZC:4CSS:HQLW:KNEK:GI4K
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 90
 Goroutines: 152
 System Time: 2017-11-10T13:10:13.190976179Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[FrenchBen]

To make sure that we're on the same page, container logs weren't going to the storage account. They were kept on disk?

[anushkaee]

yes it is on the disk

[FrenchBen]

/cc @nishanttotla

[nishanttotla]

I think @dperny is the right person to ask if this is similar to any known logs issue. If not, this might have to do purely with the Azure integration.

[FrenchBen]

@anushkaee it looks like I'm missing the diagnostics from the swarm leader. Would you mind SSH'ing into the leader (swarm-manager000001) and running the docker-diagnose again from there?