Global mode does not work correctly #20

djmaze · 2019-10-26T20:50:07Z

Behaviour

Steps to reproduce this issue

Create a 5-node swarm cluster with 3 manager nodes in play with docker
Create a swarm-cronjob stack as shown in the example
Create a stack as shown in the example for a global job
Tail the service logs for swarm-cronjob and the job service's stack

Expected behaviour

On every schedule occasion, there should be one log entry for each node.

Actual behaviour

After the first deployment output (where there is output for every node), only the output of two nodes is shown. docker stack ps shows that the service is only restarted on two of the nodes.

Most of the time, that is. Every few iterations, the service is successfully run on all nodes again.

I am able to reproduce this on PWD as well as a real swarm cluster with 3 managers and 1 worker (w/ docker 18.09.8).

Configuration

Target Docker version (the host/cluster you manage) : 19.03.4 (happens on 18.09.8 as well)
Platform (windows/linux) : Linux amd64
Target Swarm version : ?

Docker info

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)

Server:
 Containers: 6
  Running: 1
  Paused: 0
  Stopped: 5
 Images: 2
 Server Version: 19.03.4
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 3gq13ao9gjqm4e93fbt2a0g5q
  Is Manager: true
  ClusterID: ocg8epc8ygsna7jyckvnoeqcs
  Managers: 3
  Nodes: 5
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.0.51
  Manager Addresses:
   192.168.0.49:2377
   192.168.0.50:2377
   192.168.0.51:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-161-generic
 Operating System: Alpine Linux v3.10 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.4GiB
 Name: manager2
 ID: Y4T2:VAE3:BTYF:6WVY:MPZH:PPO7:U7SL:AJI4:YTHG:YS2P:B3XP:Q45A
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 62
  Goroutines: 172
  System Time: 2019-10-26T20:40:38.154648103Z
  EventsListeners: 2
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.1
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Logs

swarm-cronjob logs:

2019-10-26T20:42:53.338828516Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:53 CEST INF Starting swarm-cronjob 1.6.0
2019-10-26T20:42:53.338959117Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:53 CEST DBG Creating Docker API client
2019-10-26T20:42:53.379697258Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:53 CEST DBG 0 scheduled services found through labels
2019-10-26T20:42:53.379733758Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:53 CEST DBG Starting the cron scheduler
2019-10-26T20:42:53.379741258Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:53 CEST DBG Listening docker events...
2019-10-26T20:42:57.319945256Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:57 CEST DBG Event triggered newstate= oldstate= service=global-job_test
2019-10-26T20:42:57.335406447Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:57 CEST INF Add cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:42:57.335447247Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:57 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:42:57.335475647Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:57 CEST DBG Event triggered newstate= oldstate= service=global-job_test
2019-10-26T20:42:57.362249406Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:57 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:42:57.362274906Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:42:57 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:43:00.019533319Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Service task node=worker2 service=global-job_test status_message=starting status_state=starting task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:43:00.019564819Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Service task node=manager2 service=global-job_test status_message=starting status_state=starting task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:43:00.019572119Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Service task node=manager3 service=global-job_test status_message=starting status_state=starting task_id=o6xlg4g3t1k8nh5jfgl4x5usu
2019-10-26T20:43:00.019578119Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Service task node=manager1 service=global-job_test status_message=starting status_state=starting task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:43:00.019599419Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Service task node=worker1 service=global-job_test status_message=starting status_state=starting task_id=p8roxdmjrfvlkolsy9vnfogr0
2019-10-26T20:43:00.019605119Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST INF Start job service=global-job_test status= tasks_active=0
2019-10-26T20:43:00.027649867Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Event triggered newstate= oldstate= service=global-job_test
2019-10-26T20:43:00.696545622Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:43:00.696584223Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:43:00.696590923Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Event triggered newstate=updating oldstate= service=global-job_test
2019-10-26T20:43:00.703906666Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:43:00.704290168Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:43:14.607366999Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:14 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:43:14.622608289Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:14 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:43:14.622631989Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:14 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:43:30.027846938Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=rxu731gfkxywzztejrzqt8yz2
2019-10-26T20:43:30.027888638Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:43:30.027899038Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:43:30.027905838Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=y6jhnhv6ztacb6hvuhgmgpf03
2019-10-26T20:43:30.027912538Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:43:30.028043239Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:43:30.028067639Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:43:30.028075339Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=worker1 service=global-job_test status_message=shutdown status_state=shutdown task_id=p8roxdmjrfvlkolsy9vnfogr0
2019-10-26T20:43:30.028084040Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=o6xlg4g3t1k8nh5jfgl4x5usu
2019-10-26T20:43:30.028090540Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:43:30.038057399Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:43:30.044022234Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:43:30.079028441Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:43:30.079083141Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:43:30.079092241Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:43:30.096803246Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:43:30.096932147Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:43:32.697817939Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:32 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:43:32.707604197Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:32 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:43:32.707882299Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:43:32 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:44:00.027384935Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=6ctbo74wx1evgkl3891e46m8g
2019-10-26T20:44:00.027586137Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=yw0p0uvvwkzpjbsgx2c5t3qa5
2019-10-26T20:44:00.027606037Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=rxu731gfkxywzztejrzqt8yz2
2019-10-26T20:44:00.027613137Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:44:00.027619537Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:44:00.027625837Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=y6jhnhv6ztacb6hvuhgmgpf03
2019-10-26T20:44:00.027632237Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:44:00.027638637Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:44:00.027647737Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:44:00.027654137Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=worker1 service=global-job_test status_message=shutdown status_state=shutdown task_id=p8roxdmjrfvlkolsy9vnfogr0
2019-10-26T20:44:00.027660537Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=o6xlg4g3t1k8nh5jfgl4x5usu
2019-10-26T20:44:00.027682737Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:44:00.027695837Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:44:00.041459819Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:44:00.063773651Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:44:00.063823751Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:44:00.063830951Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:44:00.077671733Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:44:00.077749134Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:44:03.080828819Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:03 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:44:03.114022615Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:03 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:44:03.114073416Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:03 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:44:30.030278875Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=t1vlce7ha8roh8j4zg3cwyhy9
2019-10-26T20:44:30.030308275Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=x5o0z8i9xj8p3o3hz7nt9cjcc
2019-10-26T20:44:30.030315075Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=6ctbo74wx1evgkl3891e46m8g
2019-10-26T20:44:30.030320775Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=yw0p0uvvwkzpjbsgx2c5t3qa5
2019-10-26T20:44:30.030326175Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=rxu731gfkxywzztejrzqt8yz2
2019-10-26T20:44:30.030347175Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:44:30.030353075Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:44:30.030358175Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=y6jhnhv6ztacb6hvuhgmgpf03
2019-10-26T20:44:30.030364375Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:44:30.030369575Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:44:30.030374675Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:44:30.031743484Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker1 service=global-job_test status_message=shutdown status_state=shutdown task_id=p8roxdmjrfvlkolsy9vnfogr0
2019-10-26T20:44:30.031758784Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=o6xlg4g3t1k8nh5jfgl4x5usu
2019-10-26T20:44:30.031765884Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:44:30.031772084Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:44:30.045133463Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:44:30.089437725Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:44:30.097190371Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:44:30.097214972Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:44:30.160094244Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:44:30.160147244Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:44:32.684168202Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:32 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:44:32.694651664Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:32 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:44:32.694931866Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:44:32 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:45:00.031574920Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=33hdn2wdg37ys24yuj75udp1o
2019-10-26T20:45:00.031603820Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=7yta3pmm2iw8x5s9t44300xl3
2019-10-26T20:45:00.031612021Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=t1vlce7ha8roh8j4zg3cwyhy9
2019-10-26T20:45:00.031617721Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=x5o0z8i9xj8p3o3hz7nt9cjcc
2019-10-26T20:45:00.031623421Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=6ctbo74wx1evgkl3891e46m8g
2019-10-26T20:45:00.031628921Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=yw0p0uvvwkzpjbsgx2c5t3qa5
2019-10-26T20:45:00.031634821Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=rxu731gfkxywzztejrzqt8yz2
2019-10-26T20:45:00.031654321Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:45:00.031662921Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:45:00.031668921Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=y6jhnhv6ztacb6hvuhgmgpf03
2019-10-26T20:45:00.031674421Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:45:00.031679921Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:45:00.031685721Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:45:00.031691121Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker1 service=global-job_test status_message=shutdown status_state=shutdown task_id=p8roxdmjrfvlkolsy9vnfogr0
2019-10-26T20:45:00.031696521Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=o6xlg4g3t1k8nh5jfgl4x5usu
2019-10-26T20:45:00.031761521Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:45:00.031779422Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:45:00.038967364Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:45:00.057264973Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:45:00.057375673Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:45:00.060752593Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:45:00.078486898Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:45:00.078798700Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:45:03.399136490Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:03 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:45:03.425201744Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:03 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:45:03.425610447Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:03 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:45:30.030523261Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=ih0qu55x4ksl8caqkkuk7g02d
2019-10-26T20:45:30.030553661Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=r4km6owlvsdzq2yrgycs1c5h0
2019-10-26T20:45:30.030561561Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=33hdn2wdg37ys24yuj75udp1o
2019-10-26T20:45:30.030567961Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=7yta3pmm2iw8x5s9t44300xl3
2019-10-26T20:45:30.030574161Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=t1vlce7ha8roh8j4zg3cwyhy9
2019-10-26T20:45:30.030579961Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=x5o0z8i9xj8p3o3hz7nt9cjcc
2019-10-26T20:45:30.030585961Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=6ctbo74wx1evgkl3891e46m8g
2019-10-26T20:45:30.030608161Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=yw0p0uvvwkzpjbsgx2c5t3qa5
2019-10-26T20:45:30.030616761Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=rxu731gfkxywzztejrzqt8yz2
2019-10-26T20:45:30.030622461Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:45:30.030628061Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:45:30.030633661Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=y6jhnhv6ztacb6hvuhgmgpf03
2019-10-26T20:45:30.030639261Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:45:30.030644861Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:45:30.030650561Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:45:30.030656261Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=o6xlg4g3t1k8nh5jfgl4x5usu
2019-10-26T20:45:30.030661962Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:45:30.033594879Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:45:30.039685415Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:45:30.061822946Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:45:30.062170848Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:45:30.068468986Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:45:30.080570458Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:45:30.080592858Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:45:33.294997630Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:33 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:45:33.337374382Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:33 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:45:33.337403182Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:45:33 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:46:00.026643387Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=iqmkwqs3tghm0zf2vrgq44axw
2019-10-26T20:46:00.026670187Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=p4gliavd25dfa1nxwbtdewois
2019-10-26T20:46:00.026676887Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=ih0qu55x4ksl8caqkkuk7g02d
2019-10-26T20:46:00.026682587Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=r4km6owlvsdzq2yrgycs1c5h0
2019-10-26T20:46:00.026687987Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=33hdn2wdg37ys24yuj75udp1o
2019-10-26T20:46:00.026693287Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=7yta3pmm2iw8x5s9t44300xl3
2019-10-26T20:46:00.026820488Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=t1vlce7ha8roh8j4zg3cwyhy9
2019-10-26T20:46:00.026826888Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=x5o0z8i9xj8p3o3hz7nt9cjcc
2019-10-26T20:46:00.026833888Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=6ctbo74wx1evgkl3891e46m8g
2019-10-26T20:46:00.026839188Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=yw0p0uvvwkzpjbsgx2c5t3qa5
2019-10-26T20:46:00.026844188Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=rxu731gfkxywzztejrzqt8yz2
2019-10-26T20:46:00.026849388Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:46:00.026854488Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:46:00.026959589Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=y6jhnhv6ztacb6hvuhgmgpf03
2019-10-26T20:46:00.026967889Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:46:00.026973089Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:46:00.026978089Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:46:00.026991889Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:46:00.026997289Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:46:00.038284156Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:46:00.049991326Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:46:00.050014826Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:46:00.050020426Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:46:00.102332336Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:46:00.105970558Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:00 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:46:03.288794054Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:03 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:46:03.314152204Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:03 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:46:03.314178204Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:03 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:46:30.042047224Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=7u1map386f0eblw9skzoep7vo
2019-10-26T20:46:30.042071725Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=okhp1zbav1xz5oe7uea0vlqk8
2019-10-26T20:46:30.042078825Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=iqmkwqs3tghm0zf2vrgq44axw
2019-10-26T20:46:30.042084725Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=p4gliavd25dfa1nxwbtdewois
2019-10-26T20:46:30.042104225Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=ih0qu55x4ksl8caqkkuk7g02d
2019-10-26T20:46:30.042110425Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=r4km6owlvsdzq2yrgycs1c5h0
2019-10-26T20:46:30.042115925Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=33hdn2wdg37ys24yuj75udp1o
2019-10-26T20:46:30.042121525Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=7yta3pmm2iw8x5s9t44300xl3
2019-10-26T20:46:30.042128425Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=t1vlce7ha8roh8j4zg3cwyhy9
2019-10-26T20:46:30.042134225Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager3 service=global-job_test status_message=finished status_state=complete task_id=x5o0z8i9xj8p3o3hz7nt9cjcc
2019-10-26T20:46:30.042139625Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=6ctbo74wx1evgkl3891e46m8g
2019-10-26T20:46:30.042145025Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker1 service=global-job_test status_message=finished status_state=complete task_id=yw0p0uvvwkzpjbsgx2c5t3qa5
2019-10-26T20:46:30.042150225Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=mti2ir0t6hdba32cbanty7spo
2019-10-26T20:46:30.042155625Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=mx5qs3kd6wwkz2isd3xqeqwcz
2019-10-26T20:46:30.042160925Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=qxy9lnetgqfiqztj0uwjvssoi
2019-10-26T20:46:30.042171025Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager1 service=global-job_test status_message=finished status_state=complete task_id=jfm1rym73lqtu2j374ohtxj8u
2019-10-26T20:46:30.042176725Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=manager2 service=global-job_test status_message=finished status_state=complete task_id=jp0w343tiyb5joqthv67zp36u
2019-10-26T20:46:30.042182525Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Service task node=worker2 service=global-job_test status_message=finished status_state=complete task_id=nzrhwgwcj1v3fxhoymq6gb69z
2019-10-26T20:46:30.042188025Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST INF Start job service=global-job_test status=paused tasks_active=0
2019-10-26T20:46:30.046720952Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:46:30.075022120Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:46:30.075351622Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:46:30.075468123Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Event triggered newstate=updating oldstate=updating service=global-job_test
2019-10-26T20:46:30.091783520Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:46:30.092178822Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:30 CEST DBG Number of cronjob tasks: 1
2019-10-26T20:46:32.764649596Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:32 CEST DBG Event triggered newstate=paused oldstate=updating service=global-job_test
2019-10-26T20:46:32.776077264Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:32 CEST DBG Update cronjob with schedule 0/30 * * * * * service=global-job_test
2019-10-26T20:46:32.776272465Z swarm_cronjob_app.1.1d1bw0uwm17q@manager3    | Sat, 26 Oct 2019 22:46:32 CEST DBG Number of cronjob tasks: 1

logs from scheduled service:

2019-10-26T20:43:02.877993923Z global-job_test.0.nzrhwgwcj1v3@worker2    | NodeID: "v5cm1cjd4iy05zz2am73hrf0z"
2019-10-26T20:43:08.565210959Z global-job_test.0.jfm1rym73lqt@manager1    | NodeID: "ecs6mnxe31s0t08nkvei092so"
2019-10-26T20:43:08.569653485Z global-job_test.0.jp0w343tiyb5@manager2    | NodeID: "3gq13ao9gjqm4e93fbt2a0g5q"
2019-10-26T20:43:09.428232463Z global-job_test.0.qxy9lnetgqfi@worker2    | NodeID: "v5cm1cjd4iy05zz2am73hrf0z"
2019-10-26T20:43:13.127515745Z global-job_test.0.mx5qs3kd6wwk@manager1    | NodeID: "ecs6mnxe31s0t08nkvei092so"
2019-10-26T20:43:13.535353457Z global-job_test.0.mti2ir0t6hdb@manager2    | NodeID: "3gq13ao9gjqm4e93fbt2a0g5q"
2019-10-26T20:43:31.658171186Z global-job_test.0.yw0p0uvvwkzp@worker1    | NodeID: "5azgjswiu09jxa9dhhqickrtj"
2019-10-26T20:43:34.335444231Z global-job_test.0.6ctbo74wx1ev@worker2    | NodeID: "v5cm1cjd4iy05zz2am73hrf0z"
2019-10-26T20:44:02.045843889Z global-job_test.0.x5o0z8i9xj8p@manager3    | NodeID: "pmdttoyn0fco2f0t9d7vegupc"
2019-10-26T20:44:04.309006493Z global-job_test.0.t1vlce7ha8ro@worker1    | NodeID: "5azgjswiu09jxa9dhhqickrtj"
2019-10-26T20:44:31.758374216Z global-job_test.0.7yta3pmm2iw8@manager3    | NodeID: "pmdttoyn0fco2f0t9d7vegupc"
2019-10-26T20:44:33.590233772Z global-job_test.0.33hdn2wdg37y@worker1    | NodeID: "5azgjswiu09jxa9dhhqickrtj"
2019-10-26T20:45:02.283250073Z global-job_test.0.r4km6owlvsdz@manager3    | NodeID: "pmdttoyn0fco2f0t9d7vegupc"
2019-10-26T20:45:05.406355593Z global-job_test.0.ih0qu55x4ksl@worker1    | NodeID: "5azgjswiu09jxa9dhhqickrtj"
2019-10-26T20:45:31.978585819Z global-job_test.0.p4gliavd25df@manager2    | NodeID: "3gq13ao9gjqm4e93fbt2a0g5q"
2019-10-26T20:45:34.746719645Z global-job_test.0.iqmkwqs3tghm@manager3    | NodeID: "pmdttoyn0fco2f0t9d7vegupc"
2019-10-26T20:46:02.227012350Z global-job_test.0.okhp1zbav1xz@manager3    | NodeID: "pmdttoyn0fco2f0t9d7vegupc"
2019-10-26T20:46:04.194193329Z global-job_test.0.7u1map386f0e@worker1    | NodeID: "5azgjswiu09jxa9dhhqickrtj"

The text was updated successfully, but these errors were encountered:

danielgrabowski · 2019-11-17T16:59:39Z

I have the same problem. If I start a global service on 9 nodes running every 1m I can see that every minute it starts on a different set of nodes (usually 2 or 3 out of 9 more less randomly).

crazy-max · 2019-11-19T05:41:58Z

Hi guys, I did some tests on my side and it appears that this could be due to the way the tasks on the Swarm side are managed. Maybe linked to a GC issue with task reaper. Any ideas @dperny?

djmaze · 2020-01-31T21:57:29Z

So, in spite of job support finally getting into Docker (yeah!), this project will probably still be useful as long as there is no cron scheduler in Swarm. @crazy-max Do you plan on migrating swarm-cronjob to the new model?

That should then probably fix this issue as well.

crazy-max · 2020-02-10T22:54:13Z

@djmaze Yes I'm aware of this feature. That looks promising and I think it will benefit for the global mode support (and fix this issue).

danielgrabowski · 2020-06-10T06:43:58Z

Any news on this issue?

djmaze · 2020-11-14T20:38:32Z

Is anyone aware if Docker CE 20.10 will be released in the near future? That will contain the new job support which blocks this issue.

crazy-max · 2020-12-19T20:44:14Z

@danielgrabowski I'm going to start working on it but I'll need to do a large refactoring beforehand. Keep you in touch.

mostafa-asg · 2021-05-23T19:00:05Z

I have the same issue.
Cluster: 1 manager node with 2 worker nodes.
Docker version: 20.10.6
I ran busybox date example in global mode.

date_test.0.p88rnuivp25o@swarm_test1    | Sun May 23 18:48:44 UTC 2021
date_test.0.66loc48rvrgd@swarm_test3    | Sun May 23 18:50:00 UTC 2021
date_test.0.d3j43d50ls90@swarm_test2    | Sun May 23 18:51:00 UTC 2021
date_test.0.49dmp9uvt1mt@swarm_test3    | Sun May 23 18:52:00 UTC 2021
date_test.0.bxnixn2wpetx@swarm_test3    | Sun May 23 18:48:47 UTC 2021
date_test.0.xvocor1a123r@swarm_test3    | Sun May 23 18:51:01 UTC 2021
date_test.0.flw796nt9mmy@swarm_test2    | Sun May 23 18:48:32 UTC 2021
date_test.0.km0tkw07iqww@swarm_test2    | Sun May 23 18:49:00 UTC 2021

swarm_test1 only appeared once.

Luis97lol · 2023-04-12T11:16:52Z

Any news on this issue? Mode global only runs on the node that swarm-cronjob is deployed. Other nodes never ran

Luis97lol · 2023-04-12T13:41:38Z

docker swarm ca --rotate

This command apparently solves the problem

moschlar · 2023-12-05T09:42:08Z

@crazy-max @djmaze It seems that compose file syntax support for jobs only recently landed in Docker 23 (moby/moby#41895 (comment))

docker/cli#2907

djmaze mentioned this issue Oct 26, 2019

Support for Global Mode #7

Closed

crazy-max added the ⁉️ maybe bug label Oct 30, 2019

crazy-max added the kind/upstream label Dec 20, 2019

djmaze mentioned this issue Apr 11, 2020

Using a cron like settings containrrr/shepherd#16

Closed

djmaze mentioned this issue May 27, 2020

Allow prune to be on a different schedule djmaze/resticker#22

Closed

djmaze mentioned this issue Feb 24, 2021

add RANDOM_DELAY to spread backup start in swarm djmaze/resticker#79

Closed

crazy-max added 📌 pinned status/needs-investigation and removed ⁉️ maybe bug labels Mar 7, 2021

crazy-max removed the 📌 pinned label May 26, 2021

moschlar mentioned this issue Dec 5, 2023

Auto Image delete containrrr/shepherd#122

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Global mode does not work correctly #20

Global mode does not work correctly #20

djmaze commented Oct 26, 2019 •

edited

Loading

danielgrabowski commented Nov 17, 2019

crazy-max commented Nov 19, 2019 •

edited

Loading

djmaze commented Jan 31, 2020

crazy-max commented Feb 10, 2020

danielgrabowski commented Jun 10, 2020

djmaze commented Nov 14, 2020

crazy-max commented Dec 19, 2020

mostafa-asg commented May 23, 2021

Luis97lol commented Apr 12, 2023 •

edited

Loading

Luis97lol commented Apr 12, 2023

moschlar commented Dec 5, 2023

Global mode does not work correctly #20

Global mode does not work correctly #20

Comments

djmaze commented Oct 26, 2019 • edited Loading

Behaviour

Steps to reproduce this issue

Expected behaviour

Actual behaviour

Configuration

Docker info

Logs

danielgrabowski commented Nov 17, 2019

crazy-max commented Nov 19, 2019 • edited Loading

djmaze commented Jan 31, 2020

crazy-max commented Feb 10, 2020

danielgrabowski commented Jun 10, 2020

djmaze commented Nov 14, 2020

crazy-max commented Dec 19, 2020

mostafa-asg commented May 23, 2021

Luis97lol commented Apr 12, 2023 • edited Loading

Luis97lol commented Apr 12, 2023

moschlar commented Dec 5, 2023

djmaze commented Oct 26, 2019 •

edited

Loading

crazy-max commented Nov 19, 2019 •

edited

Loading

Luis97lol commented Apr 12, 2023 •

edited

Loading