add Restart=on-failure for inner docker systemd service

[medyagh]

closes #9691

Fixes the case when docker dies and we never restart it. and then when users run minkube docker-env minikube sees docker sotpped and tries to restart it and that had caused Api server container to be restarting as well.

it was partially improved by previous PR that changed restart to reload) but still didnt fix actual problem that docker was dying and nobody was birning it back to life

this PR will make Docker service to start after it is failed automatically by Systemd without sending it to an endless loop

in addition to restart always I added StartLimitIntervalSec=interval and , StartLimitBurst=burst to the systemd file to avoid infinite restart loops

systemd docs https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=

StartLimitIntervalSec=interval, StartLimitBurst=burst
Configure unit start rate limiting. Units which are started more than burst times within an interval time interval are not permitted to start any more. Use StartLimitIntervalSec= to configure the checking interval (defaults to DefaultStartLimitIntervalSec= in manager configuration file, set it to 0 to disable any kind of rate limiting). Use StartLimitBurst= to configure how many starts per interval are allowed (defaults to DefaultStartLimitBurst= in manager configuration file). These configuration options are particularly useful in conjunction with the service setting Restart= (see systemd.service(5)); however, they apply to all kinds of starts (including manual), not just those triggered by the Restart= logic. Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, after the interval has passed. From this point on, the restart logic is activated again. Note that systemctl reset-failed will cause the restart rate counter for a service to be flushed, which is useful if the administrator wants to manually start a unit and the start limit interferes with that. Note that this rate-limiting is enforced after any unit condition checks are executed, and hence unit activations with failing conditions do not count towards this rate limit. This setting does not apply to slice, target, device, and scope units, since they are unit types whose activation may either never fail, or may succeed only a single time.

When a unit is unloaded due to the garbage collection logic (see above) its rate limit counters are flushed out too. This means that configuring start rate limiting for a unit that is not referenced continuously has no effect.

link to Docker's recommended setting

https://github.com/moby/moby/blob/e1b15e1e5bf3a512ac7298ed29fcd4126c45c1ae/contrib/init/systemd/docker.service#L29-L31

[medyagh]

/ok-to-test

[minikube-pr-bot]

kvm2 Driver
Times for minikube: 57.9s 56.6s 58.6s
Average time for minikube: 57.7s

Times for Minikube (PR 9775): 59.6s 59.1s 54.2s
Average time for Minikube (PR 9775): 57.6s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9775) |
+--------------------------------+----------+--------------------+
| * minikube v1.15.1 on Debian   | 0.1s     | 0.0s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 34.3s    | 33.8s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.4 | 21.5s    | 21.3s              |
| on Docker 19.03.13 ...         |          |                    |
| * Verifying Kubernetes         | 1.5s     | 1.6s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.2s     | 0.9s               |
| storage-provisioner,           |          |                    |
| default-storageclass           |          |                    |
|                                | 0.0s     | 0.8s               |
|   - Want kubectl v1.19.4? Try  |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| cluster and "default"          |          |                    |
| namespace by default           |          |                    |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 28.9s 29.2s 29.0s
Average time for minikube: 29.0s

Times for Minikube (PR 9775): 28.6s 28.5s 28.5s
Average time for Minikube (PR 9775): 28.5s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9775) |
+--------------------------------+----------+--------------------+
| * minikube v1.15.1 on Debian   | 0.2s     | 0.2s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.1s     | 0.1s               |
| based on user configuration    |          |                    |
| * Starting control plane node  | 0.1s     | 0.1s               |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    | 8.9s     | 8.9s               |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.4 | 18.5s    | 17.9s              |
| on Docker 19.03.13 ...         |          |                    |
| * Verifying Kubernetes         | 1.2s     | 1.2s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.1s     | 0.1s               |
| storage-provisioner,           |          |                    |
| default-storageclass           |          |                    |
|                                | 0.0s     | 0.0s               |
|   - Want kubectl v1.19.4? Try  |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| cluster and "default"          |          |                    |
| namespace by default           |          |                    |
+--------------------------------+----------+--------------------+

[minikube-pr-bot]

kvm2 Driver
Times for minikube: 57.1s 55.3s 61.1s
Average time for minikube: 57.9s

Times for Minikube (PR 9775): 56.1s 60.9s 60.1s
Average time for Minikube (PR 9775): 59.0s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9775) |
+--------------------------------+----------+--------------------+
| * minikube v1.15.1 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 33.3s    | 34.2s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.4 | 22.6s    | 22.8s              |
| on Docker 19.03.13 ...         |          |                    |
| * Verifying Kubernetes         | 1.5s     | 1.5s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.3s     | 0.4s               |
| storage-provisioner,           |          |                    |
| default-storageclass           |          |                    |
|                                | 0.0s     | 0.5s               |
|   - Want kubectl v1.19.4? Try  |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| cluster and "default"          |          |                    |
| namespace by default           |          |                    |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 28.8s 28.9s 29.3s
Average time for minikube: 29.0s

Times for Minikube (PR 9775): 28.3s 29.8s 28.5s
Average time for Minikube (PR 9775): 28.9s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9775) |
+--------------------------------+----------+--------------------+
| * minikube v1.15.1 on Debian   | 0.2s     | 0.2s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.1s     | 0.1s               |
| based on user configuration    |          |                    |
| * Starting control plane node  | 0.1s     | 0.1s               |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    | 9.2s     | 9.2s               |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.4 | 18.2s    | 18.0s              |
| on Docker 19.03.13 ...         |          |                    |
| * Verifying Kubernetes         | 1.2s     | 1.2s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.1s     | 0.1s               |
| storage-provisioner,           |          |                    |
| default-storageclass           |          |                    |
|                                | 0.2s     | 0.0s               |
|   - Want kubectl v1.19.4? Try  |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| cluster and "default"          |          |                    |
| namespace by default           |          |                    |
+--------------------------------+----------+--------------------+

[medyagh]

/retest-this-please

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [medyagh,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment