Skip to content

Commit

Permalink
(Addons) Create a new command in every retry
Browse files Browse the repository at this point in the history
Whenever a `kubectl apply` fails while enabling an addon, it is
retried with exponential backoff. The command (type `*exec.Cmd`)
that this retry function runs in created outside the function -
which means that it is reused on every retry. This is a problem
because `exec.Cmd` (https://godoc.org/github.com/pkg/exec#Cmd)
states that "... Cmd cannot be reused after calling its Run or
Start methods."

This retry is a common case due to, say, a CRD and its resource
being present in the same YAML file of the addon which causes
a race condition where the resource is created before its CRD is
created in the cluster - this race is fixed by subsequent retries.
I've noticed this in the dashboard and the ambassador addon.

Due to the above mentioned bug, minikube throws errors like
`exec: already started` in every retry and the retry is never
successful, manifests are never deployed and addon creation errors
out.

Fix kubernetes#8138
Fix kubernetes#8119
Fix a few CI errors in kubernetes#8372
  • Loading branch information
concaf committed Jul 21, 2020
1 parent aaebd88 commit 3181bc0
Showing 1 changed file with 1 addition and 3 deletions.
4 changes: 1 addition & 3 deletions pkg/addons/addons.go
Original file line number Diff line number Diff line change
Expand Up @@ -250,11 +250,9 @@ func enableOrDisableAddonInternal(cc *config.ClusterConfig, addon *assets.Addon,
}
}

command := kubectlCommand(cc, deployFiles, enable)

// Retry, because sometimes we race against an apiserver restart
apply := func() error {
_, err := cmd.RunCmd(command)
_, err := cmd.RunCmd(kubectlCommand(cc, deployFiles, enable))
if err != nil {
glog.Warningf("apply failed, will retry: %v", err)
}
Expand Down

0 comments on commit 3181bc0

Please sign in to comment.