bulk project deletion causes long controller response curve

[timothysc]

The response curve for bulk operations 1<n<100 is almost immediate, but once 100<n<1000the response curve becomes very slow, and cluster cleanup takes half hour on average. I'm not exactly certain which controller on the openshift side is responsible for project deletion on state change...

oc delete projects -l purpose=test

Eventual consistency is reached, but it's very slow.

/cc @abhgupta @jeremyeder @danmcp @derekwaynecarr

[liggitt]

how many projects is that operating on? what's the split between client time (how long does the oc command take to run) and server processing before projects transition from "Terminated" to actually being gone?

[derekwaynecarr]

Relevant upstream PR:
kubernetes/kubernetes#20076

It allows us to run multiple workers in Kubernetes namespace clean-up, and
it moves all the resource deletion (except for pods and services) to
deleteCollection calls.

On Tue, Feb 9, 2016 at 4:48 PM, Jordan Liggitt [email protected]
wrote:

how many projects is that operating on? what's the split between client
time (how long does the oc command take to run) and server processing
before projects transition from "Terminated" to actually being gone?

—
Reply to this email directly or view it on GitHub
#7166 (comment).

[derekwaynecarr]

We would need to make corresponding changes in the origin namespace controller to use delete collection calls, and adopt worker pattern.

[derekwaynecarr]

What was in the projects?

[timothysc]

how many projects is that operating on?

1000 takes > 1/2 hour.

what's the split between client time (how long does the oc command take to run) and server processing before projects transition from "Terminated" to actually being gone?

Total time for oc command is less then a minute, and transition to "Terminating" is pretty fast. Draining itself is very slow.

What was in the projects?

A several deploymentConfig objects that had a bunch of rcs etc as part of a vertical scaling test.

[derekwaynecarr]

So the upstream PR that merged today may improve the time, but we would also need to move the downstream controller to follow a similar pattern. I can look to get a PR together to adopt a queue for openshift, but I am not sure if I should move to use delete_collection calls in OpenShift now or not...

[derekwaynecarr]

On further thought, I would prefer we measure after the rebase that includes the earlier mentioned PR to understand if we need further improvements since ultimately, I would like to move to a single shared controller in both Origin and Kube that uses the Discovery API.

[pweil-]

@timothysc is this something you can remeasure for us now?

[timothysc]

Yes, we'll re-eval.

[liggitt]

rebase just pulled in deletecollection changes for upstream types

[timothysc]

We'll be deploying early next week and will report back.

[timothysc]

So I'm going to close down this issue now. There is still a long decay on termination for bulk deletions, but I believe there are other things now in the pipe to handle this for a -force case.