Should this all block ovn-kube master process? Would it make sense for it to be a separate container? It could just apply changes as necessary.

I think setting this timer while the master pods startup will give us bit more deterministic behavior. If user start cluster with high number of worker nodes, we can get into the death loop of raft partition.
Also i believe the general guideline is that we don't want to allow user to change this bahavior dynamically.

Right, what I'm saying is that this should be a separate container in the master pod, rather than gating the master container.

If we think of allowing user to change this value dynamically, i think having separate container makes sense to me. But if it's one time startup config, there is nothing much for this container to do, it will come up, set the value and die, and not sure CNO will be happy with that.

This should be added the operator configuration CRD, which is defined here: https://github.com/openshift/api/blob/master/operator/v1/types_network.go#L37

However, that is a long process, and won't be in in time for 4.5. The problem is that these fields will be instantly overwritten by the CVO.

Personally i don't like this PR as well. Mainly because the method that OVN provides us to set the election-timer is not very operation friendly. I already raised a RFE for OVN team to improve this and take election-timer as a parameter to ovn-ctl. That will make the entire logic pretty simple and we can write more stable logic to set the timer. So i am hoping that we will get rid of this logic, once OVN provides us a better way to set this timer.

That makes sense. I'm arguing that this value should either be hard-coded in the code, or available as a configuration knob.

I am more in the favor of having it as a configuration knob, because for raft based implementation, choosing approximately right election-timer depends on the workload.

I agree with @squeed that it should be hard-coded for now, until we have a better idea whether it acutally does need to be changed. It should not be user-visible configuration at any rate, especially not in the API.

@dcbw In general we want to keep these numbers as minimal as possible. Currently these numbers are set for 200 nodes (for 50 pods per node workload), so the moment number of nodes increases, you will have to increase it, there is no question about it. We can hardcode it for 500 nodes as of now (60 seconds-- frankly it's very high), and live with it until and unless something else comes up that require it to change it.
For election-timer i don't have strong opinion against hardcoding, because you can change this value anyways directly on the pod if needed.
So for hardcoding what approach do we need to take ? using operator CRD? is it okay to make operator CRD to be aware of SB DB deployment model (raft or ha)?