README for new MT Scheduler with pluggable policies

[aavarghese]

Signed-off-by: aavarghese avarghese@us.ibm.com

Continuation of #768

Proposed Changes

• README

Release Note

Docs

[aavarghese]

/cc @lionelvillard

[codecov]

Codecov Report

Merging #888 (ea56516) into main (3f66360) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #888   +/-   ##
=======================================
  Coverage   75.01%   75.01%           
=======================================
  Files         152      152           
  Lines        7080     7080           
=======================================
  Hits         5311     5311           
  Misses       1485     1485           
  Partials      284      284           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f66360...ea56516. Read the comment docs.

[lionelvillard]

@aavarghese can you fix the linter errors? thx!

[pierDipi]

Thanks, I love this document!

[pierDipi]

This is not really true, a consumer group rebalance could degrade message processing especially when Kafka Consumer Incremental Rebalance Protocol is not being used (which afaik is not implemented in Sarama).

[aavarghese]

@pierDipi the pod set being referred to here is only talking about the eventing scheduler adapter pods where vreplicas are placed. Since pod will restart, the same placements can be kept without a rebalancing of the vreps.
I agree with you about the consumer group rebalancing and degradation but that may/may not happen here if the kafka pods are affected, as well.
I hope I'm not missing anything...

[pierDipi]

@pierDipi the pod set being referred to here is only talking about the eventing scheduler adapter pods where vreplicas are placed. Since pod will restart, the same placements can be kept without a rebalancing of the vreps.

This is what All existing vreplica placements will still be valid and no rebalancing is needed. is saying, I agree and it's clear to me why.

I was referring to There shouldn’t be any degradation in Kafka message processing..

I agree with you about the consumer group rebalancing and degradation but that may/may not happen here if the kafka pods are affected, as well.
I hope I'm not missing anything...

so, are you saying that if a pod where vreplicas are placed goes down that won't trigger a consumer group rebalance that affects message processing?

In the worst-case scenario, I'd expect something like this to happen (happy to be wrong):

1. Pod goes down
2. A new pod comes up (same name)
3. Kafka broker sees a new consumer that wants to join the group -> rebalance
4. Kafka detects that the consumer that was consuming messages in the dead pod (1) is not sending heartbeats anymore -> rebalance (again)

at least one rebalance happen. 2 in the worst case since terminationGracePeriodSeconds = 0 < "time for Kafka to detect that a consumer is dead"

Is the above not possible? If yes, does that count as a degradation for Kafka message processing?

[aavarghese]

You're right. This is absolutely possible.

I made an assumption that the same sticky pod (when restarted) would have the same consumer member ID and using static membership, would get the same assignment or so.

I don't have any numbers to quantify the extent of degradation for these recovery scenarios. Will need to run some performance runs to measure latency. Thank you for catching this @pierDipi !!

[lionelvillard]

/approve
/lgtm

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aavarghese, lionelvillard

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/common/scheduler/OWNERS [lionelvillard]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment