[v2]adds consumer offset reset policy option to keda kafka scaler #925

grassiale · 2020-07-09T07:47:22Z

Signed-off-by: grassiale [email protected]

This PR is aimed at solving how Keda Kafka scaler handles the absence of committed offset with relation to the offset policy the consumer is following.
At the moment, if you create a scaledobject for a new consumer group having no committed offset, keda scaler will return a lag value equal to the latest offset in the topic it is consuming from. This often causes the scaler to scale relative deployment replicas to the max, because the latest offset in the topic is probably higher than the lagTreshold. This is fine for consumers having auto.offset.reset=earliest, since they will start to consume right-away all the messages in the topic; but is wrong for
auto.offset.reset=latest consumers, that will only consume new messages on the topic. The lag for this kind of consumer is now initially set to 0.

Checklist

A PR is opened to update the documentation on https://github.com/kedacore/keda-docs
Commits are signed with Developer Certificate of Origin (DCO)
Tests have been added

zroubalik · 2020-07-10T08:14:00Z

@grassiale thanks, this is great. Could you please open this PR against v2 branch? Most likely we won't release any new v1 release , so v2 is the right place for this change.

Signed-off-by: grassiale <[email protected]>

pkg/scalers/kafka_scaler.go

zroubalik · 2020-07-10T10:28:28Z

pkg/scalers/kafka_scaler.go

 	if consumerOffset == sarama.OffsetNewest || consumerOffset == sarama.OffsetOldest {
-		lag = latestOffset
+		if s.metadata.consumerOffsetReset == latest {
+			lag = 0


shouldn't this be lag = latestOffset ?

when you have no offset committed and you create a new consumer with reset latest policy then the lag should be 0 since the consumer is aligned with the latest offset on the topic.

ok, I am not a Kafka expert, but the properties naming is pretty confusing

zroubalik · 2020-07-10T10:28:55Z

@ppatierno PTAL if you have a time :)

zroubalik · 2020-07-10T10:30:24Z

@grassiale could you please add a test that is covering this property?

Signed-off-by: grassiale <[email protected]>

grassiale · 2020-07-10T15:14:28Z

Converting to draft because I'm implementing tests.

Signed-off-by: grassiale <[email protected]>

ppatierno · 2020-07-11T10:55:07Z

pkg/scalers/kafka_scaler.go

@@ -57,6 +65,7 @@ const (
 	lagThresholdMetricName   = "lagThreshold"
 	kafkaMetricType          = "External"
 	defaultKafkaLagThreshold = 10
+	defaultOffsetReset       = earliest


in Kafka the default is "latest", why are you setting "earliest" out of curiosity?

Yes, you are right on the default in kafka. I set latest as a default because, right now, if you create a kafka scaler, it will behave as if we set earliest, if no offset is committed. So I would not break the current behavior for anyone upgrading from a previous version and not seeing the option.
But maybe, since this is a new major version of Keda, it could be safer to be coherent with Kafka defaults.
Let me know what you think, I can change it accordingly to the decision.

yes I would agree, let's see what @zroubalik thinks.

I agree, it makes sense to set it to Kafka default, a new major release is a great fit for a change like this. We could add a small note to the docs about it.

I've done the change and also changed the docs, let me know if it needs further clarification there.

ppatierno · 2020-07-11T10:57:46Z

pkg/scalers/kafka_scaler_test.go

+	brokers             []string
+	group               string
+	topic               string
+	consumerOffsetReset string


I would use the custom type offsetResetPolicy instead of string here.

Signed-off-by: grassiale <[email protected]>

ppatierno · 2020-07-12T09:26:20Z

pkg/scalers/kafka_scaler.go

+	group               string
+	topic               string
+	lagThreshold        int64
+	consumerOffsetReset offsetResetPolicy


can't we find a better name instead of consumerOffsetReset. Actually the type name would be a great name for the struct field as well so offsetResetPolicy. It's not the first time that a field name is the same as the type name, or?

Absolutely ;)

ppatierno

LGTM other than the comments I left :-)

zroubalik · 2020-07-13T09:17:20Z

LGTM other than the comments I left :-)

@ppatierno thanks for the review!

Signed-off-by: grassiale <[email protected]>

grassiale · 2020-07-13T09:36:33Z

I also started to add a small E2E test for this particular case, let me know if you don't think this is necessary.

Signed-off-by: grassiale <[email protected]>

zroubalik · 2020-07-13T10:00:46Z

@grassiale e2e tests are highly appreciated. So would be really great if you could add a scenario for this, but it is not mandatory. But if you want to test functionality that is not that much relevant (or is not direct affected by) KEDA but it is more testing Kafka capability, you don't have to add it.

grassiale · 2020-07-15T16:52:32Z

Hi @zroubalik , I've been able to implement some e2e tests for this case, the thing is:

the current consumer being used in e2e tests is not committing offsets, and this is a problem since lag for keda in "latest" configuration will always be 0.
I had to tweak some other stuff in the tests in order for the tests I've added to work.
So maybe this is a bit out of scope for this PR. Shall we go on and finalize this pr and maybe I shall open an issue for this?

zroubalik · 2020-07-16T08:01:44Z

  So maybe this is a bit out of scope for this PR. Shall we go on and finalize this pr and maybe I shall open an issue for this?

Agree, makes sense.

* adds consumer offset reset policy option to keda kafka scaler Signed-off-by: grassiale <[email protected]>

grassiale requested review from ahmelsayed and zroubalik as code owners July 9, 2020 07:47

grassiale mentioned this pull request Jul 9, 2020

[v2] adds doc for consumer offset reset policy option to keda kafka scaler kedacore/keda-docs#206

Merged

grassiale changed the base branch from master to v2 July 10, 2020 09:37

grassiale force-pushed the feature/kafka-scaler-offset-reset-policy branch from ce2b760 to 472a3b7 Compare July 10, 2020 09:42

adds consumer offset reset policy option to keda kafka scaler

b0432a9

Signed-off-by: grassiale <[email protected]>

grassiale force-pushed the feature/kafka-scaler-offset-reset-policy branch from 472a3b7 to b0432a9 Compare July 10, 2020 09:53

zroubalik requested changes Jul 10, 2020

View reviewed changes

grassiale added 2 commits July 10, 2020 12:31

fixing metadata label that was lost in rename

c1967c8

Signed-off-by: grassiale <[email protected]>

changes default offset reset policy to be coherent with current behavior

6de5df5

Signed-off-by: grassiale <[email protected]>

grassiale marked this pull request as draft July 10, 2020 15:13

adds metadata parse test for consumer offset reset parameter

81f5605

Signed-off-by: grassiale <[email protected]>

ppatierno reviewed Jul 11, 2020

View reviewed changes

uses offsetResetPolicy type in test data

0187474

Signed-off-by: grassiale <[email protected]>

grassiale changed the title ~~adds consumer offset reset policy option to keda kafka scaler~~ [v2]adds consumer offset reset policy option to keda kafka scaler Jul 12, 2020

ppatierno reviewed Jul 12, 2020

View reviewed changes

ppatierno approved these changes Jul 12, 2020

View reviewed changes

renames offset reset policy property and sets default to latest

f8a4648

Signed-off-by: grassiale <[email protected]>

grassiale force-pushed the feature/kafka-scaler-offset-reset-policy branch from 83f9c9b to f8a4648 Compare July 13, 2020 09:35

changes default outcome for offsetResetPolicy in tests

81246de

Signed-off-by: grassiale <[email protected]>

grassiale marked this pull request as ready for review July 16, 2020 10:38

zroubalik merged commit e107f26 into kedacore:v2 Jul 20, 2020

zroubalik pushed a commit that referenced this pull request Aug 6, 2020

[v2]adds consumer offset reset policy option to keda kafka scaler (#925)

dcaa94c

* adds consumer offset reset policy option to keda kafka scaler Signed-off-by: grassiale <[email protected]>

ahmelsayed mentioned this pull request Aug 11, 2020

update tests for v2 #985

Merged

3 tasks

SpiritZhou pushed a commit to SpiritZhou/keda that referenced this pull request Jul 18, 2023

fix: Provide correct URL for roadmap in navigation (kedacore#925)

1118127

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v2]adds consumer offset reset policy option to keda kafka scaler #925

[v2]adds consumer offset reset policy option to keda kafka scaler #925

grassiale commented Jul 9, 2020 •

edited

Loading

zroubalik commented Jul 10, 2020

zroubalik Jul 10, 2020

grassiale Jul 10, 2020

zroubalik Jul 10, 2020

zroubalik commented Jul 10, 2020

zroubalik commented Jul 10, 2020

grassiale commented Jul 10, 2020

ppatierno Jul 11, 2020

grassiale Jul 12, 2020

ppatierno Jul 12, 2020

zroubalik Jul 12, 2020

grassiale Jul 13, 2020

ppatierno Jul 11, 2020

ppatierno Jul 12, 2020

grassiale Jul 13, 2020

ppatierno left a comment

zroubalik commented Jul 13, 2020

grassiale commented Jul 13, 2020 •

edited

Loading

zroubalik commented Jul 13, 2020

grassiale commented Jul 15, 2020

zroubalik commented Jul 16, 2020

[v2]adds consumer offset reset policy option to keda kafka scaler #925

[v2]adds consumer offset reset policy option to keda kafka scaler #925

Conversation

grassiale commented Jul 9, 2020 • edited Loading

Checklist

zroubalik commented Jul 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zroubalik commented Jul 10, 2020

zroubalik commented Jul 10, 2020

grassiale commented Jul 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppatierno left a comment

Choose a reason for hiding this comment

zroubalik commented Jul 13, 2020

grassiale commented Jul 13, 2020 • edited Loading

zroubalik commented Jul 13, 2020

grassiale commented Jul 15, 2020

zroubalik commented Jul 16, 2020

grassiale commented Jul 9, 2020 •

edited

Loading

grassiale commented Jul 13, 2020 •

edited

Loading