Skip to content

Commit

Permalink
fix!: Remove Kafka auto-commit. Fixes #327
Browse files Browse the repository at this point in the history
  • Loading branch information
@alexec
alexec committed Sep 16, 2021
1 parent da2c037 commit 76e1d95
Show file tree
Hide file tree
Showing 19 changed files with 789 additions and 738 deletions.
929 changes: 365 additions & 564 deletions api/v1alpha1/generated.pb.go
View file

Large diffs are not rendered by default.

7 changes: 0 additions & 7 deletions api/v1alpha1/generated.proto
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 0 additions & 5 deletions api/v1alpha1/kafka_auto_commit.go
View file

This file was deleted.

2 changes: 0 additions & 2 deletions api/v1alpha1/kafka_source.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,4 @@ type KafkaSource struct {
Kafka `json:",inline" protobuf:"bytes,1,opt,name=kafka"`
// +kubebuilder:default=Last
StartOffset KafkaOffset `json:"startOffset,omitempty" protobuf:"bytes,2,opt,name=startOffset,casttype=KafkaOffset"`
// +kubebuilder:default={enable:false}
AutoCommit KafkaAutoCommit `json:"autoCommit,omitempty" protobuf:"bytes,3,opt,name=autoCommit"`
}
16 changes: 0 additions & 16 deletions api/v1alpha1/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 0 additions & 18 deletions config/ci.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3539,15 +3539,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down Expand Up @@ -10494,15 +10485,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down
9 changes: 0 additions & 9 deletions config/crd/bases/dataflow.argoproj.io_pipelines.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3534,15 +3534,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down
9 changes: 0 additions & 9 deletions config/crd/bases/dataflow.argoproj.io_steps.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3365,15 +3365,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down
18 changes: 0 additions & 18 deletions config/default.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3539,15 +3539,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down Expand Up @@ -10494,15 +10485,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down
18 changes: 0 additions & 18 deletions config/dev.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3539,15 +3539,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down Expand Up @@ -10494,15 +10485,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down
18 changes: 0 additions & 18 deletions config/quick-start.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3539,15 +3539,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down Expand Up @@ -10494,15 +10485,6 @@ spec:
type: object
kafka:
properties:
autoCommit:
default:
enable: false
properties:
enable:
type: boolean
required:
- enable
type: object
brokers:
items:
type: string
Expand Down
9 changes: 3 additions & 6 deletions dsls/python/argo_dataflow/pipeline.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -527,16 +527,13 @@ def dump(self):


class KafkaSource(Source):
def __init__(self, topic, name=None, retry=None, autoCommit=None):
def __init__(self, topic, name=None, retry=None):
super().__init__(name=name, retry=retry)
self._topic = topic
self._autoCommit = autoCommit

def dump(self):
x = super().dump()
x['topic'] = self._topic
if self._autoCommit:
x['autoCommit'] = self._autoCommit
return {'kafka': x}


Expand All @@ -560,8 +557,8 @@ def http(name=None, retry=None, serviceName=None):
return HTTPSource(name=name, serviceName=serviceName, retry=retry)


def kafka(topic, name=None, retry=None, autoCommit=None):
return KafkaSource(topic, name=name, retry=retry, autoCommit=autoCommit)
def kafka(topic, name=None, retry=None):
return KafkaSource(topic, name=name, retry=retry)


def stan(subject, name=None, retry=None):
Expand Down
2 changes: 1 addition & 1 deletion examples/git/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ module github.com/argoproj-labs/argo-dataflow/examples/git

go 1.16

require github.com/argoproj-labs/argo-dataflow v0.0.101
require github.com/argoproj-labs/argo-dataflow v0.0.102
Loading

0 comments on commit 76e1d95

Please sign in to comment.