Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove some unused pulsar properties #3866

Merged
merged 4 commits into from
Aug 6, 2024
Merged

Remove some unused pulsar properties #3866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f5afd63
remove some unused pulsar properties
d80tb7 Aug 4, 2024
60754b6
wip
d80tb7 Aug 5, 2024
b60efd6
Merge branch 'master' of github.com:armadaproject/armada into f/chris…
d80tb7 Aug 5, 2024
e91cf82
Merge branch 'master' into f/chrisma/remove-unneeded-pulsar-config
d80tb7 Aug 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions config/armada/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,9 @@ submission:
pulsar:
URL: "pulsar://pulsar:6650"
jobsetEventsTopic: "events"
redisFromPulsarSubscription: "RedisFromPulsar"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove RedisFromPulsarSubscription from types.go below too?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes- good point!

dedupTable: pulsar_submit_dedup
maxConnectionsPerBroker: 1
compressionType: zlib
compressionLevel: faster
eventsPrinter: false
eventsPrinterSubscription: "EventsPrinter"
maxAllowedEventsPerMessage: 1000
maxAllowedMessageSize: 4194304 # 4MB
sendTimeout: 5s
Expand Down
4 changes: 0 additions & 4 deletions internal/scheduleringester/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,6 @@ type Configuration struct {
BatchSize int
// Maximum time since the last batch before a batch will be inserted into the database
BatchDuration time.Duration
// Time for which the pulsar consumer will wait for a new message before retrying
PulsarReceiveTimeout time.Duration
// Time for which the pulsar consumer will back off after receiving an error on trying to receive a message
PulsarBackoffTime time.Duration
// If non-nil, configures pprof profiling
Profiling *profilingconfig.ProfilingConfig
}
8 changes: 3 additions & 5 deletions internal/server/configuration/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,13 @@ type PulsarConfig struct {
// Authentication type. For now only "JWT" auth is valid
AuthenticationType string
// Path to the JWT token (must exist). This must be set if AuthenticationType is "JWT"
JwtTokenPath string
JobsetEventsTopic string
RedisFromPulsarSubscription string
JwtTokenPath string
// The pulsar topic that Jobset Events will be published to
JobsetEventsTopic string
// Compression to use. Valid values are "None", "LZ4", "Zlib", "Zstd". Default is "None"
CompressionType pulsar.CompressionType
// Compression Level to use. Valid values are "Default", "Better", "Faster". Default is "Default"
CompressionLevel pulsar.CompressionLevel
// Settings for deduplication, which relies on a postgres server.
DedupTable string
// Maximum allowed Events per message
MaxAllowedEventsPerMessage int `validate:"gte=0"`
// Maximum allowed message size in bytes
Expand Down