Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block-builder: pull jobs from scheduler #10118

Merged
merged 79 commits into from
Jan 18, 2025
Merged
Changes from 1 commit
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
f457910
Merge from davidgrant/block-builder-scheduler-reconcile
seizethedave Nov 18, 2024
8f9d996
Don't need to get fancy with critical section: no contention is possi…
seizethedave Nov 18, 2024
6712985
Document s.committed.
seizethedave Nov 18, 2024
f97e4cb
Merge
seizethedave Nov 24, 2024
4cfeed9
Fix merge slop.
seizethedave Nov 24, 2024
eaed668
Add proto defs.
seizethedave Nov 4, 2024
d37b1f2
Initial RPC protos.
seizethedave Nov 4, 2024
35d1868
Log an error if flushing fails.
seizethedave Nov 24, 2024
81dee72
Include RPC forms of assignJob/updateJob.
seizethedave Nov 4, 2024
3772a12
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
seizethedave Nov 25, 2024
7c3f2da
Fleshing out the client module.
seizethedave Nov 25, 2024
db44d45
Appease the linter.
seizethedave Nov 25, 2024
5cc39ce
workerID param not used.
seizethedave Nov 25, 2024
3c6a566
CompleteJob: no context needed.
seizethedave Nov 25, 2024
91b8e83
The linter.
seizethedave Nov 25, 2024
7355de9
More logs.
seizethedave Nov 25, 2024
c34617d
Fix circular reference. Initialize client.
seizethedave Nov 29, 2024
9f67caa
schedulerproto -> schedulerpb
seizethedave Nov 29, 2024
47b5662
merge
seizethedave Nov 29, 2024
188217f
Fix compile stuff.
seizethedave Nov 30, 2024
179018e
Getting builder/scheduler up in docker-compose.
seizethedave Nov 30, 2024
f789447
Clean unneeded flags.
seizethedave Nov 30, 2024
bad7dc2
Fixes for BB<>scheduler communications:
seizethedave Nov 30, 2024
762012f
Fix block-builder's scheduler_config so it can connect to a scheduler.
seizethedave Nov 30, 2024
671f4b3
Shorter update interval for testing.
seizethedave Nov 30, 2024
f760fa8
Fix unloggable key.
seizethedave Nov 30, 2024
df7e43e
Longer job working time.
seizethedave Nov 30, 2024
6ad943c
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
seizethedave Dec 2, 2024
63313b0
Undo blockbuilder.go hacking.
seizethedave Dec 2, 2024
509e9a8
Peel off block-builder changes not ready for this PR.
seizethedave Dec 2, 2024
fcdd240
Fix log message when creating a new job.
seizethedave Dec 2, 2024
77b00ac
Add some tests for scheduler client module. And fix some things found…
seizethedave Dec 2, 2024
6ed93cc
Linter.
seizethedave Dec 2, 2024
23e0db2
Lint: AGPL header.
seizethedave Dec 2, 2024
fa14a29
More info in the SchedulerClient comment.
seizethedave Dec 2, 2024
54f1af4
Remove superfluous comment.
seizethedave Dec 2, 2024
02d7856
Defer Kafka offset flushing to a later PR.
seizethedave Dec 2, 2024
b175c80
Revert "Undo blockbuilder.go hacking."
seizethedave Dec 3, 2024
b419dd7
Revert "Peel off block-builder changes not ready for this PR."
seizethedave Dec 3, 2024
cb250cd
Separate run functions for standalone, pull modes.
seizethedave Dec 3, 2024
5e53ac8
Working on a separate codepath for 'pull mode'.
seizethedave Dec 4, 2024
9944af4
don't need PartitionState.Clone.
seizethedave Dec 4, 2024
ecafbf5
Better startup initialization. Restore the assigned partition validat…
seizethedave Dec 4, 2024
5af6624
Lint.
seizethedave Dec 4, 2024
f17ee08
More changes to blockbuilder to support testing. Fix terminating loop…
seizethedave Dec 5, 2024
f980d09
Working on a pull-mode consumption test.
seizethedave Dec 5, 2024
815559f
merge
seizethedave Dec 19, 2024
9d238a2
Fix build.
seizethedave Dec 19, 2024
727f684
lint
seizethedave Dec 19, 2024
3b766a0
Add CycleEndTs, CycleEndOffset fields to JobSpec.
seizethedave Dec 20, 2024
d42f7bb
Use new JobSpec fields in consumeJob
seizethedave Dec 20, 2024
4281f8c
Fix pull-mode test.
seizethedave Dec 20, 2024
23024bc
Make proto fields consistent.
seizethedave Dec 20, 2024
f9fa09d
Re-fix test.
seizethedave Dec 20, 2024
47f30b9
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
seizethedave Dec 20, 2024
3c873ef
Remove erroneous provenance headers.
seizethedave Dec 20, 2024
32aaa02
Pull-mode-ize another block-builder test.
seizethedave Dec 20, 2024
6e8c976
TestBlockBuilder_StartWithExistingCommit_PullMode
seizethedave Dec 20, 2024
cdadc8c
Add a jobIteration counter for tests.
seizethedave Dec 20, 2024
c85ce3a
More pull-mode tests.
seizethedave Dec 20, 2024
1f04656
Expose partition as a param to produceSamples
seizethedave Dec 21, 2024
904efbc
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
seizethedave Dec 21, 2024
96b34c9
Add a second job to the general PullMode test.
seizethedave Dec 21, 2024
a2eab72
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
seizethedave Jan 2, 2025
bf1f77d
Add job key details to log fields.
seizethedave Jan 2, 2025
3738b7f
Also make sure workerID isn't empty.
seizethedave Jan 2, 2025
a1e2945
remove 'failed' language from wrapped error string.
seizethedave Jan 2, 2025
6206183
Temporarily fix weird test flake.
seizethedave Jan 2, 2025
0062ff8
Cleanup of var names/comments.
seizethedave Jan 2, 2025
4e8dd7a
Validate completion keys.
seizethedave Jan 2, 2025
929ab6c
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
seizethedave Jan 16, 2025
eed567b
Flush and close gRPC conn on shutdown.
seizethedave Jan 16, 2025
fd9f31d
mv blockBuilderPullModeConfig to blockbuilder_test.go.
seizethedave Jan 16, 2025
0f4a620
Do some assertions of scheduler call counts.
seizethedave Jan 16, 2025
ded6790
Less parallel tests.
seizethedave Jan 16, 2025
78f4bbb
Replace Flush scheme with a Close method that runs after the run-loop…
seizethedave Jan 17, 2025
eeeb3ac
Replace counts() with single-use count getters.
seizethedave Jan 17, 2025
baf97cf
Verify Close() called on shutdown.
seizethedave Jan 17, 2025
53a889c
Fix order of args to Equal().
seizethedave Jan 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add proto defs.
seizethedave committed Nov 24, 2024
commit eaed668eddfe56c970040f2b5aa4af172be6e0d6
41 changes: 41 additions & 0 deletions pkg/blockbuilder/scheduler/scheduler.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Provenance-includes-location: https://github.com/cortexproject/cortex/blob/master/pkg/frontend/v1/frontendv1pb/frontend.proto
// Provenance-includes-license: Apache-2.0
// Provenance-includes-copyright: The Cortex Authors.

syntax = "proto3";

// Protobuf package should not be changed when moving around go packages
// in order to not break backward compatibility.
package scheduler;

import "gogoproto/gogo.proto";
import "github.com/grafana/dskit/httpgrpc/httpgrpc.proto";

option (gogoproto.marshaler_all) = true;
option (gogoproto.unmarshaler_all) = true;

service BlockBuilderScheduler {
rpc GetJob(GetJobRequest) returns (GetJobResponse);
}

message GetJobRequest {
int64 worker_id = 1;
}

message BlockBuilderJob {
topic string = 1;
partition int32 = 2;
start_offset int64 = 3;
end_offset int64 = 4;
/*
commitRecTs time.Time
lastSeenOffset int64
lastBlockEndTs time.Time
*/
}

message GetJobResponse {
BlockBuilderJob job = 1;
}