Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
30004c3
Initial working version with integration tests
kumar-zlai Mar 24, 2025
888b831
Additional refactoring and fixed the full dag spec unit test
kumar-zlai Mar 25, 2025
48bd7bc
Refactored PubSubClient implementation into different components with…
kumar-zlai Mar 26, 2025
7aea3f3
Updated error handling and some future todos
kumar-zlai Mar 26, 2025
3c67236
Minor changes to bump up the gax dependency version
kumar-zlai Mar 26, 2025
589e155
Initial working version after refactoring the generic traits to not h…
kumar-zlai Mar 26, 2025
d04ab00
Refactoring of generic traits and gcp specific implementations complete
kumar-zlai Mar 26, 2025
5196a68
Minor scalafmt fixes
kumar-zlai Mar 26, 2025
5e1492c
Fixed gcloud auth issues using prod config in unit tests
kumar-zlai Mar 27, 2025
aefbd03
Minor change to fix compilation errors in 2.13 build
kumar-zlai Mar 27, 2025
aca2b93
Integrated nodeDao for pulling dependencies and removed DummyNode ref…
kumar-zlai Mar 28, 2025
a212411
Initial partially working version with missingRanges psuedo code
kumar-zlai Apr 1, 2025
525d650
Partial commit after modifying activity function signatures
kumar-zlai Apr 1, 2025
7ee665f
Initial working version with activity function signatures refactoring…
kumar-zlai Apr 2, 2025
55e8a61
Bug fix to wait on specific workflow run while executing missing Node…
kumar-zlai Apr 2, 2025
4bf8871
Added documentation for temporal/persistence layer logic
kumar-zlai Apr 2, 2025
48cae6b
Initial working version with integration tests
kumar-zlai Mar 24, 2025
b446cd9
Additional refactoring and fixed the full dag spec unit test
kumar-zlai Mar 25, 2025
6ff0047
Refactored PubSubClient implementation into different components with…
kumar-zlai Mar 26, 2025
d380311
Updated error handling and some future todos
kumar-zlai Mar 26, 2025
67c564c
Minor changes to bump up the gax dependency version
kumar-zlai Mar 26, 2025
3a327d8
Initial working version after refactoring the generic traits to not h…
kumar-zlai Mar 26, 2025
7e1c366
Refactoring of generic traits and gcp specific implementations complete
kumar-zlai Mar 26, 2025
818110c
Minor scalafmt fixes
kumar-zlai Mar 26, 2025
b531902
Fixed gcloud auth issues using prod config in unit tests
kumar-zlai Mar 27, 2025
215a883
Minor change to fix compilation errors in 2.13 build
kumar-zlai Mar 27, 2025
d9b02ca
Merge branch 'pubsub_poc' into temporal_persistence_layer_integration
kumar-zlai Apr 2, 2025
41ef316
Initial working logic for missing steps and some workflow refactoring
kumar-zlai Apr 4, 2025
01f78db
Merge branch 'main' into temporal_persistence_layer_integration
kumar-zlai Apr 4, 2025
020c73d
Merge branch 'missing_steps_logic' into temporal_persistence_layer_in…
kumar-zlai Apr 4, 2025
b34865f
save
kumar-zlai Apr 4, 2025
a6429e7
Dependency resolver logic refactor to api module with unit tests
kumar-zlai Apr 7, 2025
2cf48f1
Merge branch 'kumar/dependency_resolver_refactor' into temporal_persi…
kumar-zlai Apr 7, 2025
0470a52
Refactored and cleaned up missing steps logic
kumar-zlai Apr 7, 2025
aeca4e9
Changes to persist node table dependencies used for determining parti…
kumar-zlai Apr 8, 2025
49ccfeb
Merge branch 'main' into temporal_persistence_layer_integration
kumar-zlai Apr 8, 2025
70ec3e2
Removed auto-generated thrift files from gitignore conflict
kumar-zlai Apr 8, 2025
5a6f18f
Minor fix for resolving compilation errors due to moving DependencyRe…
kumar-zlai Apr 8, 2025
f1a12ad
Addressed PR comments
kumar-zlai Apr 14, 2025
ee7705c
Merge branch 'main' into temporal_persistence_layer_integration
kumar-zlai Apr 14, 2025
72dac1d
Merge branch 'main' into temporal_persistence_layer_integration
kumar-zlai Apr 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,9 @@ object DependencyResolver {
result
}

def computeInputRange(queryRange: PartitionRange, tableDep: TableDependency)(implicit
partitionSpec: PartitionSpec): Option[PartitionRange] = {
def computeInputRange(queryRange: PartitionRange, tableDep: TableDependency): Option[PartitionRange] = {

implicit val partitionSpec: PartitionSpec = queryRange.partitionSpec

require(queryRange != null, "Query range cannot be null")
require(queryRange.start != null, "Query range start cannot be null")
Expand Down
17 changes: 8 additions & 9 deletions api/thrift/orchestration.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,14 @@ union NodeUnion {
// TODO: add other types of nodes
}

enum NodeRunStatus {
UNKNOWN = 0,
WAITING = 1,
RUNNING = 2,
SUCCEEDED = 3,
FAILED = 4
}

// ====================== End of Modular Join Spark Job Args ===================

// ====================== Orchestration Service API Types ======================
Expand Down Expand Up @@ -250,15 +258,6 @@ struct UploadResponse {

// ====================== End of Orchestration Service API Types ======================

/**
* Below are dummy thrift objects for execution layer skeleton code using temporal
* TODO: Need to update these to fill in all the above relevant fields
**/
struct DummyNode {
1: optional string name
2: optional list<DummyNode> dependencies
}

/**
* -- Phase 0 plan -- (same as chronon oss)
* StagingQuery::query - [deps.table] >> query
Expand Down
6 changes: 4 additions & 2 deletions orchestration/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ scala_library(
maven_artifact("io.temporal:temporal-sdk"),
maven_artifact("io.temporal:temporal-serviceclient"),
maven_artifact("com.fasterxml.jackson.core:jackson-databind"),
maven_artifact_with_suffix("com.fasterxml.jackson.module:jackson-module-scala"),
maven_artifact("com.google.protobuf:protobuf-java"),
maven_artifact("com.google.code.findbugs:jsr305"),
maven_artifact("io.grpc:grpc-api"),
Expand Down Expand Up @@ -50,6 +51,7 @@ test_deps = _VERTX_DEPS + _SCALA_TEST_DEPS + [
maven_artifact("io.temporal:temporal-serviceclient"),
maven_artifact("com.fasterxml.jackson.core:jackson-core"),
maven_artifact("com.fasterxml.jackson.core:jackson-databind"),
maven_artifact_with_suffix("com.fasterxml.jackson.module:jackson-module-scala"),
maven_artifact("io.grpc:grpc-api"),
maven_artifact("io.grpc:grpc-core"),
maven_artifact("io.grpc:grpc-stub"),
Expand Down Expand Up @@ -84,7 +86,7 @@ scala_test_suite(
["src/test/**/*.scala"],
# Excluding integration tests
exclude = [
"src/test/**/NodeExecutionWorkflowIntegrationSpec.scala",
"src/test/**/NodeWorkflowIntegrationSpec.scala",
"src/test/**/GcpPubSubIntegrationSpec.scala",
],
),
Expand All @@ -96,7 +98,7 @@ scala_test_suite(
name = "integration_tests",
srcs = glob(
[
"src/test/**/NodeExecutionWorkflowIntegrationSpec.scala",
"src/test/**/NodeWorkflowIntegrationSpec.scala",
"src/test/**/GcpPubSubIntegrationSpec.scala",
],
),
Expand Down
Loading