Docker image for github actions #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Warning Review failedThe pull request is closed. WalkthroughThis update includes major version changes to Scala and various Python dependencies, a new Dockerfile setup for an Alpine Linux Docker image, and modifications to the pull request template. These changes enhance compatibility with updated tools, streamline the development environment, and align dependencies with the latest versions. Changes
Poem
TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
4 tasks
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 25, 2025
Docker image for github actions
piyush-zlai
added a commit
that referenced
this pull request
May 2, 2025
…r Fetcher threadpool (#726) ## Summary PR to swap our metrics reporter from statsd to open telemetry metrics. We need otel to allow us to capture metrics in Etsy without the need of a prometheus statsd exporter sidecar that they've seen issues with occasionally. Otel in general is a popular metrics ingestion interface with a number of supported backends (e.g. prom / datadog / gcloud / aws cloudwatch). Wiring up Otel also enables us to set up traces and spans in the repo in the future. Broad changes: - Decouple bulk of the metrics reporting logic from the Metrics.Context. The metrics reporter we use is pluggable. Currently this is just the OpenTelemetry but in principle we can support others in the future. - Online module creates the appropriate [otel SDK](https://opentelemetry.io/docs/languages/java/sdk/) - either we use the [Http provider or the Prometheus Http server](https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters). We need the Http provider to plug into Vert.x as their Micrometer integration works with that. The Prom http server is what Etsy is keen we use. ## Checklist - [ ] Added Unit Tests - [X] Covered by existing CI - [X] Integration tested - [ ] Documentation update Tested via docker container and a local instance of open telemetry: Start up fetcher docker svc ``` docker run -v ~/.config/gcloud/application_default_credentials.json:/gcp/credentials.json -p 9000:9000 -e "GCP_PROJECT_ID=canary-443022" -e "GOOGLE_CLOUD_PROJECT=canary-443022" -e "GCP_BIGTABLE_INSTANCE_ID=zipline-canary-instance" -e "EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318" -e GOOGLE_APPLICATION_CREDENTIALS=/gcp/credentials.json zipline-fetcher:latest ``` And then otel: ``` ./otelcol --config otel-collector-config.yaml ... ``` We see: ``` 2025-04-18T17:35:37.351-0400 info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(ai.chronon) -> telemetry.sdk.language: Str(java) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.sdk.version: Str(1.49.0) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope ai.chronon 3.7.0-M11 Metric #0 Descriptor: -> Name: kv_store.bigtable.cache.insert -> Description: -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: kv_store.bigtable.multiGet.latency -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Count: 1 Sum: 229.000000 Min: 229.000000 Max: 229.000000 ExplicitBounds #0: 0.000000 ... Buckets #0, Count: 0 ... ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced OpenTelemetry-based metrics reporting throughout the platform, replacing the previous StatsD approach. - Added a Dockerfile and startup script for a new Fetcher service, supporting both AWS and GCP integrations with configurable metrics export. - Enhanced thread pool monitoring with a new executor that provides detailed metrics on task execution and queue status. - **Improvements** - Metrics tags are now structured as key-value maps, improving clarity and flexibility. - Metrics reporting is now context-aware, supporting per-dataset and per-table metrics. - Increased thread pool queue capacity for better throughput under load. - Replaced StatsD metrics configuration with OpenTelemetry OTLP in service launcher and build configurations. - **Bug Fixes** - Improved error handling and logging in metrics reporting and thread pool management. - **Chores** - Updated dependencies to include OpenTelemetry, Micrometer OTLP registry, Prometheus, OkHttp, and Kotlin libraries. - Refactored build and test configurations to support new telemetry libraries and remove deprecated dependencies. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
chewy-zlai
pushed a commit
that referenced
this pull request
May 15, 2025
…r Fetcher threadpool (#726) ## Summary PR to swap our metrics reporter from statsd to open telemetry metrics. We need otel to allow us to capture metrics in our clients without the need of a prometheus statsd exporter sidecar that they've seen issues with occasionally. Otel in general is a popular metrics ingestion interface with a number of supported backends (e.g. prom / datadog / gcloud / aws cloudwatch). Wiring up Otel also enables us to set up traces and spans in the repo in the future. Broad changes: - Decouple bulk of the metrics reporting logic from the Metrics.Context. The metrics reporter we use is pluggable. Currently this is just the OpenTelemetry but in principle we can support others in the future. - Online module creates the appropriate [otel SDK](https://opentelemetry.io/docs/languages/java/sdk/) - either we use the [Http provider or the Prometheus Http server](https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters). We need the Http provider to plug into Vert.x as their Micrometer integration works with that. The Prom http server is what our clients is keen we use. ## Checklist - [ ] Added Unit Tests - [X] Covered by existing CI - [X] Integration tested - [ ] Documentation update Tested via docker container and a local instance of open telemetry: Start up fetcher docker svc ``` docker run -v ~/.config/gcloud/application_default_credentials.json:/gcp/credentials.json -p 9000:9000 -e "GCP_PROJECT_ID=canary-443022" -e "GOOGLE_CLOUD_PROJECT=canary-443022" -e "GCP_BIGTABLE_INSTANCE_ID=zipline-canary-instance" -e "EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318" -e GOOGLE_APPLICATION_CREDENTIALS=/gcp/credentials.json zipline-fetcher:latest ``` And then otel: ``` ./otelcol --config otel-collector-config.yaml ... ``` We see: ``` 2025-04-18T17:35:37.351-0400 info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(ai.chronon) -> telemetry.sdk.language: Str(java) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.sdk.version: Str(1.49.0) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope ai.chronon 3.7.0-M11 Metric #0 Descriptor: -> Name: kv_store.bigtable.cache.insert -> Description: -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: kv_store.bigtable.multiGet.latency -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Count: 1 Sum: 229.000000 Min: 229.000000 Max: 229.000000 ExplicitBounds #0: 0.000000 ... Buckets #0, Count: 0 ... ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced OpenTelemetry-based metrics reporting throughout the platform, replacing the previous StatsD approach. - Added a Dockerfile and startup script for a new Fetcher service, supporting both AWS and GCP integrations with configurable metrics export. - Enhanced thread pool monitoring with a new executor that provides detailed metrics on task execution and queue status. - **Improvements** - Metrics tags are now structured as key-value maps, improving clarity and flexibility. - Metrics reporting is now context-aware, supporting per-dataset and per-table metrics. - Increased thread pool queue capacity for better throughput under load. - Replaced StatsD metrics configuration with OpenTelemetry OTLP in service launcher and build configurations. - **Bug Fixes** - Improved error handling and logging in metrics reporting and thread pool management. - **Chores** - Updated dependencies to include OpenTelemetry, Micrometer OTLP registry, Prometheus, OkHttp, and Kotlin libraries. - Refactored build and test configurations to support new telemetry libraries and remove deprecated dependencies. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
chewy-zlai
pushed a commit
that referenced
this pull request
May 15, 2025
…r Fetcher threadpool (#726) ## Summary PR to swap our metrics reporter from statsd to open telemetry metrics. We need otel to allow us to capture metrics in our clients without the need of a prometheus statsd exporter sidecar that they've seen issues with occasionally. Otel in general is a popular metrics ingestion interface with a number of supported backends (e.g. prom / datadog / gcloud / aws cloudwatch). Wiring up Otel also enables us to set up traces and spans in the repo in the future. Broad changes: - Decouple bulk of the metrics reporting logic from the Metrics.Context. The metrics reporter we use is pluggable. Currently this is just the OpenTelemetry but in principle we can support others in the future. - Online module creates the appropriate [otel SDK](https://opentelemetry.io/docs/languages/java/sdk/) - either we use the [Http provider or the Prometheus Http server](https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters). We need the Http provider to plug into Vert.x as their Micrometer integration works with that. The Prom http server is what our clients is keen we use. ## Checklist - [ ] Added Unit Tests - [X] Covered by existing CI - [X] Integration tested - [ ] Documentation update Tested via docker container and a local instance of open telemetry: Start up fetcher docker svc ``` docker run -v ~/.config/gcloud/application_default_credentials.json:/gcp/credentials.json -p 9000:9000 -e "GCP_PROJECT_ID=canary-443022" -e "GOOGLE_CLOUD_PROJECT=canary-443022" -e "GCP_BIGTABLE_INSTANCE_ID=zipline-canary-instance" -e "EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318" -e GOOGLE_APPLICATION_CREDENTIALS=/gcp/credentials.json zipline-fetcher:latest ``` And then otel: ``` ./otelcol --config otel-collector-config.yaml ... ``` We see: ``` 2025-04-18T17:35:37.351-0400 info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(ai.chronon) -> telemetry.sdk.language: Str(java) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.sdk.version: Str(1.49.0) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope ai.chronon 3.7.0-M11 Metric #0 Descriptor: -> Name: kv_store.bigtable.cache.insert -> Description: -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: kv_store.bigtable.multiGet.latency -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Count: 1 Sum: 229.000000 Min: 229.000000 Max: 229.000000 ExplicitBounds #0: 0.000000 ... Buckets #0, Count: 0 ... ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced OpenTelemetry-based metrics reporting throughout the platform, replacing the previous StatsD approach. - Added a Dockerfile and startup script for a new Fetcher service, supporting both AWS and GCP integrations with configurable metrics export. - Enhanced thread pool monitoring with a new executor that provides detailed metrics on task execution and queue status. - **Improvements** - Metrics tags are now structured as key-value maps, improving clarity and flexibility. - Metrics reporting is now context-aware, supporting per-dataset and per-table metrics. - Increased thread pool queue capacity for better throughput under load. - Replaced StatsD metrics configuration with OpenTelemetry OTLP in service launcher and build configurations. - **Bug Fixes** - Improved error handling and logging in metrics reporting and thread pool management. - **Chores** - Updated dependencies to include OpenTelemetry, Micrometer OTLP registry, Prometheus, OkHttp, and Kotlin libraries. - Refactored build and test configurations to support new telemetry libraries and remove deprecated dependencies. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
chewy-zlai
pushed a commit
that referenced
this pull request
May 16, 2025
…r Fetcher threadpool (#726) ## Summary PR to swap our metrics reporter from statsd to open telemetry metrics. We need otel to allow us to capture metrics in our clients without the need of a prometheus statsd exporter sidecar that they've seen issues with occasionally. Otel in general is a popular metrics ingestion interface with a number of supported baour clientsends (e.g. prom / datadog / gcloud / aws cloudwatch). Wiring up Otel also enables us to set up traces and spans in the repo in the future. Broad changes: - Decouple bulk of the metrics reporting logic from the Metrics.Context. The metrics reporter we use is pluggable. Currently this is just the OpenTelemetry but in principle we can support others in the future. - Online module creates the appropriate [otel SDK](https://opentelemetry.io/docs/languages/java/sdk/) - either we use the [Http provider or the Prometheus Http server](https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters). We need the Http provider to plug into Vert.x as their Micrometer integration works with that. The Prom http server is what our clients is keen we use. ## Cheour clientslist - [ ] Added Unit Tests - [X] Covered by existing CI - [X] Integration tested - [ ] Documentation update Tested via doour clientser container and a local instance of open telemetry: Start up fetcher doour clientser svc ``` doour clientser run -v ~/.config/gcloud/application_default_credentials.json:/gcp/credentials.json -p 9000:9000 -e "GCP_PROJECT_ID=canary-443022" -e "GOOGLE_CLOUD_PROJECT=canary-443022" -e "GCP_BIGTABLE_INSTANCE_ID=zipline-canary-instance" -e "EXPORTER_OTLP_ENDPOINT=http://host.doour clientser.internal:4318" -e GOOGLE_APPLICATION_CREDENTIALS=/gcp/credentials.json zipline-fetcher:latest ``` And then otel: ``` ./otelcol --config otel-collector-config.yaml ... ``` We see: ``` 2025-04-18T17:35:37.351-0400 info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(ai.chronon) -> telemetry.sdk.language: Str(java) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.sdk.version: Str(1.49.0) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope ai.chronon 3.7.0-M11 Metric #0 Descriptor: -> Name: kv_store.bigtable.cache.insert -> Description: -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: kv_store.bigtable.multiGet.latency -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> dataset: Str(TableId{tableId=CHRONON_METADATA}) -> environment: Str(kv_store) -> production: Str(false) StartTimestamp: 2025-04-18 21:31:52.180857637 +0000 UTC Timestamp: 2025-04-18 21:35:37.18442138 +0000 UTC Count: 1 Sum: 229.000000 Min: 229.000000 Max: 229.000000 ExplicitBounds #0: 0.000000 ... Buour clientsets #0, Count: 0 ... ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced OpenTelemetry-based metrics reporting throughout the platform, replacing the previous StatsD approach. - Added a Doour clientserfile and startup script for a new Fetcher service, supporting both AWS and GCP integrations with configurable metrics export. - Enhanced thread pool monitoring with a new executor that provides detailed metrics on task execution and queue status. - **Improvements** - Metrics tags are now structured as key-value maps, improving clarity and flexibility. - Metrics reporting is now context-aware, supporting per-dataset and per-table metrics. - Increased thread pool queue capacity for better throughput under load. - Replaced StatsD metrics configuration with OpenTelemetry OTLP in service launcher and build configurations. - **Bug Fixes** - Improved error handling and logging in metrics reporting and thread pool management. - **Chores** - Updated dependencies to include OpenTelemetry, Micrometer OTLP registry, Prometheus, OkHttp, and Kotlin libraries. - Refactored build and test configurations to support new telemetry libraries and remove deprecated dependencies. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
as title
Checklist
docker build --progress=plain -t chronon-base .Summary by CodeRabbit
New Features
Updates
black,isort,pytest,tox, and more.Documentation