diff --git a/docs/design-docs/logger.md b/docs/design-docs/logger.md index b13436423b..1c45529540 100644 --- a/docs/design-docs/logger.md +++ b/docs/design-docs/logger.md @@ -1,6 +1,6 @@ # Logger -The logger is designed to track key training metrics (including distributed metrics with reductions and timing), as well as providing integration with logging backends like WandB and Tensorboard. +The logger is designed to track key training metrics (including distributed metrics with reductions and timing), as well as providing integration with logging backends like WandB, Tensorboard, and MLflow. ## Requirements @@ -9,12 +9,13 @@ The logger is designed to track key training metrics (including distributed metr * Logging: * WandB * Tensorboard + * MLflow ## Overall Design Since there is a single controller, the single process running the main training loop will gather the metrics and do the logging. -To handle multiple logger backends, we will have a {py:class}`LoggerInterface ` interface that the {py:class}`TensorboardLogger ` and {py:class}`WandbLogger ` will implement: +To handle multiple logger backends, we will have a {py:class}`LoggerInterface ` interface that the {py:class}`TensorboardLogger `, {py:class}`WandbLogger `, and {py:class}`MLflowLogger ` will implement: ```python class LoggerInterface(ABC): @@ -34,10 +35,11 @@ class LoggerInterface(ABC): A {py:class}`Logger ` wrapper class will also implement {py:class}`LoggerInterface ` and maintain a list of loggers to which it delegates writing logs. This will be the main class the user uses in the training loop. Usage example: ```python -# Initialize logger with both wandb and tensorboard enabled +# Initialize logger with wandb, tensorboard, and mlflow enabled logging_config = { "wandb_enabled": True, "tensorboard_enabled": False, + "mlflow_enabled": True, "wandb": { "project": "grpo-dev", @@ -46,17 +48,72 @@ logging_config = { "tensorboard": { "log_dir": "logs", }, + "mlflow": { + "experiment_name": "nemo-rl-experiment", + "run_name": "grpo-dev-run", + "tracking_uri": None, # Use local tracking + }, } logger = Logger( cfg=logger_config, ) -# Log metrics, will go to both wandb and tensorboard +# Log metrics, will go to all enabled backends logger.log_metrics({ "loss": 0.123, }, step=10) ``` +## Supported Logging Backends + +The logger supports three main logging backends: + +### WandB (Weights & Biases) +- Provides cloud-based experiment tracking +- Supports custom step metrics for better visualization +- Includes built-in hyperparameter logging +- Offers rich visualization and collaboration features + +### Tensorboard +- Local file-based logging +- Standard TensorBoard visualization +- Supports hyperparameter logging via HParams +- Lightweight and self-contained + +### MLflow +- Comprehensive platform for experiment tracking and model management +- Supports both local and remote tracking servers +- Provides model versioning and artifact management +- Includes a web UI for experiment visualization +- Supports model deployment and serving + +#### MLflow Configuration + +MLflow can be configured with the following parameters: + +```python +mlflow: + experiment_name: "nemo-rl-experiment" # Name of the MLflow experiment + run_name: "my-training-run" # Run name + tracking_uri: "http://localhost:5000" # Optional tracking server URI +``` + + +#### MLflow UI + +After starting training with MLflow enabled, you can view the MLflow UI to monitor your experiments: + +```bash +# Start MLflow UI (run in a separate terminal) +mlflow ui --host 0.0.0.0 --port 5000 +``` + +Then access the UI at `http://127.0.0.1:5000/` to view: +- Training runs and experiments +- Metrics (loss, validation metrics, etc.) +- Hyperparameters +- Model artifacts and checkpoints + ## Validation Pretty Logging The logger supports pretty-formatted logging of validation samples to help visualize model outputs during training. This feature is controlled by the `num_val_samples_to_print` configuration parameter. @@ -65,6 +122,7 @@ The logger supports pretty-formatted logging of validation samples to help visua logger: wandb_enabled: false tensorboard_enabled: false + mlflow_enabled: false num_val_samples_to_print: 10 ``` @@ -82,9 +140,9 @@ When enabled, the pretty logging will generate formatted text similar to: ## GPU Metric Logging -NeMo RL monitors GPU memory and utilization through [system metrics](https://docs.ray.io/en/latest/ray-observability/reference/system-metrics.html#system-metrics) exposed by Ray nodes. While Ray makes these metrics available for tools like Prometheus, NeMo RL directly polls GPU memory and utilization data and logs them to TensorBoard and/or WandB. +NeMo RL monitors GPU memory and utilization through [system metrics](https://docs.ray.io/en/latest/ray-observability/reference/system-metrics.html#system-metrics) exposed by Ray nodes. While Ray makes these metrics available for tools like Prometheus, NeMo RL directly polls GPU memory and utilization data and logs them to TensorBoard, WandB, and/or MLflow. -This approach allows us to offer the same GPU metric tracking on all loggers (not just Wandb) and simplifies the implementation greatly. +This approach allows us to offer the same GPU metric tracking on all loggers and simplifies the implementation greatly. This feature is enabled with the `monitor_gpus` configuration parameter. The frequency of data collection and flushing to the loggers is controlled by the `gpu_collection_interval` and `gpu_flush_interval` parameters, both specified in seconds. @@ -92,6 +150,7 @@ This feature is enabled with the `monitor_gpus` configuration parameter. The fre logger: wandb_enabled: false tensorboard_enabled: false + mlflow_enabled: false monitor_gpus: true gpu_monitoring: collection_interval: 10 @@ -103,7 +162,7 @@ While it is feasible to monitor using remote workers, the implementation require * Logs sent back to the driver do not introduce significant overhead. * Metrics remain clear and interpretable, avoiding issues like double counting caused by colocated workers. * Workers can gracefully flush their logs in case of failure. -* Logging behaves consistently across TensorBoard and Wandb. +* Logging behaves consistently across TensorBoard, WandB, and MLflow. * Workers that spawn other workers accurately report the total resource usage of any grandchild workers. Due to these complexities, we opted for a simpler approach: collecting metrics exposed by the Ray metrics server from the driver. diff --git a/examples/configs/dpo.yaml b/examples/configs/dpo.yaml index 110729a966..41062a01d5 100755 --- a/examples/configs/dpo.yaml +++ b/examples/configs/dpo.yaml @@ -153,6 +153,7 @@ logger: log_dir: "logs" # Base directory for all logs wandb_enabled: false # Make sure you do a ``wandb login [Your API key]'' before running tensorboard_enabled: false + mlflow_enabled: false # Disable MLflow logging monitor_gpus: true # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "dpo-dev" diff --git a/examples/configs/grpo-deepscaler-1.5b-8K.yaml b/examples/configs/grpo-deepscaler-1.5b-8K.yaml index ce5ed73c17..2cb647dc95 100644 --- a/examples/configs/grpo-deepscaler-1.5b-8K.yaml +++ b/examples/configs/grpo-deepscaler-1.5b-8K.yaml @@ -127,6 +127,7 @@ logger: num_val_samples_to_print: 0 # Number of validation samples to pretty print on terminal wandb_enabled: false tensorboard_enabled: false + mlflow_enabled: false monitor_gpus: false # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "grpo-dev" diff --git a/examples/configs/grpo_math_1B.yaml b/examples/configs/grpo_math_1B.yaml index fd944fa9e7..b70ab94e7d 100644 --- a/examples/configs/grpo_math_1B.yaml +++ b/examples/configs/grpo_math_1B.yaml @@ -132,6 +132,7 @@ logger: num_val_samples_to_print: 0 # Number of validation samples to pretty print on terminal wandb_enabled: false tensorboard_enabled: false + mlflow_enabled: false # Disable MLflow logging monitor_gpus: true # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "grpo-dev" diff --git a/examples/configs/grpo_math_1B_megatron.yaml b/examples/configs/grpo_math_1B_megatron.yaml index 7a8a651a54..e5eb8fab87 100644 --- a/examples/configs/grpo_math_1B_megatron.yaml +++ b/examples/configs/grpo_math_1B_megatron.yaml @@ -153,6 +153,7 @@ logger: num_val_samples_to_print: 0 # Number of validation samples to pretty print on terminal wandb_enabled: false tensorboard_enabled: false + mlflow_enabled: false # Disable MLflow logging monitor_gpus: false # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "grpo-dev" diff --git a/examples/configs/grpo_sliding_puzzle.yaml b/examples/configs/grpo_sliding_puzzle.yaml index 8493bfc40e..f7e7e2fba3 100644 --- a/examples/configs/grpo_sliding_puzzle.yaml +++ b/examples/configs/grpo_sliding_puzzle.yaml @@ -52,6 +52,7 @@ logger: num_val_samples_to_print: 0 # Number of validation samples to pretty print on terminal wandb_enabled: false tensorboard_enabled: false + mlflow_enabled: false monitor_gpus: true # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "grpo-dev" diff --git a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp1.v2.yaml b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp1.v2.yaml index 368aafd705..8655dede0a 100644 --- a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp1.v2.yaml +++ b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp1.v2.yaml @@ -78,6 +78,7 @@ logger: log_dir: "logs" wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp2-quick.v2.yaml b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp2-quick.v2.yaml index 91e2fb9569..e8e8f472c0 100644 --- a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp2-quick.v2.yaml +++ b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-fsdp2tp2-quick.v2.yaml @@ -78,6 +78,7 @@ logger: log_dir: "logs" wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatron.yaml b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatron.yaml index 6139ce6788..3dbb98006e 100644 --- a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatron.yaml +++ b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatron.yaml @@ -111,6 +111,7 @@ logger: log_dir: "logs" wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatrontp2pp2-quick.yaml b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatrontp2pp2-quick.yaml index 733cce01da..082520095e 100644 --- a/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatrontp2pp2-quick.yaml +++ b/examples/configs/recipes/llm/dpo-llama3.1-8b-instruct-4n8g-megatrontp2pp2-quick.yaml @@ -111,6 +111,7 @@ logger: log_dir: "logs" wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/dpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v2.yaml b/examples/configs/recipes/llm/dpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v2.yaml index c12edd2404..afe19bf4ea 100644 --- a/examples/configs/recipes/llm/dpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v2.yaml +++ b/examples/configs/recipes/llm/dpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v2.yaml @@ -78,6 +78,7 @@ logger: log_dir: "logs" wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-gemma3-1b-it-1n8g-fsdp2tp1.yaml b/examples/configs/recipes/llm/grpo-gemma3-1b-it-1n8g-fsdp2tp1.yaml index 6bbcd95edd..a2c61ebce9 100644 --- a/examples/configs/recipes/llm/grpo-gemma3-1b-it-1n8g-fsdp2tp1.yaml +++ b/examples/configs/recipes/llm/grpo-gemma3-1b-it-1n8g-fsdp2tp1.yaml @@ -108,6 +108,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-gemma3-27b-it-16n8g-fsdp2tp8sp-actckpt-long.yaml b/examples/configs/recipes/llm/grpo-gemma3-27b-it-16n8g-fsdp2tp8sp-actckpt-long.yaml index af4bb6945d..0fe72a150d 100644 --- a/examples/configs/recipes/llm/grpo-gemma3-27b-it-16n8g-fsdp2tp8sp-actckpt-long.yaml +++ b/examples/configs/recipes/llm/grpo-gemma3-27b-it-16n8g-fsdp2tp8sp-actckpt-long.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-llama3.1-8b-instruct-4n8g-fsdp2tp1-long.v3.yaml b/examples/configs/recipes/llm/grpo-llama3.1-8b-instruct-4n8g-fsdp2tp1-long.v3.yaml index b854eb7d38..2ad3228001 100644 --- a/examples/configs/recipes/llm/grpo-llama3.1-8b-instruct-4n8g-fsdp2tp1-long.v3.yaml +++ b/examples/configs/recipes/llm/grpo-llama3.1-8b-instruct-4n8g-fsdp2tp1-long.v3.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v3.yaml b/examples/configs/recipes/llm/grpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v3.yaml index 9f92be089b..3caf0ccdbd 100644 --- a/examples/configs/recipes/llm/grpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v3.yaml +++ b/examples/configs/recipes/llm/grpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v3.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt-long.v3.yaml b/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt-long.v3.yaml index 2a1a151ea5..336326a5e9 100644 --- a/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt-long.v3.yaml +++ b/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt-long.v3.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt.v3.yaml b/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt.v3.yaml index 06ae6b4637..fcc7b0460a 100644 --- a/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt.v3.yaml +++ b/examples/configs/recipes/llm/grpo-qwen2.5-32b-16n8g-fsdp2tp8sp-actckpt.v3.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-qwen2.5-7b-instruct-4n8g-fsdp2tp4sp.v3.yaml b/examples/configs/recipes/llm/grpo-qwen2.5-7b-instruct-4n8g-fsdp2tp4sp.v3.yaml index 00a40de4d0..585a8f5d88 100644 --- a/examples/configs/recipes/llm/grpo-qwen2.5-7b-instruct-4n8g-fsdp2tp4sp.v3.yaml +++ b/examples/configs/recipes/llm/grpo-qwen2.5-7b-instruct-4n8g-fsdp2tp4sp.v3.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3.yaml b/examples/configs/recipes/llm/grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3.yaml index d3bbc266f2..78bfeee82d 100644 --- a/examples/configs/recipes/llm/grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3.yaml +++ b/examples/configs/recipes/llm/grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3.yaml @@ -109,6 +109,7 @@ logger: num_val_samples_to_print: 0 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp1-long.v2.yaml b/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp1-long.v2.yaml index 264d8d028d..a35a13533e 100644 --- a/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp1-long.v2.yaml +++ b/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp1-long.v2.yaml @@ -58,6 +58,7 @@ logger: log_dir: logs/sft-llama3.1-8b-instruct-1n8g-fsdp2tp1-long wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp2sp.v2.yaml b/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp2sp.v2.yaml index a1d77244c0..608edace8d 100644 --- a/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp2sp.v2.yaml +++ b/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-fsdp2tp2sp.v2.yaml @@ -58,6 +58,7 @@ logger: log_dir: logs/sft-llama3.1-8b-instruct-1n8g-fsdp2tp2sp wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-megatron.yaml b/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-megatron.yaml index 585ea7cafa..4fdfb5d37b 100644 --- a/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-megatron.yaml +++ b/examples/configs/recipes/llm/sft-llama3.1-8b-instruct-1n8g-megatron.yaml @@ -102,6 +102,7 @@ logger: log_dir: logs/sft-llama3.1-8b-instruct-1n8g-fsdp1 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/sft-llama3.2-1b-1n8g-fsdp2tp1.v2.yaml b/examples/configs/recipes/llm/sft-llama3.2-1b-1n8g-fsdp2tp1.v2.yaml index b958faeb66..8dac5cb980 100644 --- a/examples/configs/recipes/llm/sft-llama3.2-1b-1n8g-fsdp2tp1.v2.yaml +++ b/examples/configs/recipes/llm/sft-llama3.2-1b-1n8g-fsdp2tp1.v2.yaml @@ -50,7 +50,7 @@ policy: fused: false data: max_input_seq_length: 1024 - dataset_name: squad + dataset_name: squad add_bos: true add_eos: true add_generation_prompt: false @@ -58,6 +58,7 @@ logger: log_dir: logs/sft-llama3.2-1b-1n8g-fsdp2tp1 wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/recipes/llm/sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt.v2.yaml b/examples/configs/recipes/llm/sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt.v2.yaml index bb2e0e1572..bf38f37eb7 100644 --- a/examples/configs/recipes/llm/sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt.v2.yaml +++ b/examples/configs/recipes/llm/sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt.v2.yaml @@ -58,6 +58,7 @@ logger: log_dir: logs/sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt wandb_enabled: true tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true wandb: project: nemo-rl diff --git a/examples/configs/sft.yaml b/examples/configs/sft.yaml index 3839d455e2..9dc12d4487 100644 --- a/examples/configs/sft.yaml +++ b/examples/configs/sft.yaml @@ -133,6 +133,7 @@ logger: log_dir: "logs" # Base directory for all logs wandb_enabled: true # Make sure you do a ``wandb login [Your API key]'' before running tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: true # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "sft-dev" diff --git a/examples/configs/sft_openmathinstruct2.yaml b/examples/configs/sft_openmathinstruct2.yaml index 2040bdd5ff..08e38f89b9 100644 --- a/examples/configs/sft_openmathinstruct2.yaml +++ b/examples/configs/sft_openmathinstruct2.yaml @@ -71,6 +71,7 @@ logger: log_dir: "logs" # Base directory for all logs wandb_enabled: true # Make sure you do a ``wandb login [Your API key]'' before running tensorboard_enabled: true + mlflow_enabled: false monitor_gpus: false # If true, will monitor GPU usage and log to wandb and/or tensorboard wandb: project: "sft-dev" diff --git a/nemo_rl/utils/logger.py b/nemo_rl/utils/logger.py index 0a83204ee6..243447437f 100644 --- a/nemo_rl/utils/logger.py +++ b/nemo_rl/utils/logger.py @@ -19,11 +19,13 @@ import os import re import subprocess +import tempfile import threading import time from abc import ABC, abstractmethod -from typing import Any, Callable, Mapping, Optional, TypedDict +from typing import Any, Callable, Mapping, NotRequired, Optional, TypedDict +import mlflow import ray import requests import torch @@ -53,6 +55,12 @@ class TensorboardConfig(TypedDict): log_dir: str +class MLflowConfig(TypedDict): + experiment_name: str + run_name: str + tracking_uri: NotRequired[str] + + class GPUMonitoringConfig(TypedDict): collection_interval: int | float flush_interval: int | float @@ -62,8 +70,10 @@ class LoggerConfig(TypedDict): log_dir: str wandb_enabled: bool tensorboard_enabled: bool + mlflow_enabled: bool wandb: WandbConfig tensorboard: TensorboardConfig + mlflow: MLflowConfig monitor_gpus: bool gpu_monitoring: GPUMonitoringConfig @@ -535,7 +545,7 @@ def _collect(self, metrics: bool = False, sku: bool = False) -> dict[str, Any]: unique_metric_addresses[metrics_address] = True # Process each node's metrics - collected_metrics = {} + collected_metrics: dict[str, Any] = {} for node_idx, metric_address in enumerate(unique_metric_addresses): metrics = self._fetch_and_parse_metrics( node_idx, metric_address, parser_fn @@ -610,6 +620,91 @@ def flush(self) -> None: self.metrics_buffer = [] +class MLflowLogger(LoggerInterface): + """MLflow logger backend.""" + + def __init__(self, cfg: MLflowConfig, log_dir: Optional[str] = None): + """Initialize MLflow logger. + + Args: + cfg: MLflow configuration + log_dir: Optional log directory + """ + if cfg["tracking_uri"]: + mlflow.set_tracking_uri(cfg["tracking_uri"]) + + experiment = mlflow.get_experiment_by_name(cfg["experiment_name"]) + if experiment is None: + if log_dir: + mlflow.create_experiment( + name=cfg["experiment_name"], + artifact_location=log_dir, + ) + else: + mlflow.create_experiment(cfg["experiment_name"]) + else: + mlflow.set_experiment(cfg["experiment_name"]) + + # Start run + run_kwargs: dict[str, str] = {} + run_kwargs["run_name"] = cfg["run_name"] + + self.run = mlflow.start_run(**run_kwargs) + print( + f"Initialized MLflowLogger for experiment {cfg['experiment_name']}, " + f"run {cfg['run_name']}" + ) + + def log_metrics( + self, + metrics: dict[str, Any], + step: int, + prefix: Optional[str] = "", + step_metric: Optional[str] = None, + ) -> None: + """Log metrics to MLflow. + + Args: + metrics: Dict of metrics to log + step: Global step value + prefix: Optional prefix for metric names + step_metric: Optional step metric name (ignored in MLflow) + """ + for name, value in metrics.items(): + if prefix: + name = f"{prefix}/{name}" + mlflow.log_metric(name, value, step=step) + + def log_hyperparams(self, params: Mapping[str, Any]) -> None: + """Log hyperparameters to MLflow. + + Args: + params: Dictionary of hyperparameters to log + """ + # MLflow does not support nested dicts + mlflow.log_params(flatten_dict(params)) + + def log_plot(self, figure: plt.Figure, step: int, name: str) -> None: + """Log a plot to MLflow. + + Args: + figure: Matplotlib figure to log + step: Global step value + name: Name of the plot + """ + with tempfile.NamedTemporaryFile(suffix=".png", delete=True) as tmp_file: + figure.savefig(tmp_file.name, format="png", bbox_inches="tight") + mlflow.log_artifact(tmp_file.name, f"plots/{name}") + + def __del__(self) -> None: + """Clean up resources when the logger is destroyed.""" + try: + mlflow.end_run() + except Exception: + # Ignore errors during cleanup + pass + + class Logger(LoggerInterface): """Main logger class that delegates to multiple backend loggers.""" @@ -620,8 +715,10 @@ def __init__(self, cfg: LoggerConfig): cfg: Config dict with the following keys: - wandb_enabled - tensorboard_enabled + - mlflow_enabled - wandb - tensorboard + - mlflow - monitor_gpus - gpu_collection_interval - gpu_flush_interval @@ -646,6 +743,12 @@ def __init__(self, cfg: LoggerConfig): ) self.loggers.append(tensorboard_logger) + if cfg["mlflow_enabled"]: + mlflow_log_dir = os.path.join(self.base_log_dir, "mlflow") + os.makedirs(mlflow_log_dir, exist_ok=True) + mlflow_logger = MLflowLogger(cfg["mlflow"], log_dir=mlflow_log_dir) + self.loggers.append(mlflow_logger) + # Initialize GPU monitoring if requested self.gpu_monitor = None if cfg["monitor_gpus"]: @@ -764,13 +867,15 @@ def log_plot_token_mult_prob_error( return generation_logprob = generation_logprobs[ - sample_idx, generation_start_idx:generation_end_idx + sample_idx, int(generation_start_idx) : int(generation_end_idx) ] prev_logprob = ( - prev_logprobs[sample_idx, generation_start_idx:generation_end_idx] - * mask[sample_idx, generation_start_idx:generation_end_idx] + prev_logprobs[ + sample_idx, int(generation_start_idx) : int(generation_end_idx) + ] + * mask[sample_idx, int(generation_start_idx) : int(generation_end_idx)] ) - diff_i = diff[sample_idx, generation_start_idx:generation_end_idx] + diff_i = diff[sample_idx, int(generation_start_idx) : int(generation_end_idx)] # Find max absolute error token max_abs_error_idx = torch.argmax(diff_i).item() @@ -784,7 +889,7 @@ def log_plot_token_mult_prob_error( max_rel_error = relative_error[max_rel_error_idx].item() fig = plt.figure() - step_idx = torch.arange(generation_start_idx, generation_end_idx) + step_idx = torch.arange(int(generation_start_idx), int(generation_end_idx)) plt.plot(step_idx, generation_logprob, label="logprob (inference engine)") plt.plot(step_idx, prev_logprob, label="logprob (reference policy)") diff --git a/pyproject.toml b/pyproject.toml index a17cced3e8..5dce7857a3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -43,7 +43,8 @@ dependencies = [ "debugpy", "nvtx", "matplotlib", - "plotly" + "plotly", + "mlflow", ] [project.optional-dependencies] diff --git a/tests/unit/utils/test_logger.py b/tests/unit/utils/test_logger.py index c17cb3ae05..e61120f312 100644 --- a/tests/unit/utils/test_logger.py +++ b/tests/unit/utils/test_logger.py @@ -21,6 +21,7 @@ from nemo_rl.utils.logger import ( Logger, + MLflowLogger, RayGpuMonitorLogger, TensorboardLogger, WandbLogger, @@ -262,6 +263,154 @@ def test_log_hyperparams(self, mock_wandb): mock_run.config.update.assert_called_once_with(params) +class TestMLflowLogger: + """Test the MLflowLogger class.""" + + @pytest.fixture + def temp_dir(self): + """Create a temporary directory for logs.""" + temp_dir = tempfile.mkdtemp() + yield temp_dir + shutil.rmtree(temp_dir) + + @patch("nemo_rl.utils.logger.mlflow") + def test_init_basic_config(self, mock_mlflow, temp_dir): + """Test initialization of MLflowLogger with basic config.""" + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": None, + } + MLflowLogger(cfg, log_dir=temp_dir) + + mock_mlflow.set_experiment.assert_called_once_with("test-experiment") + mock_mlflow.start_run.assert_called_once_with(run_name="test-run") + + @patch("nemo_rl.utils.logger.mlflow") + def test_init_full_config(self, mock_mlflow, temp_dir): + """Test initialization of MLflowLogger with full config.""" + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": "http://localhost:5000", + } + MLflowLogger(cfg, log_dir=temp_dir) + + mock_mlflow.set_tracking_uri.assert_called_once_with("http://localhost:5000") + mock_mlflow.set_experiment.assert_called_once_with("test-experiment") + mock_mlflow.start_run.assert_called_once_with(run_name="test-run") + + @patch("nemo_rl.utils.logger.mlflow") + def test_log_metrics(self, mock_mlflow, temp_dir): + """Test logging metrics to MLflowLogger.""" + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": None, + } + logger = MLflowLogger(cfg, log_dir=temp_dir) + + metrics = {"loss": 0.5, "accuracy": 0.8} + step = 10 + logger.log_metrics(metrics, step) + + # Check that log_metric was called for each metric + assert mock_mlflow.log_metric.call_count == 2 + mock_mlflow.log_metric.assert_any_call("loss", 0.5, step=10) + mock_mlflow.log_metric.assert_any_call("accuracy", 0.8, step=10) + + @patch("nemo_rl.utils.logger.mlflow") + def test_log_metrics_with_prefix(self, mock_mlflow, temp_dir): + """Test logging metrics with a prefix to MLflowLogger.""" + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": None, + } + logger = MLflowLogger(cfg, log_dir=temp_dir) + + metrics = {"loss": 0.5, "accuracy": 0.8} + step = 10 + prefix = "train" + logger.log_metrics(metrics, step, prefix) + + # Check that log_metric was called for each metric with prefix + assert mock_mlflow.log_metric.call_count == 2 + mock_mlflow.log_metric.assert_any_call("train/loss", 0.5, step=10) + mock_mlflow.log_metric.assert_any_call("train/accuracy", 0.8, step=10) + + @patch("nemo_rl.utils.logger.mlflow") + def test_log_hyperparams(self, mock_mlflow, temp_dir): + """Test logging hyperparameters to MLflowLogger.""" + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": None, + } + logger = MLflowLogger(cfg, log_dir=temp_dir) + + params = {"lr": 0.001, "batch_size": 32, "model": {"hidden_size": 128}} + logger.log_hyperparams(params) + + # Check that log_params was called with flattened params + mock_mlflow.log_params.assert_called_once_with( + { + "lr": 0.001, + "batch_size": 32, + "model.hidden_size": 128, + } + ) + + @patch("nemo_rl.utils.logger.mlflow") + @patch("nemo_rl.utils.logger.plt") + @patch("nemo_rl.utils.logger.os") + def test_log_plot(self, mock_os, mock_plt, mock_mlflow, temp_dir): + """Test logging plots to MLflowLogger.""" + import tempfile + + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": None, + } + logger = MLflowLogger(cfg, log_dir=temp_dir) + + # Mock the figure + mock_figure = mock_plt.Figure.return_value + + # Mock tempfile.NamedTemporaryFile + mock_temp_file = type("MockTempFile", (), {"name": "/tmp/test.png"})() + with patch.object(tempfile, "NamedTemporaryFile") as mock_tempfile: + mock_tempfile.return_value.__enter__.return_value = mock_temp_file + mock_tempfile.return_value.__exit__.return_value = None + + logger.log_plot(mock_figure, step=10, name="test_plot") + + # Check that figure was saved and logged as artifact + mock_figure.savefig.assert_called_once_with( + "/tmp/test.png", format="png", bbox_inches="tight" + ) + mock_mlflow.log_artifact.assert_called_once_with( + "/tmp/test.png", "plots/test_plot" + ) + + @patch("nemo_rl.utils.logger.mlflow") + def test_cleanup(self, mock_mlflow, temp_dir): + """Test cleanup when logger is destroyed.""" + cfg = { + "experiment_name": "test-experiment", + "run_name": "test-run", + "tracking_uri": None, + } + logger = MLflowLogger(cfg, log_dir=temp_dir) + + # Trigger cleanup + logger.__del__() + + # Check that end_run was called + mock_mlflow.end_run.assert_called_once() + + class TestRayGpuMonitorLogger: """Test the RayGpuMonitorLogger class.""" @@ -768,6 +917,7 @@ def test_init_with_gpu_monitoring( cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": True, "gpu_monitoring": { "collection_interval": 15.0, @@ -813,6 +963,7 @@ def test_gpu_monitoring_without_wandb( cfg = { "wandb_enabled": False, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": True, "gpu_monitoring": { "collection_interval": 15.0, @@ -851,6 +1002,7 @@ def test_gpu_monitoring_no_main_loggers( cfg = { "wandb_enabled": False, "tensorboard_enabled": False, + "mlflow_enabled": False, "monitor_gpus": True, "gpu_monitoring": { "collection_interval": 15.0, @@ -902,6 +1054,7 @@ def test_init_no_loggers(self, mock_tb_logger, mock_wandb_logger, temp_dir): cfg = { "wandb_enabled": False, "tensorboard_enabled": False, + "mlflow_enabled": False, "monitor_gpus": False, "log_dir": temp_dir, } @@ -918,6 +1071,7 @@ def test_init_wandb_only(self, mock_tb_logger, mock_wandb_logger, temp_dir): cfg = { "wandb_enabled": True, "tensorboard_enabled": False, + "mlflow_enabled": False, "monitor_gpus": False, "wandb": {"project": "test-project"}, "log_dir": temp_dir, @@ -937,6 +1091,7 @@ def test_init_tensorboard_only(self, mock_tb_logger, mock_wandb_logger, temp_dir cfg = { "wandb_enabled": False, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": False, "tensorboard": {"log_dir": "test_logs"}, "log_dir": temp_dir, @@ -956,6 +1111,7 @@ def test_init_both_loggers(self, mock_tb_logger, mock_wandb_logger, temp_dir): cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": False, "wandb": {"project": "test-project"}, "tensorboard": {"log_dir": "test_logs"}, @@ -979,6 +1135,7 @@ def test_log_metrics(self, mock_tb_logger, mock_wandb_logger, temp_dir): cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": False, "wandb": {"project": "test-project"}, "tensorboard": {"log_dir": "test_logs"}, @@ -1005,6 +1162,7 @@ def test_log_hyperparams(self, mock_tb_logger, mock_wandb_logger, temp_dir): cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": False, "wandb": {"project": "test-project"}, "tensorboard": {"log_dir": "test_logs"}, @@ -1033,6 +1191,7 @@ def test_init_with_gpu_monitoring( cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": True, "gpu_monitoring": { "collection_interval": 15.0, @@ -1077,6 +1236,7 @@ def test_log_metrics_with_prefix_and_step_metric( cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": False, "wandb": {"project": "test-project"}, "tensorboard": {"log_dir": "test_logs"}, @@ -1114,6 +1274,7 @@ def test_log_plot_token_mult_prob_error( cfg = { "wandb_enabled": True, "tensorboard_enabled": True, + "mlflow_enabled": False, "monitor_gpus": False, "wandb": {"project": "test-project"}, "tensorboard": {"log_dir": "test_logs"}, @@ -1159,3 +1320,124 @@ def test_log_plot_token_mult_prob_error( legend_texts = [text.get_text() for text in ax.get_legend().get_texts()] assert any("Max abs error" in text for text in legend_texts) assert any("Max rel error (prob)" in text for text in legend_texts) + + @patch("nemo_rl.utils.logger.WandbLogger") + @patch("nemo_rl.utils.logger.TensorboardLogger") + def test_init_mlflow_only(self, mock_tb_logger, mock_wandb_logger, temp_dir): + """Test initialization with only MLflowLogger enabled.""" + cfg = { + "wandb_enabled": False, + "tensorboard_enabled": False, + "mlflow_enabled": True, + "monitor_gpus": False, + "mlflow": { + "experiment_name": "test-experiment", + "tracking_uri": None, + "run_name": "test-run", + }, + "log_dir": temp_dir, + } + logger = Logger(cfg) + + assert len(logger.loggers) == 1 + mock_wandb_logger.assert_not_called() + mock_tb_logger.assert_not_called() + + @patch("nemo_rl.utils.logger.WandbLogger") + @patch("nemo_rl.utils.logger.TensorboardLogger") + @patch("nemo_rl.utils.logger.MLflowLogger") + def test_init_all_loggers( + self, mock_mlflow_logger, mock_tb_logger, mock_wandb_logger, temp_dir + ): + """Test initialization with all loggers enabled.""" + cfg = { + "wandb_enabled": True, + "tensorboard_enabled": True, + "mlflow_enabled": True, + "monitor_gpus": False, + "wandb": {"project": "test-project"}, + "tensorboard": {"log_dir": "test_logs"}, + "mlflow": { + "experiment_name": "test-experiment", + "tracking_uri": None, + "run_name": "test-run", + }, + "log_dir": temp_dir, + } + logger = Logger(cfg) + + assert len(logger.loggers) == 3 + mock_wandb_logger.assert_called_once() + mock_tb_logger.assert_called_once() + mock_mlflow_logger.assert_called_once() + + @patch("nemo_rl.utils.logger.WandbLogger") + @patch("nemo_rl.utils.logger.TensorboardLogger") + @patch("nemo_rl.utils.logger.MLflowLogger") + def test_log_metrics_with_mlflow( + self, mock_mlflow_logger, mock_tb_logger, mock_wandb_logger, temp_dir + ): + """Test logging metrics to all enabled loggers including MLflow.""" + cfg = { + "wandb_enabled": True, + "tensorboard_enabled": True, + "mlflow_enabled": True, + "monitor_gpus": False, + "wandb": {"project": "test-project"}, + "tensorboard": {"log_dir": "test_logs"}, + "mlflow": { + "experiment_name": "test-experiment", + "tracking_uri": None, + "run_name": "test-run", + }, + "log_dir": temp_dir, + } + logger = Logger(cfg) + + # Create mock logger instances + mock_wandb_instance = mock_wandb_logger.return_value + mock_tb_instance = mock_tb_logger.return_value + mock_mlflow_instance = mock_mlflow_logger.return_value + + metrics = {"loss": 0.5, "accuracy": 0.8} + step = 10 + logger.log_metrics(metrics, step) + + # Check that log_metrics was called on all loggers + mock_wandb_instance.log_metrics.assert_called_once_with(metrics, step, "", None) + mock_tb_instance.log_metrics.assert_called_once_with(metrics, step, "", None) + mock_mlflow_instance.log_metrics.assert_called_once_with( + metrics, step, "", None + ) + + @patch("nemo_rl.utils.logger.WandbLogger") + @patch("nemo_rl.utils.logger.TensorboardLogger") + @patch("nemo_rl.utils.logger.MLflowLogger") + def test_log_hyperparams_with_mlflow( + self, mock_mlflow_logger, mock_tb_logger, mock_wandb_logger, temp_dir + ): + """Test logging hyperparameters to all enabled loggers including MLflow.""" + cfg = { + "wandb_enabled": True, + "tensorboard_enabled": True, + "mlflow_enabled": True, + "monitor_gpus": False, + "wandb": {"project": "test-project"}, + "tensorboard": {"log_dir": "test_logs"}, + "mlflow": {"experiment_name": "test-experiment"}, + "log_dir": temp_dir, + } + logger = Logger(cfg) + + # Create mock logger instances + mock_wandb_instance = mock_wandb_logger.return_value + mock_tb_instance = mock_tb_logger.return_value + mock_mlflow_instance = mock_mlflow_logger.return_value + + params = {"lr": 0.001, "batch_size": 32} + logger.log_hyperparams(params) + + # Check that log_hyperparams was called on all loggers + mock_wandb_instance.log_hyperparams.assert_called_once_with(params) + mock_tb_instance.log_hyperparams.assert_called_once_with(params) + mock_mlflow_instance.log_hyperparams.assert_called_once_with(params) diff --git a/uv.lock b/uv.lock index fdce004e44..30abb6a7d3 100644 --- a/uv.lock +++ b/uv.lock @@ -165,6 +165,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7e/b3/6b4067be973ae96ba0d615946e314c5ae35f9f993eca561b356540bb0c2b/alabaster-1.0.0-py3-none-any.whl", hash = "sha256:fc6786402dc3fcb2de3cabd5fe455a2db534b371124f1f21de8731783dec828b", size = 13929, upload-time = "2024-07-26T18:15:02.05Z" }, ] +[[package]] +name = "alembic" +version = "1.16.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mako" }, + { name = "sqlalchemy" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/52/72e791b75c6b1efa803e491f7cbab78e963695e76d4ada05385252927e76/alembic-1.16.4.tar.gz", hash = "sha256:efab6ada0dd0fae2c92060800e0bf5c1dc26af15a10e02fb4babff164b4725e2", size = 1968161, upload-time = "2025-07-10T16:17:20.192Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/62/96b5217b742805236614f05904541000f55422a6060a90d7fd4ce26c172d/alembic-1.16.4-py3-none-any.whl", hash = "sha256:b05e51e8e82efc1abd14ba2af6392897e145930c3e0a2faf2b0da2f7f7fd660d", size = 247026, upload-time = "2025-07-10T16:17:21.845Z" }, +] + [[package]] name = "aniso8601" version = "10.0.1" @@ -732,6 +746,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" }, ] +[[package]] +name = "databricks-sdk" +version = "0.58.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "google-auth" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b7/64/58d7e0d5d24da115980751d3e33356f3dcd9f091141fffcadd24502a5664/databricks_sdk-0.58.0.tar.gz", hash = "sha256:9be3985f1297c5e605103e0df3aa5a2ee6fda56e9d83721f376af7c6855bcb90", size = 783365, upload-time = "2025-07-09T08:36:04.21Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3c/55/035b52fa6e6cad22ef7091ea21135a11ce80fdc3a22f90de58d91cfc588e/databricks_sdk-0.58.0-py3-none-any.whl", hash = "sha256:8f5a35e42cc3f8a065d2e7c2d1aa5fb2fbaf3f5f0b475b75f5af17234f02abbd", size = 741412, upload-time = "2025-07-09T08:36:02.297Z" }, +] + [[package]] name = "datasets" version = "3.6.0" @@ -1242,6 +1269,42 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/86/f1/62a193f0227cf15a920390abe675f386dec35f7ae3ffe6da582d3ade42c7/googleapis_common_protos-1.70.0-py3-none-any.whl", hash = "sha256:b8bfcca8c25a2bb253e0e0b0adaf8c00773e5e6af6fd92397576680b807e0fd8", size = 294530, upload-time = "2025-04-14T10:17:01.271Z" }, ] +[[package]] +name = "graphene" +version = "3.4.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "graphql-core" }, + { name = "graphql-relay" }, + { name = "python-dateutil" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cc/f6/bf62ff950c317ed03e77f3f6ddd7e34aaa98fe89d79ebd660c55343d8054/graphene-3.4.3.tar.gz", hash = "sha256:2a3786948ce75fe7e078443d37f609cbe5bb36ad8d6b828740ad3b95ed1a0aaa", size = 44739, upload-time = "2024-11-09T20:44:25.757Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/66/e0/61d8e98007182e6b2aca7cf65904721fb2e4bce0192272ab9cb6f69d8812/graphene-3.4.3-py2.py3-none-any.whl", hash = "sha256:820db6289754c181007a150db1f7fff544b94142b556d12e3ebc777a7bf36c71", size = 114894, upload-time = "2024-11-09T20:44:23.851Z" }, +] + +[[package]] +name = "graphql-core" +version = "3.2.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c4/16/7574029da84834349b60ed71614d66ca3afe46e9bf9c7b9562102acb7d4f/graphql_core-3.2.6.tar.gz", hash = "sha256:c08eec22f9e40f0bd61d805907e3b3b1b9a320bc606e23dc145eebca07c8fbab", size = 505353, upload-time = "2025-01-26T16:36:27.374Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ae/4f/7297663840621022bc73c22d7d9d80dbc78b4db6297f764b545cd5dd462d/graphql_core-3.2.6-py3-none-any.whl", hash = "sha256:78b016718c161a6fb20a7d97bbf107f331cd1afe53e45566c59f776ed7f0b45f", size = 203416, upload-time = "2025-01-26T16:36:24.868Z" }, +] + +[[package]] +name = "graphql-relay" +version = "3.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "graphql-core" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d1/13/98fbf8d67552f102488ffc16c6f559ce71ea15f6294728d33928ab5ff14d/graphql-relay-3.2.0.tar.gz", hash = "sha256:1ff1c51298356e481a0be009ccdff249832ce53f30559c1338f22a0e0d17250c", size = 50027, upload-time = "2022-04-16T11:03:45.447Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/74/16/a4cf06adbc711bd364a73ce043b0b08d8fa5aae3df11b6ee4248bcdad2e0/graphql_relay-3.2.0-py3-none-any.whl", hash = "sha256:c9b22bd28b170ba1fe674c74384a8ff30a76c8e26f88ac3aa1584dd3179953e5", size = 16940, upload-time = "2022-04-16T11:03:43.895Z" }, +] + [[package]] name = "graphviz" version = "0.21" @@ -1251,6 +1314,39 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/91/4c/e0ce1ef95d4000ebc1c11801f9b944fa5910ecc15b5e351865763d8657f8/graphviz-0.21-py3-none-any.whl", hash = "sha256:54f33de9f4f911d7e84e4191749cac8cc5653f815b06738c54db9a15ab8b1e42", size = 47300, upload-time = "2025-06-15T09:35:04.433Z" }, ] +[[package]] +name = "greenlet" +version = "3.2.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/92/bb85bd6e80148a4d2e0c59f7c0c2891029f8fd510183afc7d8d2feeed9b6/greenlet-3.2.3.tar.gz", hash = "sha256:8b0dd8ae4c0d6f5e54ee55ba935eeb3d735a9b58a8a1e5b5cbab64e01a39f365", size = 185752, upload-time = "2025-06-05T16:16:09.955Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f3/94/ad0d435f7c48debe960c53b8f60fb41c2026b1d0fa4a99a1cb17c3461e09/greenlet-3.2.3-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:25ad29caed5783d4bd7a85c9251c651696164622494c00802a139c00d639242d", size = 271992, upload-time = "2025-06-05T16:11:23.467Z" }, + { url = "https://files.pythonhosted.org/packages/93/5d/7c27cf4d003d6e77749d299c7c8f5fd50b4f251647b5c2e97e1f20da0ab5/greenlet-3.2.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:88cd97bf37fe24a6710ec6a3a7799f3f81d9cd33317dcf565ff9950c83f55e0b", size = 638820, upload-time = "2025-06-05T16:38:52.882Z" }, + { url = "https://files.pythonhosted.org/packages/c6/7e/807e1e9be07a125bb4c169144937910bf59b9d2f6d931578e57f0bce0ae2/greenlet-3.2.3-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:baeedccca94880d2f5666b4fa16fc20ef50ba1ee353ee2d7092b383a243b0b0d", size = 653046, upload-time = "2025-06-05T16:41:36.343Z" }, + { url = "https://files.pythonhosted.org/packages/9d/ab/158c1a4ea1068bdbc78dba5a3de57e4c7aeb4e7fa034320ea94c688bfb61/greenlet-3.2.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:be52af4b6292baecfa0f397f3edb3c6092ce071b499dd6fe292c9ac9f2c8f264", size = 647701, upload-time = "2025-06-05T16:48:19.604Z" }, + { url = "https://files.pythonhosted.org/packages/cc/0d/93729068259b550d6a0288da4ff72b86ed05626eaf1eb7c0d3466a2571de/greenlet-3.2.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0cc73378150b8b78b0c9fe2ce56e166695e67478550769536a6742dca3651688", size = 649747, upload-time = "2025-06-05T16:13:04.628Z" }, + { url = "https://files.pythonhosted.org/packages/f6/f6/c82ac1851c60851302d8581680573245c8fc300253fc1ff741ae74a6c24d/greenlet-3.2.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:706d016a03e78df129f68c4c9b4c4f963f7d73534e48a24f5f5a7101ed13dbbb", size = 605461, upload-time = "2025-06-05T16:12:50.792Z" }, + { url = "https://files.pythonhosted.org/packages/98/82/d022cf25ca39cf1200650fc58c52af32c90f80479c25d1cbf57980ec3065/greenlet-3.2.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:419e60f80709510c343c57b4bb5a339d8767bf9aef9b8ce43f4f143240f88b7c", size = 1121190, upload-time = "2025-06-05T16:36:48.59Z" }, + { url = "https://files.pythonhosted.org/packages/f5/e1/25297f70717abe8104c20ecf7af0a5b82d2f5a980eb1ac79f65654799f9f/greenlet-3.2.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:93d48533fade144203816783373f27a97e4193177ebaaf0fc396db19e5d61163", size = 1149055, upload-time = "2025-06-05T16:12:40.457Z" }, + { url = "https://files.pythonhosted.org/packages/1f/8f/8f9e56c5e82eb2c26e8cde787962e66494312dc8cb261c460e1f3a9c88bc/greenlet-3.2.3-cp312-cp312-win_amd64.whl", hash = "sha256:7454d37c740bb27bdeddfc3f358f26956a07d5220818ceb467a483197d84f849", size = 297817, upload-time = "2025-06-05T16:29:49.244Z" }, + { url = "https://files.pythonhosted.org/packages/b1/cf/f5c0b23309070ae93de75c90d29300751a5aacefc0a3ed1b1d8edb28f08b/greenlet-3.2.3-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:500b8689aa9dd1ab26872a34084503aeddefcb438e2e7317b89b11eaea1901ad", size = 270732, upload-time = "2025-06-05T16:10:08.26Z" }, + { url = "https://files.pythonhosted.org/packages/48/ae/91a957ba60482d3fecf9be49bc3948f341d706b52ddb9d83a70d42abd498/greenlet-3.2.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a07d3472c2a93117af3b0136f246b2833fdc0b542d4a9799ae5f41c28323faef", size = 639033, upload-time = "2025-06-05T16:38:53.983Z" }, + { url = "https://files.pythonhosted.org/packages/6f/df/20ffa66dd5a7a7beffa6451bdb7400d66251374ab40b99981478c69a67a8/greenlet-3.2.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:8704b3768d2f51150626962f4b9a9e4a17d2e37c8a8d9867bbd9fa4eb938d3b3", size = 652999, upload-time = "2025-06-05T16:41:37.89Z" }, + { url = "https://files.pythonhosted.org/packages/51/b4/ebb2c8cb41e521f1d72bf0465f2f9a2fd803f674a88db228887e6847077e/greenlet-3.2.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:5035d77a27b7c62db6cf41cf786cfe2242644a7a337a0e155c80960598baab95", size = 647368, upload-time = "2025-06-05T16:48:21.467Z" }, + { url = "https://files.pythonhosted.org/packages/8e/6a/1e1b5aa10dced4ae876a322155705257748108b7fd2e4fae3f2a091fe81a/greenlet-3.2.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2d8aa5423cd4a396792f6d4580f88bdc6efcb9205891c9d40d20f6e670992efb", size = 650037, upload-time = "2025-06-05T16:13:06.402Z" }, + { url = "https://files.pythonhosted.org/packages/26/f2/ad51331a157c7015c675702e2d5230c243695c788f8f75feba1af32b3617/greenlet-3.2.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2c724620a101f8170065d7dded3f962a2aea7a7dae133a009cada42847e04a7b", size = 608402, upload-time = "2025-06-05T16:12:51.91Z" }, + { url = "https://files.pythonhosted.org/packages/26/bc/862bd2083e6b3aff23300900a956f4ea9a4059de337f5c8734346b9b34fc/greenlet-3.2.3-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:873abe55f134c48e1f2a6f53f7d1419192a3d1a4e873bace00499a4e45ea6af0", size = 1119577, upload-time = "2025-06-05T16:36:49.787Z" }, + { url = "https://files.pythonhosted.org/packages/86/94/1fc0cc068cfde885170e01de40a619b00eaa8f2916bf3541744730ffb4c3/greenlet-3.2.3-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:024571bbce5f2c1cfff08bf3fbaa43bbc7444f580ae13b0099e95d0e6e67ed36", size = 1147121, upload-time = "2025-06-05T16:12:42.527Z" }, + { url = "https://files.pythonhosted.org/packages/27/1a/199f9587e8cb08a0658f9c30f3799244307614148ffe8b1e3aa22f324dea/greenlet-3.2.3-cp313-cp313-win_amd64.whl", hash = "sha256:5195fb1e75e592dd04ce79881c8a22becdfa3e6f500e7feb059b1e6fdd54d3e3", size = 297603, upload-time = "2025-06-05T16:20:12.651Z" }, + { url = "https://files.pythonhosted.org/packages/d8/ca/accd7aa5280eb92b70ed9e8f7fd79dc50a2c21d8c73b9a0856f5b564e222/greenlet-3.2.3-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:3d04332dddb10b4a211b68111dabaee2e1a073663d117dc10247b5b1642bac86", size = 271479, upload-time = "2025-06-05T16:10:47.525Z" }, + { url = "https://files.pythonhosted.org/packages/55/71/01ed9895d9eb49223280ecc98a557585edfa56b3d0e965b9fa9f7f06b6d9/greenlet-3.2.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8186162dffde068a465deab08fc72c767196895c39db26ab1c17c0b77a6d8b97", size = 683952, upload-time = "2025-06-05T16:38:55.125Z" }, + { url = "https://files.pythonhosted.org/packages/ea/61/638c4bdf460c3c678a0a1ef4c200f347dff80719597e53b5edb2fb27ab54/greenlet-3.2.3-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f4bfbaa6096b1b7a200024784217defedf46a07c2eee1a498e94a1b5f8ec5728", size = 696917, upload-time = "2025-06-05T16:41:38.959Z" }, + { url = "https://files.pythonhosted.org/packages/22/cc/0bd1a7eb759d1f3e3cc2d1bc0f0b487ad3cc9f34d74da4b80f226fde4ec3/greenlet-3.2.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:ed6cfa9200484d234d8394c70f5492f144b20d4533f69262d530a1a082f6ee9a", size = 692443, upload-time = "2025-06-05T16:48:23.113Z" }, + { url = "https://files.pythonhosted.org/packages/67/10/b2a4b63d3f08362662e89c103f7fe28894a51ae0bc890fabf37d1d780e52/greenlet-3.2.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:02b0df6f63cd15012bed5401b47829cfd2e97052dc89da3cfaf2c779124eb892", size = 692995, upload-time = "2025-06-05T16:13:07.972Z" }, + { url = "https://files.pythonhosted.org/packages/5a/c6/ad82f148a4e3ce9564056453a71529732baf5448ad53fc323e37efe34f66/greenlet-3.2.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:86c2d68e87107c1792e2e8d5399acec2487a4e993ab76c792408e59394d52141", size = 655320, upload-time = "2025-06-05T16:12:53.453Z" }, + { url = "https://files.pythonhosted.org/packages/5c/4f/aab73ecaa6b3086a4c89863d94cf26fa84cbff63f52ce9bc4342b3087a06/greenlet-3.2.3-cp314-cp314-win_amd64.whl", hash = "sha256:8c47aae8fbbfcf82cc13327ae802ba13c9c36753b67e760023fd116bc124a62a", size = 301236, upload-time = "2025-06-05T16:15:20.111Z" }, +] + [[package]] name = "grpcio" version = "1.73.0" @@ -1279,6 +1375,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d7/35/347db7d2e7674b621afd21b12022e7f48c7b0861b5577134b4e939536141/grpcio-1.73.0-cp313-cp313-win_amd64.whl", hash = "sha256:38cf518cc54cd0c47c9539cefa8888549fcc067db0b0c66a46535ca8032020c4", size = 4335872, upload-time = "2025-06-09T10:04:29.032Z" }, ] +[[package]] +name = "gunicorn" +version = "23.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "packaging" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/34/72/9614c465dc206155d93eff0ca20d42e1e35afc533971379482de953521a4/gunicorn-23.0.0.tar.gz", hash = "sha256:f014447a0101dc57e294f6c18ca6b40227a4c90e9bdb586042628030cba004ec", size = 375031, upload-time = "2024-08-10T20:25:27.378Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/7d/6dac2a6e1eba33ee43f318edbed4ff29151a49b5d37f080aad1e6469bca4/gunicorn-23.0.0-py3-none-any.whl", hash = "sha256:ec400d38950de4dfd418cff8328b2c8faed0edb0d517d3394e457c317908ca4d", size = 85029, upload-time = "2024-08-10T20:25:24.996Z" }, +] + [[package]] name = "h11" version = "0.16.0" @@ -1889,6 +1997,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fc/14/c115516c62a7d2499781d2d3d7215218c0731b2c940753bf9f9b7b73924d/lxml-5.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:bcb7a1096b4b6b24ce1ac24d4942ad98f983cd3810f9711bcd0293f43a9d8b9f", size = 3814606, upload-time = "2025-04-23T01:47:39.028Z" }, ] +[[package]] +name = "mako" +version = "1.3.10" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9e/38/bd5b78a920a64d708fe6bc8e0a2c075e1389d53bef8413725c63ba041535/mako-1.3.10.tar.gz", hash = "sha256:99579a6f39583fa7e5630a28c3c1f440e4e97a414b80372649c0ce338da2ea28", size = 392474, upload-time = "2025-04-10T12:44:31.16Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/87/fb/99f81ac72ae23375f22b7afdb7642aba97c00a713c217124420147681a2f/mako-1.3.10-py3-none-any.whl", hash = "sha256:baef24a52fc4fc514a0887ac600f9f1cff3d82c61d4d700a1fa84d597b88db59", size = 78509, upload-time = "2025-04-10T12:50:53.297Z" }, +] + [[package]] name = "markdown" version = "3.8.2" @@ -2106,6 +2226,59 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/45/c1a1ccfdd02bc4173ca0f4a2d327683a27df85797b885eb1da1ca325b85c/ml_dtypes-0.5.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d13755f8e8445b3870114e5b6240facaa7cb0c3361e54beba3e07fa912a6e12b", size = 5052731, upload-time = "2025-01-07T03:34:45.308Z" }, ] +[[package]] +name = "mlflow" +version = "3.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "alembic" }, + { name = "docker" }, + { name = "flask" }, + { name = "graphene" }, + { name = "gunicorn", marker = "sys_platform != 'win32'" }, + { name = "matplotlib" }, + { name = "mlflow-skinny" }, + { name = "numpy" }, + { name = "pandas" }, + { name = "pyarrow" }, + { name = "scikit-learn" }, + { name = "scipy" }, + { name = "sqlalchemy" }, + { name = "waitress", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2b/e1/0cba7a8fc2c81078b4d31948f65fb1580cee1831e955a86028159724d057/mlflow-3.1.1.tar.gz", hash = "sha256:ee98fe929d61625b72ae5010fbf12a7c6d15527790397827191fd6e8246c33e5", size = 24098836, upload-time = "2025-06-25T09:12:56.416Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/07/9f28e7e2b1c9552e64e6161cd3943b02349f8164176cea6b75e69d7df94a/mlflow-3.1.1-py3-none-any.whl", hash = "sha256:16853335292217fde203a645fd50f38d5567ce7818587ed5236040418918872e", size = 24673365, upload-time = "2025-06-25T09:12:53.482Z" }, +] + +[[package]] +name = "mlflow-skinny" +version = "3.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cachetools" }, + { name = "click" }, + { name = "cloudpickle" }, + { name = "databricks-sdk" }, + { name = "fastapi" }, + { name = "gitpython" }, + { name = "importlib-metadata" }, + { name = "opentelemetry-api" }, + { name = "opentelemetry-sdk" }, + { name = "packaging" }, + { name = "protobuf" }, + { name = "pydantic" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "sqlparse" }, + { name = "typing-extensions" }, + { name = "uvicorn" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/dd/52/e63c0244a24ed23b5f82b30efffce150c19f126b8ef977b78a56f6d192c9/mlflow_skinny-3.1.1.tar.gz", hash = "sha256:9c2ea510eef6c115c7241305b65f7090d7fdc02399de2a6e8ddae5f285bb7a99", size = 1603411, upload-time = "2025-06-25T05:52:22.717Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f6/45/24d553e0f550f82aaadd8b9d08f1410a3d750c51733a5f43fcc6def1be00/mlflow_skinny-3.1.1-py3-none-any.whl", hash = "sha256:73b1be5d0ef3099c2d0e5ec3ca7fd0b85d4a6def7d7ab35feda9f06bf8bf7049", size = 1926660, upload-time = "2025-06-25T05:52:20.556Z" }, +] + [[package]] name = "mpmath" version = "1.3.0" @@ -2311,6 +2484,7 @@ dependencies = [ { name = "hydra-core" }, { name = "math-verify" }, { name = "matplotlib" }, + { name = "mlflow" }, { name = "ninja" }, { name = "numpy" }, { name = "nvidia-ml-py" }, @@ -2390,6 +2564,7 @@ requires-dist = [ { name = "math-verify" }, { name = "matplotlib" }, { name = "megatron-core", marker = "extra == 'mcore'", editable = "3rdparty/Megatron-LM-workspace" }, + { name = "mlflow" }, { name = "nemo-tron", marker = "extra == 'mcore'", editable = "3rdparty/NeMo-workspace" }, { name = "ninja" }, { name = "numpy" }, @@ -4527,6 +4702,44 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/52/a7/d2782e4e3f77c8450f727ba74a8f12756d5ba823d81b941f1b04da9d033a/sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl", hash = "sha256:6e2cb0eef194e10c27ec0023bfeb25badbbb5868244cf5bc5bdc04e4464bf331", size = 92072, upload-time = "2024-07-29T01:10:08.203Z" }, ] +[[package]] +name = "sqlalchemy" +version = "2.0.41" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "greenlet", marker = "(python_full_version < '3.14' and platform_machine == 'AMD64') or (python_full_version < '3.14' and platform_machine == 'WIN32') or (python_full_version < '3.14' and platform_machine == 'aarch64') or (python_full_version < '3.14' and platform_machine == 'amd64') or (python_full_version < '3.14' and platform_machine == 'ppc64le') or (python_full_version < '3.14' and platform_machine == 'win32') or (python_full_version < '3.14' and platform_machine == 'x86_64')" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/63/66/45b165c595ec89aa7dcc2c1cd222ab269bc753f1fc7a1e68f8481bd957bf/sqlalchemy-2.0.41.tar.gz", hash = "sha256:edba70118c4be3c2b1f90754d308d0b79c6fe2c0fdc52d8ddf603916f83f4db9", size = 9689424, upload-time = "2025-05-14T17:10:32.339Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/2a/f1f4e068b371154740dd10fb81afb5240d5af4aa0087b88d8b308b5429c2/sqlalchemy-2.0.41-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:81f413674d85cfd0dfcd6512e10e0f33c19c21860342a4890c3a2b59479929f9", size = 2119645, upload-time = "2025-05-14T17:55:24.854Z" }, + { url = "https://files.pythonhosted.org/packages/9b/e8/c664a7e73d36fbfc4730f8cf2bf930444ea87270f2825efbe17bf808b998/sqlalchemy-2.0.41-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:598d9ebc1e796431bbd068e41e4de4dc34312b7aa3292571bb3674a0cb415dd1", size = 2107399, upload-time = "2025-05-14T17:55:28.097Z" }, + { url = "https://files.pythonhosted.org/packages/5c/78/8a9cf6c5e7135540cb682128d091d6afa1b9e48bd049b0d691bf54114f70/sqlalchemy-2.0.41-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a104c5694dfd2d864a6f91b0956eb5d5883234119cb40010115fd45a16da5e70", size = 3293269, upload-time = "2025-05-14T17:50:38.227Z" }, + { url = "https://files.pythonhosted.org/packages/3c/35/f74add3978c20de6323fb11cb5162702670cc7a9420033befb43d8d5b7a4/sqlalchemy-2.0.41-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6145afea51ff0af7f2564a05fa95eb46f542919e6523729663a5d285ecb3cf5e", size = 3303364, upload-time = "2025-05-14T17:51:49.829Z" }, + { url = "https://files.pythonhosted.org/packages/6a/d4/c990f37f52c3f7748ebe98883e2a0f7d038108c2c5a82468d1ff3eec50b7/sqlalchemy-2.0.41-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b46fa6eae1cd1c20e6e6f44e19984d438b6b2d8616d21d783d150df714f44078", size = 3229072, upload-time = "2025-05-14T17:50:39.774Z" }, + { url = "https://files.pythonhosted.org/packages/15/69/cab11fecc7eb64bc561011be2bd03d065b762d87add52a4ca0aca2e12904/sqlalchemy-2.0.41-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41836fe661cc98abfae476e14ba1906220f92c4e528771a8a3ae6a151242d2ae", size = 3268074, upload-time = "2025-05-14T17:51:51.736Z" }, + { url = "https://files.pythonhosted.org/packages/5c/ca/0c19ec16858585d37767b167fc9602593f98998a68a798450558239fb04a/sqlalchemy-2.0.41-cp312-cp312-win32.whl", hash = "sha256:a8808d5cf866c781150d36a3c8eb3adccfa41a8105d031bf27e92c251e3969d6", size = 2084514, upload-time = "2025-05-14T17:55:49.915Z" }, + { url = "https://files.pythonhosted.org/packages/7f/23/4c2833d78ff3010a4e17f984c734f52b531a8c9060a50429c9d4b0211be6/sqlalchemy-2.0.41-cp312-cp312-win_amd64.whl", hash = "sha256:5b14e97886199c1f52c14629c11d90c11fbb09e9334fa7bb5f6d068d9ced0ce0", size = 2111557, upload-time = "2025-05-14T17:55:51.349Z" }, + { url = "https://files.pythonhosted.org/packages/d3/ad/2e1c6d4f235a97eeef52d0200d8ddda16f6c4dd70ae5ad88c46963440480/sqlalchemy-2.0.41-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4eeb195cdedaf17aab6b247894ff2734dcead6c08f748e617bfe05bd5a218443", size = 2115491, upload-time = "2025-05-14T17:55:31.177Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8d/be490e5db8400dacc89056f78a52d44b04fbf75e8439569d5b879623a53b/sqlalchemy-2.0.41-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d4ae769b9c1c7757e4ccce94b0641bc203bbdf43ba7a2413ab2523d8d047d8dc", size = 2102827, upload-time = "2025-05-14T17:55:34.921Z" }, + { url = "https://files.pythonhosted.org/packages/a0/72/c97ad430f0b0e78efaf2791342e13ffeafcbb3c06242f01a3bb8fe44f65d/sqlalchemy-2.0.41-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a62448526dd9ed3e3beedc93df9bb6b55a436ed1474db31a2af13b313a70a7e1", size = 3225224, upload-time = "2025-05-14T17:50:41.418Z" }, + { url = "https://files.pythonhosted.org/packages/5e/51/5ba9ea3246ea068630acf35a6ba0d181e99f1af1afd17e159eac7e8bc2b8/sqlalchemy-2.0.41-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dc56c9788617b8964ad02e8fcfeed4001c1f8ba91a9e1f31483c0dffb207002a", size = 3230045, upload-time = "2025-05-14T17:51:54.722Z" }, + { url = "https://files.pythonhosted.org/packages/78/2f/8c14443b2acea700c62f9b4a8bad9e49fc1b65cfb260edead71fd38e9f19/sqlalchemy-2.0.41-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:c153265408d18de4cc5ded1941dcd8315894572cddd3c58df5d5b5705b3fa28d", size = 3159357, upload-time = "2025-05-14T17:50:43.483Z" }, + { url = "https://files.pythonhosted.org/packages/fc/b2/43eacbf6ccc5276d76cea18cb7c3d73e294d6fb21f9ff8b4eef9b42bbfd5/sqlalchemy-2.0.41-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f67766965996e63bb46cfbf2ce5355fc32d9dd3b8ad7e536a920ff9ee422e23", size = 3197511, upload-time = "2025-05-14T17:51:57.308Z" }, + { url = "https://files.pythonhosted.org/packages/fa/2e/677c17c5d6a004c3c45334ab1dbe7b7deb834430b282b8a0f75ae220c8eb/sqlalchemy-2.0.41-cp313-cp313-win32.whl", hash = "sha256:bfc9064f6658a3d1cadeaa0ba07570b83ce6801a1314985bf98ec9b95d74e15f", size = 2082420, upload-time = "2025-05-14T17:55:52.69Z" }, + { url = "https://files.pythonhosted.org/packages/e9/61/e8c1b9b6307c57157d328dd8b8348ddc4c47ffdf1279365a13b2b98b8049/sqlalchemy-2.0.41-cp313-cp313-win_amd64.whl", hash = "sha256:82ca366a844eb551daff9d2e6e7a9e5e76d2612c8564f58db6c19a726869c1df", size = 2108329, upload-time = "2025-05-14T17:55:54.495Z" }, + { url = "https://files.pythonhosted.org/packages/1c/fc/9ba22f01b5cdacc8f5ed0d22304718d2c758fce3fd49a5372b886a86f37c/sqlalchemy-2.0.41-py3-none-any.whl", hash = "sha256:57df5dc6fdb5ed1a88a1ed2195fd31927e705cad62dedd86b46972752a80f576", size = 1911224, upload-time = "2025-05-14T17:39:42.154Z" }, +] + +[[package]] +name = "sqlparse" +version = "0.5.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e5/40/edede8dd6977b0d3da179a342c198ed100dd2aba4be081861ee5911e4da4/sqlparse-0.5.3.tar.gz", hash = "sha256:09f67787f56a0b16ecdbde1bfc7f5d9c3371ca683cfeaa8e6ff60b4807ec9272", size = 84999, upload-time = "2024-12-10T12:05:30.728Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a9/5c/bfd6bd0bf979426d405cc6e71eceb8701b148b16c21d2dc3c261efc61c7b/sqlparse-0.5.3-py3-none-any.whl", hash = "sha256:cf2196ed3418f3ba5de6af7e82c694a9fbdbfecccdfc72e281548517081f16ca", size = 44415, upload-time = "2024-12-10T12:05:27.824Z" }, +] + [[package]] name = "starlette" version = "0.46.2" @@ -5145,6 +5358,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cd/08/cb88fd52c08df57ccc4f722241150643d521b3174f8d0c3a1ec5549c3927/vllm-0.9.0-cp38-abi3-manylinux1_x86_64.whl", hash = "sha256:a130715cc915377f78e84088fc35c426266e278a0793be0b2ad78deda2e2f55e", size = 377192911, upload-time = "2025-05-28T01:30:28.547Z" }, ] +[[package]] +name = "waitress" +version = "3.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/cb/04ddb054f45faa306a230769e868c28b8065ea196891f09004ebace5b184/waitress-3.0.2.tar.gz", hash = "sha256:682aaaf2af0c44ada4abfb70ded36393f0e307f4ab9456a215ce0020baefc31f", size = 179901, upload-time = "2024-11-16T20:02:35.195Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8d/57/a27182528c90ef38d82b636a11f606b0cbb0e17588ed205435f8affe3368/waitress-3.0.2-py3-none-any.whl", hash = "sha256:c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e", size = 56232, upload-time = "2024-11-16T20:02:33.858Z" }, +] + [[package]] name = "wandb" version = "0.20.1"