diff --git a/.github/workflows/cicd-main.yml b/.github/workflows/cicd-main.yml index ffa909a22a..f2edb2bdf9 100644 --- a/.github/workflows/cicd-main.yml +++ b/.github/workflows/cicd-main.yml @@ -111,6 +111,7 @@ jobs: timeout: 40 - script: L2_Launch_recipes_llama_1b - script: L2_Launch_recipes_llama_3b + - script: L2_Launch_recipes_llama_distill - script: L2_Launch_recipes_mamba - script: L2_Launch_recipes_qwen - script: L2_Launch_data diff --git a/docs/index.md b/docs/index.md index cf9a9529bd..5b21eb2119 100644 --- a/docs/index.md +++ b/docs/index.md @@ -49,6 +49,7 @@ training/activation-recomputation.md training/cpu-offloading.md training/peft.md training/packed-sequences.md +training/distillation.md ``` ```{toctree} diff --git a/docs/training/distillation.md b/docs/training/distillation.md new file mode 100644 index 0000000000..1a014ef89b --- /dev/null +++ b/docs/training/distillation.md @@ -0,0 +1,121 @@ +# Knowledge Distillation + +Megatron Bridge provides a streamlined setup for Knowledge Distillation (KD) training, making it easy to enable and integrate into your workflow. This section explains how to use this feature effectively. + +Knowledge Distillation is a technique where a pre-trained model (the "teacher") transfers its learned knowledge to a second model (the "student"), which is typically smaller and faster. This process helps the student model learn more efficiently by mimicking the behavior of the teacher. KD offers two key advantages over traditional training: faster convergence and higher final accuracy. + +In Megatron Bridge, KD is enabled by NVIDIA TensorRT Model Optimizer (ModelOpt) — a library to optimize deep-learning models for inference on GPUs. + +## Knowledge Distillation Process + +The KD process involves these steps: + +1. **Loads Checkpoints**: Loads both the student and teacher model checkpoints. +2. **Replaces Loss Function**: Replaces the standard loss function with the KL-Divergence between the output logits (and potentially additional losses between pairs of intermediate model states). +3. **Trains Models**: Runs forward passes on both models, but executes the backward pass only on the student model. +4. **Saves Checkpoints**: Saves only the student model checkpoint, allowing it to be used later in the same manner as before. + +## Limitations + +* Only GPT-based checkpoints are currently supported. +* Student and teacher models must support the same parallelism strategy. +* If Pipeline Parallelism is enabled, intermediate-state based KD losses are only supported on the final pipeline stage. + +## Configuration + +### Knowledge Distillation Config + +You can configure the KD process via the `ModelOptDistillConfig` class or a YAML file. The configuration includes: + +* `logit_layers`: The layer names of student and teacher model logit layers. These names correspond to the PyTorch submodule attributes of the Megatron Core model. (For GPT-based models, this is `"output_layer"`). Default: `["output_layer", "output_layer"]` +* `intermediate_layer_pairs`: A list of pairs of intermediate layer names. These pairs will by default have a Cosine-Similarity loss between them, and if tensor-parallelism is enabled, these layers must have sequence parallel outputs (i.e. LayerNorms), as Cosine loss cannot have a split hidden dimension. Default: `[["decoder.final_layernorm", "decoder.final_layernorm"]]` +* `skip_lm_loss`: Whether to skip the default language modeling (LM) loss. If `false`, it will be added to the distillation loss. (Note it consumes more memory). Default: `true` +* `kd_loss_scale`: Relative scale factor for the distillation loss. The cumulative logits-and-intermediate loss gets scaled to `kd_loss_scale` times the magnitude of the LM loss. Not used if `skip_lm_loss` is `true`. Default: `1.0` +* `logit_kl_temperature`: Temperature variable for KL Divergence loss calculation. Default: `1.0` + +Example YAML configuration: + +```yaml +logit_layers: ["output_layer", "output_layer"] +intermediate_layer_pairs: + - ["decoder.final_layernorm", "decoder.final_layernorm"] +logit_kl_temperature: 2.0 +``` + +## Usage + +### Basic Usage with Default Configuration + +The simplest way to run knowledge distillation is to use or adapt one of the provided recipe scripts. Here's an example for distilling Llama3.2-3B into Llama3.2-1B: + +```bash +torchrun --nproc_per_node=1 examples/recipes/llama/distill_llama32_3b-1b.py +``` + +### Using a Custom YAML Config File + +You can provide a custom YAML configuration file to override default settings: + +```bash +torchrun --nproc_per_node=1 examples/recipes/llama/distill_llama32_3b-1b.py \ + --config-file my_custom_config.yaml +``` + +### Using CLI Overrides + +Megatron Bridge supports Hydra-style CLI overrides for flexible configuration: + +```bash +torchrun --nproc_per_node=2 examples/recipes/llama/distill_llama32_3b-1b.py \ + model.tensor_model_parallel_size=2 \ + model.teacher.tensor_model_parallel_size=2 +``` + +### Combining YAML and CLI Overrides + +CLI overrides take precedence over YAML configuration: + +```bash +torchrun --nproc_per_node=2 examples/recipes/llama/distill_llama32_3b-1b.py \ + --config-file conf/my_config.yaml \ + train.global_batch_size=512 +``` + +## Model Support + +Currently, distillation is supported for GPT and Mamba-based models + +To enable distillation for a model: + +1. Use `GPTDistillationProvider` instead of `GPTModelProvider` +2. Set the `teacher` attribute to the teacher model configuration +3. Configure `kd_config` with desired distillation settings + +## Checkpointing + +During distillation training: + +* Only the **student model** checkpoints are saved +* Teacher model remains frozen and is not modified +* Checkpoints can be used for inference or further training like any standard checkpoint + +## Best Practices + +1. **Match Parallelism**: Ensure student and teacher use compatible parallelism configurations +2. **Monitor Loss**: Track both distillation loss and (if enabled) language modeling loss +3. **Batch Size**: Use larger batch sizes for better stability during distillation +4. **Learning Rate**: Start with a smaller LR than pretraining +5. **Data Quality**: Use high-quality, diverse training data for best distillation results + +## Troubleshooting + +### Out of Memory Errors + +* Reduce `train.micro_batch_size` +* Increase parallelism sizes +* Set `model.kd_config.skip_lm_loss = True` to save memory + +## References + +For more information on the underlying implementation, see: +* [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) diff --git a/examples/recipes/llama/conf/llama32_3b-1b_distill_override_example.yaml b/examples/recipes/llama/conf/llama32_3b-1b_distill_override_example.yaml new file mode 100644 index 0000000000..1a777d4348 --- /dev/null +++ b/examples/recipes/llama/conf/llama32_3b-1b_distill_override_example.yaml @@ -0,0 +1,82 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Example override file + +# To override a parameter, ensure the structure matches the ConfigContainer +# and its sub-configurations (e.g., model, train, etc.) +# Top-level ConfigContainer fields are dataclasses themselves + +model: + seq_length: 4096 + tensor_model_parallel_size: 2 + pipeline_model_parallel_size: 1 + context_parallel_size: 1 + sequence_parallel: true + teacher: + seq_length: 4096 + tensor_model_parallel_size: 2 + pipeline_model_parallel_size: 1 + context_parallel_size: 1 + sequence_parallel: true + kd_config: + logit_layers: ["output_layer", "output_layer"] + intermediate_layer_pairs: [] + skip_lm_loss: true + kd_loss_scale: 1.0 + logit_kl_temperature: 1.0 + +train: + train_iters: 10 + global_batch_size: 8 + micro_batch_size: 1 + eval_iters: 8 + +optimizer: + lr: 1e-4 + min_lr: 1e-5 + +scheduler: + lr_warmup_iters: 3 + +checkpoint: + # Directory to save to. If null, no checkpoint will be saved. + save: "./distill_llama32_3b-1b" + +dist: + use_megatron_fsdp: false + use_torch_fsdp2: false + +logger: + log_interval: 1 + +dataset: + sequence_length: 4096 + +rng: + seed: 42 + +ddp: + grad_reduce_in_fp32: true + +profiling: + # For optional fields in the config, specify the target to instantiate the object. + _target_: megatron.bridge.training.config.ProfilingConfig + use_nsys_profiler: false + profile_step_start: 5 + profile_step_end: 10 + use_pytorch_profiler: true + profile_ranks: [0, 1] + record_shapes: true + diff --git a/examples/recipes/llama/distill_llama32_3b-1b.py b/examples/recipes/llama/distill_llama32_3b-1b.py new file mode 100644 index 0000000000..231fedf685 --- /dev/null +++ b/examples/recipes/llama/distill_llama32_3b-1b.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Llama3.2 Knowledge Distillation Script with YAML and CLI Configuration Overrides. + +This script provides an example of knowledge distillation using a Llama3.2-3B teacher model +to distill knowledge into a Llama3.2-1B student model using Megatron-Bridge with support for +both YAML configuration files and command-line overrides using Hydra-style syntax. + +Examples: + Basic usage with default configuration: + $ torchrun --nproc_per_node=8 distill_llama32_3b-1b.py + + Using a custom YAML config file: + $ torchrun --nproc_per_node=8 distill_llama32_3b-1b.py --config-file my_custom_config.yaml + + Using CLI overrides: + $ torchrun --nproc_per_node=8 distill_llama32_3b-1b.py \ + model.tensor_model_parallel_size=4 \ + model.teacher.tensor_model_parallel_size=4 \ + train.train_iters=100000 + + Combining YAML and CLI overrides (CLI takes precedence): + $ torchrun --nproc_per_node=8 distill_llama32_3b-1b.py --config-file conf/my_config.yaml \ + model.pipeline_dtype=torch.float16 \ + model.teacher.pipeline_dtype=torch.float16 \ + train.global_batch_size=512 + +Configuration Precedence: + 1. Base configuration from student and teacher pretrain_config() recipes + 2. YAML overrides from --config-file (if provided) + 3. CLI overrides (highest precedence) + +Supported Override Syntax: + - Standard assignment: key=value + - Nested assignment: section.subsection.key=value + - Addition: +new_key=value + - Deletion: ~key_to_remove + - Type conversion: Automatic for basic types (int, float, bool, str) + - Complex types: torch.dtype, enums, etc. are supported +""" + +import argparse +import logging +import os +import sys +from pathlib import Path +from typing import Tuple + +import torch +from omegaconf import OmegaConf + +from megatron.bridge.models.gpt_provider import GPTDistillationProvider +from megatron.bridge.recipes.llama import llama32_1b_pretrain_config, llama32_3b_pretrain_config +from megatron.bridge.training.config import ConfigContainer +from megatron.bridge.training.distill import distill +from megatron.bridge.training.post_training.distillation import ModelOptDistillConfig +from megatron.bridge.training.utils.omegaconf_utils import ( + apply_overrides, + create_omegaconf_dict_config, + parse_hydra_overrides, +) +from megatron.bridge.utils.common_utils import get_rank_safe + + +logger: logging.Logger = logging.getLogger(__name__) + + +# Define paths relative to this script's location +# Assumes this script (distill_llama32_3b-1b.py) is in Megatron-Bridge/examples/recipes/llama/ +# and the config is in a 'conf' subdirectory. +SCRIPT_DIR: Path = Path(__file__).parent.resolve() +DEFAULT_CONFIG_FILENAME: str = "llama32_3b-1b_distill_override_example.yaml" +DEFAULT_CONFIG_FILE_PATH: Path = SCRIPT_DIR / "conf" / DEFAULT_CONFIG_FILENAME + + +def parse_cli_args() -> Tuple[argparse.Namespace, list[str]]: + """Parse command line arguments, separating known script args from OmegaConf overrides.""" + parser = argparse.ArgumentParser( + description="Knowledge distillation with Llama3.2 using Megatron-Bridge with YAML and CLI overrides", + formatter_class=argparse.RawTextHelpFormatter, + ) + parser.add_argument( + "--config-file", + type=str, + default=str(DEFAULT_CONFIG_FILE_PATH), + help="Path to the YAML OmegaConf override file. Default: conf/llama32_3b-1b_distill_override_example.yaml", + ) + parser.add_argument("--debug", action="store_true", help="Enable debug logging") + + # Parse known args for the script, remaining will be treated as overrides + args, cli_dotlist_overrides = parser.parse_known_args() + return args, cli_dotlist_overrides + + +def main() -> None: + """ + Entry point for the Llama3.2 knowledge distillation script. + + This function orchestrates the complete configuration workflow: + 1. Loads the base student configuration (Llama3.2-1B) and teacher configuration (Llama3.2-3B) + 2. Wraps both in a GPTDistillationProvider to create a unified distillation model + 3. Applies YAML overrides from --config-file (if exists) + 4. Applies CLI overrides using Hydra-style syntax + 5. Starts Megatron distillation with the final merged configuration + + The config.model structure contains student and teacher model providers: + - config.model: The Llama3.2-1B student model configuration + - config.model.teacher: The Llama3.2-3B teacher model configuration + - config.model.kd_config: Knowledge distillation-specific settings + + Configuration merging preserves callable fields (like activation functions) + and handles type conversions automatically. + + Examples of CLI usage: + # Use default config with custom learning rate + torchrun --nproc_per_node=8 distill_llama32_3b-1b.py optimizer.lr=0.0002 + + # Custom config file with additional overrides + torchrun --nproc_per_node=8 distill_llama32_3b-1b.py --config-file my_config.yaml train.train_iters=50000 + + # Multiple overrides for distributed training + torchrun --nproc_per_node=8 distill_llama32_3b-1b.py \ + model.tensor_model_parallel_size=4 \ + model.pipeline_model_parallel_size=2 \ + model.teacher.tensor_model_parallel_size=4 \ + model.teacher.pipeline_model_parallel_size=2 \ + train.global_batch_size=512 + """ + args, cli_overrides = parse_cli_args() + + logger.info("Megatron-Bridge Llama3.2 3B-1B Distillation Script with YAML & CLI Overrides") + logger.info("------------------------------------------------------------------") + + # Load base configurations as recipes and wrap provider for distillation mode + cfg: ConfigContainer = llama32_1b_pretrain_config() + cfg.model.__class__ = GPTDistillationProvider + cfg.model.teacher = llama32_3b_pretrain_config().model + cfg.model.kd_config = ModelOptDistillConfig() + logger.info("Loaded base student and teacher configurations") + + # Print configuration on rank 0 + if get_rank_safe() == 0: + cfg.print_yaml() + + # Convert the initial Python dataclass to an OmegaConf DictConfig for merging + merged_omega_conf, excluded_fields = create_omegaconf_dict_config(cfg) + + # Load and merge YAML overrides if a config file is provided + if args.config_file: + logger.debug(f"Loading YAML overrides from: {args.config_file}") + if not os.path.exists(args.config_file): + logger.error(f"Override YAML file not found: {args.config_file}") + sys.exit(1) + yaml_overrides_omega = OmegaConf.load(args.config_file) + merged_omega_conf = OmegaConf.merge(merged_omega_conf, yaml_overrides_omega) + logger.debug("YAML overrides merged successfully.") + + # Apply command-line overrides using Hydra-style parsing + if cli_overrides: + logger.debug(f"Applying Hydra-style command-line overrides: {cli_overrides}") + merged_omega_conf = parse_hydra_overrides(merged_omega_conf, cli_overrides) + logger.debug("Hydra-style command-line overrides applied successfully.") + + # Apply the final merged OmegaConf configuration back to the original ConfigContainer + logger.debug("Applying final merged configuration back to Python ConfigContainer...") + final_overrides_as_dict = OmegaConf.to_container(merged_omega_conf, resolve=True) + # Apply overrides while preserving excluded fields + apply_overrides(cfg, final_overrides_as_dict, excluded_fields) + + # Display final configuration + if get_rank_safe() == 0: + logger.info("--- Final Merged Configuration ---") + cfg.print_yaml() + logger.info("----------------------------------") + + # Start training + logger.debug("Starting distillation...") + distill(config=cfg) + + # Cleanup process group + if torch.distributed.is_initialized(): + torch.distributed.barrier() + torch.distributed.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/pyproject.toml b/pyproject.toml index df51d8ea65..a1fc9b6a8a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -103,6 +103,7 @@ no-build-isolation-package = [ ] prerelease = "allow" override-dependencies = [ + "nvidia-modelopt[torch]>=0.37.0", "torch; sys_platform == 'never'", "torchvision; sys_platform == 'never'", "triton; sys_platform == 'never'", diff --git a/src/megatron/bridge/models/gpt_provider.py b/src/megatron/bridge/models/gpt_provider.py index 0d0222a1f0..1a5ffc9a9c 100644 --- a/src/megatron/bridge/models/gpt_provider.py +++ b/src/megatron/bridge/models/gpt_provider.py @@ -17,8 +17,10 @@ import logging from dataclasses import dataclass, field from functools import partial -from typing import Any, Callable, Literal, Optional, Union +from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Union +import modelopt.torch.distill as mtd +import modelopt.torch.distill.plugins.megatron as mtd_mcore import torch from megatron.core import parallel_state from megatron.core.models.gpt import GPTModel as MCoreGPTModel @@ -38,6 +40,10 @@ from megatron.bridge.utils.vocab_utils import calculate_padded_vocab_size +if TYPE_CHECKING: + from megatron.bridge.training.post_training.distillation import ModelOptDistillConfig + + logger = logging.getLogger(__name__) @@ -291,6 +297,61 @@ def provide(self, pre_process=None, post_process=None, vp_stage=None) -> MCoreGP return model +@dataclass +class GPTDistillationProvider(GPTModelProvider): + """Provider for Megatron Core GPT models in distillation mode.""" + + teacher: Optional["GPTModelProvider"] = None + kd_config: Optional["ModelOptDistillConfig"] = None + + def __post_init__(self): + assert self.teacher is not None, "Teacher model must be provided." + shared_attrs = [ + "tensor_model_parallel_size", + "pipeline_model_parallel_size", + "context_parallel_size", + "seq_length", + "pipeline_dtype", + ] + for attr in shared_attrs: + if getattr(self, attr) != getattr(self.teacher, attr): + raise ValueError(f"Student and teacher providers must have the same {attr}.") + + def provide(self, pre_process=None, post_process=None, vp_stage=None) -> MCoreGPTModel: + """Configure and instantiate a ModelOpt DistillationModel based on this configuration. + + Args: + pre_process: Whether to include pre-processing in the model, defaults to first pipeline stage + post_process: Whether to include post-processing in the model, defaults to last pipeline stage + vp_stage: Virtual pipeline stage + + Returns: + MCoreGPTModel: Configured ModelOpt DistillationModel instance + """ + if vp_stage is not None: + raise ValueError("ModelOpt KD currently does not support virtual-pipeline parallel.") + + student_model = super().provide(pre_process, post_process, vp_stage) + teacher_model = self.teacher.provide(pre_process, post_process, vp_stage) + + kd_cfg = mtd_mcore.setup_distillation_config(self.kd_config, student_model.config, teacher_model.config) + modelopt_cfg = { + "teacher_model": teacher_model, + "criterion": kd_cfg.criterion, + "loss_balancer": kd_cfg.loss_balancer, + } + kd_model = mtd.convert(student_model, mode=[("kd_loss", modelopt_cfg)]) + mtd_mcore.adjust_distillation_model_for_mcore(kd_model, kd_cfg) + + return kd_model + + def __setattr__(self, name, value): + super().__setattr__(name, value) + # Mirror to teacher if it has that attribute + if hasattr(self.teacher, name): + setattr(self.teacher, name, value) + + def mtp_block_spec(config: "GPTModelProvider", vp_stage: Optional[int] = None) -> Optional[ModuleSpec]: """Pass in the MTP block spec if model has MTP layers. diff --git a/src/megatron/bridge/training/checkpointing.py b/src/megatron/bridge/training/checkpointing.py index 67a5cc58cc..204e8a7723 100644 --- a/src/megatron/bridge/training/checkpointing.py +++ b/src/megatron/bridge/training/checkpointing.py @@ -47,6 +47,12 @@ from megatron.core.rerun_state_machine import get_rerun_state_machine from megatron.core.transformer import MegatronModule from megatron.core.utils import unwrap_model +from modelopt.torch.opt.plugins import ( + restore_modelopt_state, + restore_sharded_modelopt_state, + save_modelopt_state, + save_sharded_modelopt_state, +) from megatron.bridge.peft.base import PEFT from megatron.bridge.training import fault_tolerance @@ -91,20 +97,6 @@ except ImportError: HAVE_MEGATRON_FSDP = False - -# [ModelOpt]: Import -try: - from modelopt.torch.opt.plugins import ( - restore_modelopt_state, - restore_sharded_modelopt_state, - save_modelopt_state, - save_sharded_modelopt_state, - ) - - has_nvidia_modelopt = True -except Exception: - has_nvidia_modelopt = False - TRACKER_PREFIX = "latest" _CHECKPOINT_VERSION = None @@ -619,15 +611,13 @@ def save_checkpoint( content_metadata=sharded_sd_metadata, ) # [ModelOpt]: save sharded modelopt_state - if has_nvidia_modelopt: - save_sharded_modelopt_state(model, checkpoint_name, (ckpt_cfg.ckpt_format, 1)) + save_sharded_modelopt_state(model, checkpoint_name, (ckpt_cfg.ckpt_format, 1)) else: # [ModelOpt]: Inject modelopt_state into state_dict - if has_nvidia_modelopt: - if ckpt_type == CheckpointType.LOCAL: - print_rank_0("WARNING: Local checkpointing does not support nvidia_modelopt.") - else: # GLOBAL checkpoint type - save_modelopt_state(model, state_dict) + if ckpt_type == CheckpointType.LOCAL: + print_rank_0("WARNING: Local checkpointing does not support nvidia_modelopt.") + else: # GLOBAL checkpoint type + save_modelopt_state(model, state_dict) end_ckpt = time() logger.debug(f"rank: {rank}, takes {end_ckpt - start_ckpt} to prepare state dict for ckpt ") @@ -1151,8 +1141,8 @@ def _load_model_weights_from_checkpoint( print_rank_0(f"sharded_state_dict metadata loaded from the checkpoint: {sharded_sd_metadata}") model_sd_kwargs = dict(metadata=sharded_sd_metadata) - if has_nvidia_modelopt: - restore_modelopt_state(model, state_dict) + # [ModelOpt]: Restore state + restore_modelopt_state(model, state_dict) model = unwrap_model(model) sharded_state_dict = _generate_model_state_dict(model, model_sd_kwargs) @@ -1395,13 +1385,12 @@ def _load_checkpoint_from_path( model_sd_kwargs = dict(metadata=sharded_sd_metadata) # ModelOpt restoration - if has_nvidia_modelopt: - if ckpt_type == CheckpointType.LOCAL: - print_rank_0("WARNING: Local checkpointing does not support nvidia_modelopt.") - elif ckpt_type == CheckpointType.GLOBAL: - restore_modelopt_state(model, state_dict) - else: - restore_sharded_modelopt_state(model, checkpoint_name) + if ckpt_type == CheckpointType.LOCAL: + print_rank_0("WARNING: Local checkpointing does not support nvidia_modelopt.") + elif ckpt_type == CheckpointType.GLOBAL: + restore_modelopt_state(model, state_dict) + else: + restore_sharded_modelopt_state(model, checkpoint_name) # Build sharded state dict for loading with contextlib.ExitStack() as stack: diff --git a/src/megatron/bridge/training/distill.py b/src/megatron/bridge/training/distill.py new file mode 100644 index 0000000000..462327465a --- /dev/null +++ b/src/megatron/bridge/training/distill.py @@ -0,0 +1,37 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from megatron.bridge.models.gpt_provider import GPTDistillationProvider +from megatron.bridge.training.config import ConfigContainer +from megatron.bridge.training.gpt_step import forward_step_modelopt +from megatron.bridge.training.pretrain import pretrain +from megatron.bridge.utils.decorators import experimental_fn + + +@experimental_fn +def distill( + config: ConfigContainer, +) -> None: + """Main function to run knowledge distillation (KD). + + Args: + config: The main configuration container holding all necessary parameters. + + Warnings: + This is an experimental API and is subject to change in backwards + incompatible ways without notice. + """ + assert isinstance(config.model, GPTDistillationProvider), "Distillation requires a GPTDistillationProvider" + + return pretrain(config, forward_step_modelopt) diff --git a/src/megatron/bridge/training/gpt_step.py b/src/megatron/bridge/training/gpt_step.py index 3a13b3e08c..fefae74c19 100644 --- a/src/megatron/bridge/training/gpt_step.py +++ b/src/megatron/bridge/training/gpt_step.py @@ -16,13 +16,15 @@ from functools import partial from typing import Iterable +import modelopt.torch.distill as mtd import torch from megatron.core import parallel_state from megatron.core.models.gpt import GPTModel -from megatron.core.utils import get_batch_on_this_cp_rank, get_model_config +from megatron.core.utils import get_batch_on_this_cp_rank, get_model_config, unwrap_model from megatron.bridge.training.config import ConfigContainer from megatron.bridge.training.losses import masked_next_token_loss +from megatron.bridge.training.post_training.distillation import loss_func_kd from megatron.bridge.training.state import GlobalState from megatron.bridge.training.utils.packed_seq_utils import get_packed_seq_params @@ -122,9 +124,9 @@ def get_batch( ) -def forward_step( +def _forward_step_common( state: GlobalState, data_iterator: Iterable, model: GPTModel, return_schedule_plan: bool = False -) -> tuple[torch.Tensor, partial]: +) -> tuple[torch.Tensor, torch.Tensor]: """Forward training step. Args: @@ -134,7 +136,7 @@ def forward_step( return_schedule_plan (bool): Whether to return the schedule plan instead of the output tensor Returns: - tuple containing the output tensor and the loss function + tuple containing the output tensor and loss mask """ timers = state.timers straggler_timer = state.straggler_timer @@ -165,9 +167,6 @@ def forward_step( } forward_args["packed_seq_params"] = get_packed_seq_params(packed_seq_params) - check_for_nan_in_loss = state.cfg.rerun_state_machine.check_for_nan_in_loss - check_for_spiky_loss = state.cfg.rerun_state_machine.check_for_spiky_loss - with straggler_timer: if return_schedule_plan: assert config.overlap_moe_expert_parallel_comm, ( @@ -176,25 +175,108 @@ def forward_step( schedule_plan = model.build_schedule_plan( tokens, position_ids, attention_mask, labels=labels, loss_mask=loss_mask ) - loss_function = _create_loss_function(loss_mask, check_for_nan_in_loss, check_for_spiky_loss) - return schedule_plan, loss_function + return schedule_plan, loss_mask else: output_tensor = model(**forward_args) - loss_function = _create_loss_function(loss_mask, check_for_nan_in_loss, check_for_spiky_loss) + return output_tensor, loss_mask - return output_tensor, loss_function + +def forward_step( + state: GlobalState, data_iterator: Iterable, model: GPTModel, return_schedule_plan: bool = False +) -> tuple[torch.Tensor, partial]: + """Forward training step. + + Args: + state: Global state for the run + data_iterator: Input data iterator + model: The GPT Model + return_schedule_plan (bool): Whether to return the schedule plan instead of the output tensor + + Returns: + tuple containing the output tensor and the loss function + """ + output, loss_mask = _forward_step_common(state, data_iterator, model, return_schedule_plan) + + loss_function = _create_loss_function( + loss_mask, + check_for_nan_in_loss=state.cfg.rerun_state_machine.check_for_nan_in_loss, + check_for_spiky_loss=state.cfg.rerun_state_machine.check_for_spiky_loss, + ) + + return output, loss_function def _create_loss_function(loss_mask: torch.Tensor, check_for_nan_in_loss: bool, check_for_spiky_loss: bool) -> partial: """Create a partial loss function with the specified configuration. + Args: + loss_mask: Used to mask out some portions of the loss + check_for_nan_in_loss: Whether to check for NaN values in the loss + check_for_spiky_loss: Whether to check for spiky loss values + + Returns: + A partial function that can be called with output_tensor to compute the loss + """ + return partial( + masked_next_token_loss, + loss_mask, + check_for_nan_in_loss=check_for_nan_in_loss, + check_for_spiky_loss=check_for_spiky_loss, + ) + + +def forward_step_modelopt( + state: GlobalState, data_iterator: Iterable, model: GPTModel, return_schedule_plan: bool = False +) -> tuple[torch.Tensor, partial]: + """Forward training step with ModelOpt required modifications. + + Args: + state: Global state for the run + data_iterator: Input data iterator + model: The GPT Model + return_schedule_plan (bool): Whether to return the schedule plan instead of the output tensor + + Returns: + tuple containing the output tensor and the loss function + """ + output, loss_mask = _forward_step_common(state, data_iterator, model, return_schedule_plan) + + loss_function = _create_loss_function_modelopt( + loss_mask, + model, + check_for_nan_in_loss=state.cfg.rerun_state_machine.check_for_nan_in_loss, + check_for_spiky_loss=state.cfg.rerun_state_machine.check_for_spiky_loss, + ) + + return output, loss_function + + +def _create_loss_function_modelopt( + loss_mask: torch.Tensor, model: GPTModel, check_for_nan_in_loss: bool, check_for_spiky_loss: bool +) -> partial: + """Create a partial loss function with the specified configuration. + Kept here for backward compatibility with tests and callers that patch `megatron.bridge.training.gpt_step.masked_next_token_loss`. + + Args: + loss_mask: Used to mask out some portions of the loss + model: The GPT Model + check_for_nan_in_loss: Whether to check for NaN values in the loss + check_for_spiky_loss: Whether to check for spiky loss values + + Returns: + A partial function that can be called with output_tensor to compute the loss """ - return partial( + mnt_loss_func = partial( masked_next_token_loss, loss_mask, check_for_nan_in_loss=check_for_nan_in_loss, check_for_spiky_loss=check_for_spiky_loss, ) + unwrapped_model = unwrap_model(model) + if isinstance(unwrapped_model, mtd.DistillationModel): + return partial(loss_func_kd, loss_mask=loss_mask, original_loss_fn=mnt_loss_func, model=unwrapped_model) + else: + return mnt_loss_func diff --git a/src/megatron/bridge/training/model_load_save.py b/src/megatron/bridge/training/model_load_save.py index 1b50666dc5..5e8756fea3 100644 --- a/src/megatron/bridge/training/model_load_save.py +++ b/src/megatron/bridge/training/model_load_save.py @@ -253,7 +253,7 @@ def build_and_load_model( from megatron.bridge.training.mlm_compat.model import _get_model, _gpt_provider, _mamba_provider from megatron.bridge.training.post_training.checkpointing import has_modelopt_state - if has_modelopt_state(checkpoint_path): + if has_modelopt_state(checkpoint_path, ignore_kd_state=True): if hasattr(model_cfg, "restore_modelopt_state"): model_cfg.restore_modelopt_state = True diff --git a/src/megatron/bridge/training/post_training/__init__.py b/src/megatron/bridge/training/post_training/__init__.py new file mode 100644 index 0000000000..341a77c5bc --- /dev/null +++ b/src/megatron/bridge/training/post_training/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/src/megatron/bridge/training/post_training/checkpointing.py b/src/megatron/bridge/training/post_training/checkpointing.py index 773dc41421..2fcfe7ead1 100644 --- a/src/megatron/bridge/training/post_training/checkpointing.py +++ b/src/megatron/bridge/training/post_training/checkpointing.py @@ -20,26 +20,35 @@ raise ImportError('Required `"nvidia-modelopt[torch]"` is not installed!') from e import os.path -from typing import List +import torch +from megatron.core.dist_checkpointing.strategies.common import COMMON_STATE_FNAME from megatron.core.transformer.module import MegatronModule from megatron.core.utils import unwrap_model -def has_modelopt_state(checkpoint_path: str) -> bool: +def has_modelopt_state(checkpoint_path: str, ignore_kd_state: bool = False) -> bool: """Check if modelopt_state folder exists inside the checkpoint path. Args: checkpoint_path: Path to the checkpoint directory + ignore_kd_state: If True, ignore the distillation state, as it is a placeholder Returns: - True if modelopt_state folder exists, False otherwise + True if modelopt_state folder exists when ignore_kd_state is False, + True if modelopt_state folder exists when ignore_kd_state is True and has only + distillation state, False otherwise """ modelopt_state_path = os.path.join(checkpoint_path, "modelopt_state") - return os.path.isdir(modelopt_state_path) + if not os.path.isdir(modelopt_state_path): + return False + elif ignore_kd_state: + return _has_only_kd_state(modelopt_state_path) + else: + return True -def load_modelopt_state(model: List[MegatronModule], checkpoint_path: str) -> None: +def load_modelopt_state(model: list[MegatronModule], checkpoint_path: str) -> None: """Load modelopt_state from a checkpoint. Args: model: The model to load the modelopt_state into @@ -47,3 +56,11 @@ def load_modelopt_state(model: List[MegatronModule], checkpoint_path: str) -> No """ unwrapped_model = unwrap_model(model) restore_sharded_modelopt_state(unwrapped_model, checkpoint_path) + + +def _has_only_kd_state(modelopt_state_path: str) -> bool: + modelopt_state = torch.load(modelopt_state_path + "/" + COMMON_STATE_FNAME, weights_only=False) + modes_dict = modelopt_state["modelopt_state_dict"] + if len(modes_dict) == 1 and modes_dict[0][0] == "kd_loss": + return True + return False diff --git a/src/megatron/bridge/training/post_training/distillation.py b/src/megatron/bridge/training/post_training/distillation.py new file mode 100644 index 0000000000..8598a56a7e --- /dev/null +++ b/src/megatron/bridge/training/post_training/distillation.py @@ -0,0 +1,90 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Callable + +import modelopt.torch.distill as mtd +import modelopt.torch.distill.plugins.megatron as mtd_mcore +import torch +from megatron.core import parallel_state +from megatron.core.transformer import MegatronModule + + +class ModelOptDistillConfig(mtd_mcore.DistillationConfig): + """Configuration settings for Model Optimizer distillation.""" + + pass + + +def loss_func_kd( + output_tensor: torch.Tensor, loss_mask: torch.Tensor, original_loss_fn: Callable, model: MegatronModule +): + """Loss function (with KD Loss support). + + Args: + output_tensor (Tensor): The tensor with the losses + loss_mask (Tensor): Used to mask out some portions of the loss + original_loss_fn (Callable): The original loss function + model (GPTModel): The model (can be wrapped) + """ + assert isinstance(model, mtd.DistillationModel), "Model must be a ModelOpt DistillationModel" + + # Standard lm loss + loss_lm, num_tokens, report = original_loss_fn(output_tensor) + + # Handle knowledge distillation + losses_kd = model.compute_kd_loss( + student_loss=loss_lm, + loss_reduction_fn=lambda x: _mask_loss(x, loss_mask), + ) + + report["total loss"] = torch.cat([losses_kd["kd_loss"].clone().detach().view(1), num_tokens.view(1)]) + report["logits distillation loss"] = torch.cat( + [losses_kd["logits_loss"].clone().detach().view(1), num_tokens.view(1)] + ) + report["intermediate distillation loss"] = torch.cat( + [losses_kd["intermediate_loss"].clone().detach().view(1), num_tokens.view(1)] + ) + + # Validation loss remains unchanged + if model.training: + loss = losses_kd["kd_loss"] + else: + loss = loss_lm + + return loss, num_tokens, report + + +def _mask_loss(output_tensor: torch.Tensor, loss_mask: torch.Tensor): + if isinstance(output_tensor, tuple): + # Special distillation flags indicating whether to perform additional tensor-parallel adjustments. + output_tensor, tp_reduce, is_sequence_parallel = output_tensor + else: + tp_reduce, is_sequence_parallel = False, False + tp_group = parallel_state.get_tensor_model_parallel_group() + + if is_sequence_parallel: + # Sequence-parallel tensor derived from intermediate activation - need to split loss mask. + idx = tp_group.rank() + loss_mask = torch.tensor_split(loss_mask, tp_group.size(), dim=1)[idx] + + losses = output_tensor.view(-1).float() + loss_mask = loss_mask.reshape(-1).float() + loss = torch.sum(losses * loss_mask) + + if tp_reduce or is_sequence_parallel: + # Losses on parallel tensors require extra all-reduce to sync across MP ranks. + torch.distributed.all_reduce(loss, group=tp_group) + + return loss diff --git a/src/megatron/bridge/training/train.py b/src/megatron/bridge/training/train.py index 0299b158c3..b25b1d298b 100644 --- a/src/megatron/bridge/training/train.py +++ b/src/megatron/bridge/training/train.py @@ -38,6 +38,7 @@ from megatron.core.transformer import MegatronModule from megatron.core.transformer.cuda_graphs import TECudaGraphHelper from megatron.core.utils import check_param_hashes_across_dp_replicas, get_model_config +from modelopt.torch.distill.plugins.megatron import get_tensor_shapes_adjust_fn_for_distillation from megatron.bridge.training import fault_tolerance from megatron.bridge.training.checkpointing import maybe_finalize_async_save, save_checkpoint @@ -537,6 +538,14 @@ def train_step( seq_key="tokens", ) + # [ModelOpt]: Pipeline-parallel Distillation stacks student and teacher tensors + adjust_tensor_shapes_fn = get_tensor_shapes_adjust_fn_for_distillation( + model, + seq_length=model_config.seq_length, + micro_batch_size=train_config.micro_batch_size, + decoder_seq_length=model_config.seq_length, + ) + # Forward pass. forward_backward_func = get_forward_backward_func() losses_reduced = forward_backward_func( @@ -548,6 +557,7 @@ def train_step( micro_batch_size=train_config.micro_batch_size, decoder_seq_length=seq_length, forward_only=False, + adjust_tensor_shapes_fn=adjust_tensor_shapes_fn, ) should_checkpoint, should_exit, exit_code = rerun_state_machine.should_checkpoint_and_exit() if should_exit: diff --git a/tests/functional_tests/L2_Launch_recipes_llama_distill.sh b/tests/functional_tests/L2_Launch_recipes_llama_distill.sh new file mode 100755 index 0000000000..e3ea38fd95 --- /dev/null +++ b/tests/functional_tests/L2_Launch_recipes_llama_distill.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -xeuo pipefail # Exit immediately if a command exits with a non-zero status + +export CUDA_VISIBLE_DEVICES="0,1" + +# Run distillation recipe functional tests on 2 GPUs +# This script tests distillation recipe configurations with their default settings to ensure +# they can run basic distillation training without crashes +python -m torch.distributed.run --nproc_per_node=2 --nnodes=1 -m coverage run --data-file=/opt/Megatron-Bridge/.coverage --source=/opt/Megatron-Bridge/ --parallel-mode -m pytest -o log_cli=true -o log_cli_level=INFO -v -s -x -m "not pleasefixme" --tb=short -rA tests/functional_tests/recipes/test_llama_recipes_distill_3b-1b.py +coverage combine -q + diff --git a/tests/functional_tests/recipes/test_llama_recipes_distill_3b-1b.py b/tests/functional_tests/recipes/test_llama_recipes_distill_3b-1b.py new file mode 100644 index 0000000000..8e4dffb479 --- /dev/null +++ b/tests/functional_tests/recipes/test_llama_recipes_distill_3b-1b.py @@ -0,0 +1,164 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path +from typing import Callable, Optional + +import pytest + +from megatron.bridge.models.gpt_provider import GPTDistillationProvider +from megatron.bridge.recipes.llama import ( + llama32_1b_pretrain_config, + llama32_3b_pretrain_config, +) +from megatron.bridge.training.config import ConfigContainer +from megatron.bridge.training.distill import distill +from megatron.bridge.training.post_training.distillation import ModelOptDistillConfig +from tests.functional_tests.utils import ( + broadcast_path, + clear_directories, + initialize_distributed, + verify_checkpoint_files, +) + + +LLAMA_DISTILL_RECIPES = [ + # (student_config_func, teacher_config_func, name, parallelism_overrides) + (llama32_1b_pretrain_config, llama32_3b_pretrain_config, "llama32_3b-1b", {"tensor_parallelism": 2}), +] + + +class TestLlamaDistillRecipes: + """Test class for LLaMA distillation recipe functional tests.""" + + @pytest.mark.run_only_on("GPU") + @pytest.mark.parametrize( + "student_config_func,teacher_config_func,recipe_name,parallelism_overrides", LLAMA_DISTILL_RECIPES + ) + def test_llama_distill_recipes( + self, student_config_func, teacher_config_func, recipe_name, parallelism_overrides, tmp_path + ): + """Functional test for LLaMA distillation recipes with appropriate parallelism configurations.""" + run_distill_recipe_test( + student_config_func, teacher_config_func, recipe_name, tmp_path, **parallelism_overrides + ) + + +def run_distill_recipe_test( + student_config_func: Callable, + teacher_config_func: Callable, + recipe_name: str, + tmp_path: Path, + tensor_parallelism: Optional[int] = None, + pipeline_parallelism: Optional[int] = None, + expert_parallelism: Optional[int] = None, + model_overrides: Optional[dict] = None, +): + """ + Common test implementation for distillation recipe configurations. + + This function runs a minimal distillation session to verify that: + 1. The recipe config can be loaded without errors + 2. Distillation can start and run for a few iterations + 3. Checkpoints are saved correctly + 4. No crashes occur during the process + + Args: + student_config_func: The student model's pretrain_config function + teacher_config_func: The teacher model's pretrain_config function + recipe_name: Name of the recipe for logging/debugging + tmp_path: Temporary directory for test outputs + tensor_parallelism: Override tensor parallelism (None = use recipe default) + pipeline_parallelism: Override pipeline parallelism (None = use recipe default) + expert_parallelism: Override expert parallelism (None = use recipe default) + model_overrides: Optional mapping of model attribute overrides to apply + """ + initialize_distributed() + shared_base_dir = broadcast_path(tmp_path) + + try: + # Load student config and wrap it with GPTDistillationProvider + config: ConfigContainer = student_config_func( + dir=str(shared_base_dir), name=f"{recipe_name}_functional_test", mock=True + ) + config.model.__class__ = GPTDistillationProvider + + # Load teacher config and add to student config + teacher_config = teacher_config_func( + dir=str(shared_base_dir), name=f"{recipe_name}_teacher_functional_test", mock=True + ) + config.model.teacher = teacher_config.model + + # Set default distillation + config.model.kd_config = ModelOptDistillConfig() + + config.train.train_iters = 10 + config.train.eval_interval = 5 + config.train.eval_iters = 2 + config.scheduler.lr_warmup_iters = 2 + test_seq_length = 512 + config.model.seq_length = test_seq_length + config.model.teacher.seq_length = test_seq_length + config.dataset.sequence_length = test_seq_length + config.train.global_batch_size = 8 + # Keep dataloader light-weight for CI + if hasattr(config.dataset, "pin_memory"): + config.dataset.pin_memory = False + if hasattr(config.dataset, "num_workers"): + config.dataset.num_workers = 0 + if hasattr(config.dataset, "persistent_workers"): + config.dataset.persistent_workers = False + + train_samples_needed = config.train.train_iters * config.train.global_batch_size + eval_samples_needed = config.train.eval_iters * config.train.global_batch_size + test_samples_needed = 100 # Minimal test samples + + total_samples = train_samples_needed + eval_samples_needed + test_samples_needed + + # Set dataset split ratios for minimal dataset + train_split = train_samples_needed / total_samples + valid_split = eval_samples_needed / total_samples + test_split = test_samples_needed / total_samples + + config.dataset.split = [train_split, valid_split, test_split] + + # Apply parallelism overrides to both student and teacher models + if tensor_parallelism is not None: + if hasattr(config.model, "tensor_model_parallel_size"): + config.model.tensor_model_parallel_size = tensor_parallelism + if hasattr(config.model.teacher, "tensor_model_parallel_size"): + config.model.teacher.tensor_model_parallel_size = tensor_parallelism + if pipeline_parallelism is not None: + if hasattr(config.model, "pipeline_model_parallel_size"): + config.model.pipeline_model_parallel_size = pipeline_parallelism + if hasattr(config.model.teacher, "pipeline_model_parallel_size"): + config.model.teacher.pipeline_model_parallel_size = pipeline_parallelism + if expert_parallelism is not None: + if hasattr(config.model, "expert_model_parallel_size"): + config.model.expert_model_parallel_size = expert_parallelism + if hasattr(config.model.teacher, "expert_model_parallel_size"): + config.model.teacher.expert_model_parallel_size = expert_parallelism + + # Apply any model-specific overrides provided by the caller + if model_overrides: + for attribute_name, attribute_value in model_overrides.items(): + setattr(config.model, attribute_name, attribute_value) + + distill(config=config) + + # Basic verification that training completed successfully + verify_checkpoint_files(config.checkpoint.save, 10) + + finally: + clear_directories(tmp_path) diff --git a/tests/unit_tests/models/test_gpt_provider.py b/tests/unit_tests/models/test_gpt_provider.py index f7ae358da4..7017574c3f 100644 --- a/tests/unit_tests/models/test_gpt_provider.py +++ b/tests/unit_tests/models/test_gpt_provider.py @@ -14,7 +14,11 @@ from unittest.mock import Mock, patch -from megatron.bridge.models import GPTModelProvider +import pytest +import torch + +from megatron.bridge.models.gpt_provider import GPTDistillationProvider, GPTModelProvider +from megatron.bridge.training.post_training.distillation import ModelOptDistillConfig class TestGPTModelProvider: @@ -400,3 +404,232 @@ def test_default_layer_spec_default_case(self, mock_te_full_spec, mock_te_spec, mock_te_full_spec.assert_not_called() mock_te_spec.assert_called_once_with(provider) assert result == "te_spec" + + +class TestGPTDistillationProvider: + """Test cases for GPTDistillationProvider class.""" + + def test_initialization_with_teacher(self): + """Test GPTDistillationProvider can be initialized with a teacher.""" + teacher = GPTModelProvider( + num_layers=24, + hidden_size=4096, + num_attention_heads=32, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + ) + student = GPTDistillationProvider( + num_layers=12, + hidden_size=2048, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + teacher=teacher, + ) + + assert student.teacher is teacher + assert student.num_layers == 12 + assert student.hidden_size == 2048 + assert student.num_attention_heads == 16 + + def test_initialization_without_teacher_raises_error(self): + """Test GPTDistillationProvider raises error when teacher is None.""" + with pytest.raises(AssertionError, match="Teacher model must be provided"): + GPTDistillationProvider( + num_layers=12, + hidden_size=2048, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + teacher=None, + ) + + def test_post_init_validates_shared_attributes(self): + """Test __post_init__ validates that shared attributes match between student and teacher.""" + teacher = GPTModelProvider( + num_layers=24, + hidden_size=4096, + num_attention_heads=32, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=torch.float32, + ) + + # Test mismatched tensor_model_parallel_size + with pytest.raises(ValueError): + GPTDistillationProvider( + num_layers=12, + hidden_size=2048, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=2, # Different from teacher + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + teacher=teacher, + ) + + def test_post_init_validates_seq_length(self): + """Test __post_init__ validates seq_length.""" + teacher = GPTModelProvider( + num_layers=24, + hidden_size=4096, + num_attention_heads=32, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=2048, + pipeline_dtype=torch.float32, + ) + + with pytest.raises(ValueError): + GPTDistillationProvider( + num_layers=12, + hidden_size=2048, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, # Different from teacher + pipeline_dtype=torch.float32, + teacher=teacher, + ) + + @patch("modelopt.torch.distill.plugins.megatron.parallel_state") + @patch("megatron.bridge.models.gpt_provider.parallel_state") + @patch("megatron.bridge.models.gpt_provider.calculate_padded_vocab_size", return_value=1024) + @patch("megatron.bridge.models.gpt_provider.MCoreGPTModel") + def test_provide_method_creates_distillation_model( + self, + mock_mcore_gpt, + mock_calc_vocab, + mock_parallel_state, + mock_mtd_parallel_state, + ): + """Test provide method creates a ModelOpt DistillationModel.""" + mock_parallel_state.is_pipeline_first_stage.return_value = True + mock_parallel_state.is_pipeline_last_stage.return_value = True + mock_mtd_parallel_state.is_pipeline_first_stage.return_value = True + mock_mtd_parallel_state.is_pipeline_last_stage.return_value = True + + teacher = GPTModelProvider( + num_layers=24, + hidden_size=4096, + num_attention_heads=32, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + ) + student = GPTDistillationProvider( + num_layers=12, + hidden_size=4096, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + teacher=teacher, + kd_config=ModelOptDistillConfig(), + ) + + # Mock the provide method calls and modelopt functions + mock_student_model = Mock() + mock_teacher_model = Mock() + mock_student_model.config = Mock() + mock_teacher_model.config = Mock() + # Avoid ProjectionLayer being created here + mock_student_model.config.hidden_size = mock_teacher_model.config.hidden_size = 4096 + mock_kd_model = Mock() + + # Set the side effects for the model provider - student first, then teacher + mock_mcore_gpt.side_effect = [mock_student_model, mock_teacher_model] + + with patch("megatron.bridge.models.gpt_provider.mtd.convert", return_value=mock_kd_model): + result = student.provide() + + # Verify that both student and teacher models were created + assert mock_mcore_gpt.call_count == 2 + assert result is mock_kd_model + + def test_setattr_mirrors_to_teacher(self): + """Test __setattr__ mirrors attributes to teacher when teacher has that attribute.""" + teacher = GPTModelProvider( + num_layers=24, + hidden_size=4096, + num_attention_heads=32, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + ) + student = GPTDistillationProvider( + num_layers=12, + hidden_size=2048, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + teacher=teacher, + ) + + student.num_layers = 10 # This exists on teacher, so it should be mirrored + assert student.num_layers == 10 + assert teacher.num_layers == 10 + + def test_setattr_does_not_mirror_when_teacher_lacks_attribute(self): + """Test __setattr__ does not mirror attributes that teacher doesn't have.""" + teacher = GPTModelProvider( + num_layers=24, + hidden_size=4096, + num_attention_heads=32, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + ) + student = GPTDistillationProvider( + num_layers=12, + hidden_size=2048, + num_attention_heads=16, + vocab_size=1000, + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + context_parallel_size=1, + seq_length=1024, + pipeline_dtype=None, + teacher=teacher, + ) + + student.new_attribute = "test_value" # Should not be reflected on teacher + assert student.new_attribute == "test_value" + assert not hasattr(teacher, "new_attribute") diff --git a/tests/unit_tests/training/post_training/test_checkpointing.py b/tests/unit_tests/training/post_training/test_checkpointing.py index 2c92ff9eb5..28b0cf3784 100644 --- a/tests/unit_tests/training/post_training/test_checkpointing.py +++ b/tests/unit_tests/training/post_training/test_checkpointing.py @@ -19,52 +19,15 @@ import pytest import torch +from megatron.core.dist_checkpointing.strategies.common import COMMON_STATE_FNAME from megatron.bridge.training.post_training.checkpointing import ( + _has_only_kd_state, has_modelopt_state, load_modelopt_state, ) -class TestPostTrainingImports: - """Test import handling for post-training checkpointing.""" - - def test_modelopt_import_available(self): - """Test that modelopt import is available in the current environment.""" - # This test ensures that the import succeeded and the functions are available - # This covers the successful import path (lines 17-18) - try: - from modelopt.torch.opt.plugins import restore_sharded_modelopt_state - - # If we get here, the import succeeded - assert callable(restore_sharded_modelopt_state) - except ImportError: - # If modelopt is not available, we should still be able to import our module - # but it should have raised the custom ImportError - pytest.fail("modelopt should be available in test environment") - - def test_import_error_scenario_simulation(self): - """Test the ImportError handling logic by simulating the scenario.""" - # This test simulates what would happen if modelopt was not available - # We can't easily test the actual import failure since the module is already loaded, - # but we can test the error handling logic - - original_error = ImportError("No module named 'modelopt.torch.opt.plugins'") - expected_message = 'Required `"nvidia-modelopt[torch]"` is not installed!' - - # Simulate the exception chain that would occur - try: - try: - raise original_error - except ImportError as e: - raise ImportError(expected_message) from e - except ImportError as final_error: - # Verify the error message and chaining - assert expected_message in str(final_error) - assert final_error.__cause__ is original_error - assert "modelopt.torch.opt.plugins" in str(final_error.__cause__) - - @pytest.fixture def mock_model_fixtures(): """Fixture for model testing.""" @@ -137,6 +100,87 @@ def test_has_modelopt_state_with_empty_string_path(self): result = has_modelopt_state("") assert result is False + def test_has_only_kd_state_returns_true(self): + """Test _has_only_kd_state when modelopt_state contains only kd_loss state.""" + with tempfile.TemporaryDirectory() as temp_dir: + modelopt_state_path = Path(temp_dir) + common_state_file = modelopt_state_path / COMMON_STATE_FNAME + + # Create modelopt_state_dict with only kd_loss + modelopt_state = {"modelopt_state_dict": [("kd_loss", {"some": "data"})]} + torch.save(modelopt_state, common_state_file) + + result = _has_only_kd_state(str(modelopt_state_path)) + assert result is True + + def test_has_only_kd_state_returns_false_multiple_states(self): + """Test _has_only_kd_state when modelopt_state contains multiple states.""" + with tempfile.TemporaryDirectory() as temp_dir: + modelopt_state_path = Path(temp_dir) + common_state_file = modelopt_state_path / COMMON_STATE_FNAME + + # Create modelopt_state_dict with multiple states including kd_loss + modelopt_state = { + "modelopt_state_dict": [ + ("kd_loss", {"some": "data"}), + ("quantization", {"other": "data"}), + ] + } + torch.save(modelopt_state, common_state_file) + + result = _has_only_kd_state(str(modelopt_state_path)) + assert result is False + + def test_has_only_kd_state_returns_false_different_state(self): + """Test _has_only_kd_state when modelopt_state contains a single state that is not kd_loss.""" + with tempfile.TemporaryDirectory() as temp_dir: + modelopt_state_path = Path(temp_dir) + common_state_file = modelopt_state_path / COMMON_STATE_FNAME + + # Create modelopt_state_dict with only quantization state (not kd_loss) + modelopt_state = {"modelopt_state_dict": [("quantization", {"some": "data"})]} + torch.save(modelopt_state, common_state_file) + + result = _has_only_kd_state(str(modelopt_state_path)) + assert result is False + + def test_has_modelopt_state_with_ignore_kd_state_true_only_kd(self): + """Test has_modelopt_state with ignore_kd_state=True when only kd_loss state exists.""" + with tempfile.TemporaryDirectory() as temp_dir: + checkpoint_path = Path(temp_dir) + modelopt_state_path = checkpoint_path / "modelopt_state" + modelopt_state_path.mkdir() + common_state_file = modelopt_state_path / COMMON_STATE_FNAME + + # Create modelopt_state_dict with only kd_loss + modelopt_state = {"modelopt_state_dict": [("kd_loss", {"some": "data"})]} + torch.save(modelopt_state, common_state_file) + + # When ignore_kd_state=True and only kd_loss exists, should return True + result = has_modelopt_state(str(checkpoint_path), ignore_kd_state=True) + assert result is True + + def test_has_modelopt_state_with_ignore_kd_state_true_multiple_states(self): + """Test has_modelopt_state with ignore_kd_state=True when multiple states exist.""" + with tempfile.TemporaryDirectory() as temp_dir: + checkpoint_path = Path(temp_dir) + modelopt_state_path = checkpoint_path / "modelopt_state" + modelopt_state_path.mkdir() + common_state_file = modelopt_state_path / COMMON_STATE_FNAME + + # Create modelopt_state_dict with multiple states + modelopt_state = { + "modelopt_state_dict": [ + ("kd_loss", {"some": "data"}), + ("quantization", {"other": "data"}), + ] + } + torch.save(modelopt_state, common_state_file) + + # When ignore_kd_state=True but multiple states exist, should return False + result = has_modelopt_state(str(checkpoint_path), ignore_kd_state=True) + assert result is False + class TestLoadModeloptState: """Test load_modelopt_state function.""" diff --git a/tests/unit_tests/training/test_checkpointing.py b/tests/unit_tests/training/test_checkpointing.py index c6343e039c..587e7295bb 100644 --- a/tests/unit_tests/training/test_checkpointing.py +++ b/tests/unit_tests/training/test_checkpointing.py @@ -39,7 +39,6 @@ get_checkpoint_tracker_filename, get_checkpoint_train_state_filename, get_rng_state, - has_nvidia_modelopt, init_checkpointing_context, load_checkpoint, read_metadata, @@ -364,13 +363,6 @@ def save_checkpoint_fixtures(): } -def _patch_modelopt_state_saver(): - """Conditionally patch modelopt state saving function.""" - if has_nvidia_modelopt: - return patch("megatron.bridge.training.checkpointing.save_sharded_modelopt_state") - return patch.object(_dummy_obj, "save_sharded_modelopt_state") - - class TestSaveCheckpoint: """Test checkpoint saving functionality.""" @@ -379,7 +371,7 @@ class TestSaveCheckpoint: @patch("builtins.open", new_callable=mock_open) @patch("torch.save") @patch("shutil.copy") - @_patch_modelopt_state_saver() + @patch("megatron.bridge.training.checkpointing.save_sharded_modelopt_state") @patch("megatron.bridge.training.checkpointing.unwrap_model") @patch("megatron.bridge.training.checkpointing.get_rng_state") @patch("megatron.bridge.training.checkpointing.get_rerun_state_machine") diff --git a/tests/unit_tests/training/test_gpt_step.py b/tests/unit_tests/training/test_gpt_step.py index 9f8521c24b..141fd9827f 100644 --- a/tests/unit_tests/training/test_gpt_step.py +++ b/tests/unit_tests/training/test_gpt_step.py @@ -13,12 +13,16 @@ # limitations under the License. from functools import partial -from unittest.mock import patch +from unittest.mock import Mock, patch +import modelopt.torch.distill as mtd import torch from megatron.core.packed_seq_params import PackedSeqParams -from megatron.bridge.training.gpt_step import get_packed_seq_params +from megatron.bridge.training.gpt_step import ( + _create_loss_function_modelopt, + get_packed_seq_params, +) from megatron.bridge.training.losses import ( create_masked_next_token_loss_function as _create_loss_function, ) @@ -241,3 +245,76 @@ def test_create_loss_function_callable(self, mock_loss_func): # Verify the result assert result == expected_result + + +class TestCreateLossFunctionModelopt: + """Tests for the _create_loss_function_modelopt helper function.""" + + def test_create_loss_function_modelopt_regular_model(self): + """Test _create_loss_function_modelopt with a regular (non-DistillationModel) model.""" + loss_mask = torch.tensor([[1.0, 1.0, 0.0]]) + mock_model = Mock() + mock_unwrapped_model = Mock() + + with patch("megatron.bridge.training.gpt_step.unwrap_model", return_value=mock_unwrapped_model): + loss_func = _create_loss_function_modelopt( + loss_mask=loss_mask, + model=mock_model, + check_for_nan_in_loss=True, + check_for_spiky_loss=True, + ) + + # Verify it returns a partial function for masked_next_token_loss (regular loss) + assert isinstance(loss_func, partial) + assert loss_func.func.__name__ == "masked_next_token_loss" + + # Verify the partial has correct arguments + assert torch.equal(loss_func.args[0], loss_mask) + assert loss_func.keywords["check_for_nan_in_loss"] == True + assert loss_func.keywords["check_for_spiky_loss"] == True + + def test_create_loss_function_modelopt_distillation_model(self): + """Test _create_loss_function_modelopt with a DistillationModel.""" + loss_mask = torch.tensor([[1.0, 0.0, 1.0]]) + mock_model = Mock() + mock_distillation_model = Mock(spec=mtd.DistillationModel) + + with patch("megatron.bridge.training.gpt_step.unwrap_model", return_value=mock_distillation_model): + loss_func = _create_loss_function_modelopt( + loss_mask=loss_mask, + model=mock_model, + check_for_nan_in_loss=False, + check_for_spiky_loss=True, + ) + + # Verify it returns a partial function for loss_func_kd (distillation loss) + assert isinstance(loss_func, partial) + assert loss_func.func.__name__ == "loss_func_kd" + + # Verify the partial has correct keyword arguments + assert torch.equal(loss_func.keywords["loss_mask"], loss_mask) + assert loss_func.keywords["model"] == mock_distillation_model + assert isinstance(loss_func.keywords["original_loss_fn"], partial) + # Verify original_loss_fn is correctly configured + assert loss_func.keywords["original_loss_fn"].func.__name__ == "masked_next_token_loss" + assert loss_func.keywords["original_loss_fn"].keywords["check_for_nan_in_loss"] == False + assert loss_func.keywords["original_loss_fn"].keywords["check_for_spiky_loss"] == True + + def test_create_loss_function_modelopt_both_flags_false(self): + """Test _create_loss_function_modelopt with both flags as False.""" + loss_mask = torch.tensor([[0.0, 1.0, 1.0]]) + mock_model = Mock() + mock_unwrapped_model = Mock() + + with patch("megatron.bridge.training.gpt_step.unwrap_model", return_value=mock_unwrapped_model): + loss_func = _create_loss_function_modelopt( + loss_mask=loss_mask, + model=mock_model, + check_for_nan_in_loss=False, + check_for_spiky_loss=False, + ) + + # Verify the partial has correct arguments + assert torch.equal(loss_func.args[0], loss_mask) + assert loss_func.keywords["check_for_nan_in_loss"] == False + assert loss_func.keywords["check_for_spiky_loss"] == False diff --git a/tests/unit_tests/training/test_model_load_save.py b/tests/unit_tests/training/test_model_load_save.py index 958594610e..918bc3c8a7 100644 --- a/tests/unit_tests/training/test_model_load_save.py +++ b/tests/unit_tests/training/test_model_load_save.py @@ -425,7 +425,7 @@ def test_load_mbridge_saved_model_with_modelopt_state( result = load_megatron_model(ckpt_path, return_state_dict=True, use_cpu_init=True) # Verify modelopt state was detected and set - mock_has_modelopt_state.assert_called_once_with(ckpt_path) + mock_has_modelopt_state.assert_called_once_with(ckpt_path, ignore_kd_state=True) assert mock_model_cfg.restore_modelopt_state is True # Verify modelopt state was loaded @@ -503,7 +503,7 @@ def test_load_mlm_saved_model_without_modelopt_support( result = load_megatron_model(ckpt_path, model_type="gpt", return_state_dict=True, use_cpu_init=True) # Verify modelopt state was detected but not set (no attribute on TransformerConfig) - mock_has_modelopt_state.assert_called_once_with(ckpt_path) + mock_has_modelopt_state.assert_called_once_with(ckpt_path, ignore_kd_state=True) # TransformerConfig doesn't have restore_modelopt_state, so hasattr returns False assert not hasattr(mock_model_cfg, "restore_modelopt_state") diff --git a/uv.lock b/uv.lock index d890bc5e55..72491835d9 100644 --- a/uv.lock +++ b/uv.lock @@ -23,6 +23,7 @@ prerelease-mode = "allow" [manifest] overrides = [ + { name = "nvidia-modelopt", extras = ["torch"], specifier = ">=0.37.0" }, { name = "torch", marker = "sys_platform == 'never'" }, { name = "torchvision", marker = "sys_platform == 'never'" }, { name = "transformer-engine", extras = ["pytorch"], git = "https://github.com/NVIDIA/TransformerEngine.git?rev=release_v2.9" }, @@ -2298,7 +2299,7 @@ dev = [ { name = "megatron-energon", extra = ["av-decode"] }, { name = "multi-storage-client" }, { name = "nv-grouped-gemm" }, - { name = "nvidia-modelopt", marker = "sys_platform != 'darwin'" }, + { name = "nvidia-modelopt" }, { name = "nvidia-resiliency-ext" }, { name = "nvtx" }, { name = "onnxscript", version = "0.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" }, @@ -3053,13 +3054,12 @@ wheels = [ [[package]] name = "nvidia-modelopt" -version = "0.33.1" +version = "0.37.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "ninja" }, { name = "numpy" }, { name = "nvidia-ml-py" }, - { name = "nvidia-modelopt-core" }, { name = "packaging" }, { name = "pulp" }, { name = "pydantic" }, @@ -3070,25 +3070,10 @@ dependencies = [ { name = "scipy", version = "1.16.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, { name = "torch", marker = "sys_platform == 'never'" }, { name = "torchprofile" }, - { name = "torchvision", marker = "sys_platform == 'never'" }, { name = "tqdm" }, ] wheels = [ - { url = "https://files.pythonhosted.org/packages/ca/cb/4af39357792a96f334c7877ea0380c9337aec210ff4794a7dd95beb7c349/nvidia_modelopt-0.33.1-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:6c51091683a117cd40fdb96a0ec28579f2276f6b627db7ccddc370df544e1dd7", size = 751683, upload-time = "2025-08-12T18:37:48.832Z" }, - { url = "https://files.pythonhosted.org/packages/0a/b1/fc2f468d140ef58e90fac584759d0cc449db9bc4f64668cdff750ef38fef/nvidia_modelopt-0.33.1-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:ef78a98901890f265596ec413dffac177d4a1865201d89a14f29f4fa0cf8e710", size = 751683, upload-time = "2025-08-12T18:36:59.964Z" }, -] - -[[package]] -name = "nvidia-modelopt-core" -version = "0.33.1" -source = { registry = "https://pypi.org/simple" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/6a/21/d12ca11f5554340684d11958aae6c6e7755cf0aaae10a2d2c9db217228cf/nvidia_modelopt_core-0.33.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:f25f6a817609c693ee39d1bcf2d3aeef462b9769f971590133de8b1b0310885b", size = 1307716, upload-time = "2025-08-12T18:41:12.086Z" }, - { url = "https://files.pythonhosted.org/packages/eb/df/7bead24d4854274d9f2818f1ae780fc24260aab60b7b6f73e1af4f056ce5/nvidia_modelopt_core-0.33.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:195f32f06d19bc9f9d858811f1864bddcc1db6278974d98ea6309cb3553427f1", size = 1326896, upload-time = "2025-08-12T18:39:48.243Z" }, - { url = "https://files.pythonhosted.org/packages/a1/36/3318980c670292d827ace5ac6110ab6054d0f2d87e507382842ea9e7c78f/nvidia_modelopt_core-0.33.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:ffd008a90d8867660ae41c98002156b526e368a4cdf39e225fe20f478adce8b2", size = 1376104, upload-time = "2025-08-12T18:41:47.358Z" }, - { url = "https://files.pythonhosted.org/packages/27/97/99d1ddabe01ab262c18621619c996e1c2c119bc058607d2bc9ce7eb85fe7/nvidia_modelopt_core-0.33.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:be49121b2f74db4cb73955396a7bb83935d92232c5a20bcfd7b8e7cae68e482f", size = 1393729, upload-time = "2025-08-12T18:40:07.86Z" }, - { url = "https://files.pythonhosted.org/packages/9b/b5/ba79b1c52b634b24e45dca409f133f947217a5c7ec5c256266e4ec5fa3eb/nvidia_modelopt_core-0.33.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:1ddd9279d8312f8e972b302692a26e6180f1c9fd277232f5925a5589f42b1b76", size = 1338081, upload-time = "2025-08-12T18:40:36.156Z" }, - { url = "https://files.pythonhosted.org/packages/13/40/4427583475dfd8eb1b8c7522d75d4d059f0512ff03dcc62d6986a22ab918/nvidia_modelopt_core-0.33.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:69d5ace564f2b056c916117be2023f2b7fc01cd1501073915e6b2ced2b8a5394", size = 1363366, upload-time = "2025-08-12T18:39:28.854Z" }, + { url = "https://files.pythonhosted.org/packages/27/a8/85034a33753a56ef120b931dda1183d98efef1010e379c54d3c3214a5048/nvidia_modelopt-0.37.0-py3-none-any.whl", hash = "sha256:3490b6d6aea3541aa5d475d81230fee627e2c16ff47bbab1cba4b80a1eb119a2", size = 831271, upload-time = "2025-10-08T18:37:03.951Z" }, ] [[package]] @@ -4785,26 +4770,46 @@ dependencies = [ ] sdist = { url = "https://files.pythonhosted.org/packages/0f/37/6964b830433e654ec7485e45a00fc9a27cf868d622838f6b6d9c5ec0d532/scipy-1.15.3.tar.gz", hash = "sha256:eae3cf522bc7df64b42cad3925c876e1b0b6c35c1337c93e12c0f366f55b0eaf", size = 59419214, upload-time = "2025-05-08T16:13:05.955Z" } wheels = [ + { url = "https://files.pythonhosted.org/packages/78/2f/4966032c5f8cc7e6a60f1b2e0ad686293b9474b65246b0c642e3ef3badd0/scipy-1.15.3-cp310-cp310-macosx_10_13_x86_64.whl", hash = "sha256:a345928c86d535060c9c2b25e71e87c39ab2f22fc96e9636bd74d1dbf9de448c", size = 38702770, upload-time = "2025-05-08T16:04:20.849Z" }, + { url = "https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:ad3432cb0f9ed87477a8d97f03b763fd1d57709f1bbde3c9369b1dff5503b253", size = 30094511, upload-time = "2025-05-08T16:04:27.103Z" }, + { url = "https://files.pythonhosted.org/packages/ea/b1/4deb37252311c1acff7f101f6453f0440794f51b6eacb1aad4459a134081/scipy-1.15.3-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:aef683a9ae6eb00728a542b796f52a5477b78252edede72b8327a886ab63293f", size = 22368151, upload-time = "2025-05-08T16:04:31.731Z" }, + { url = "https://files.pythonhosted.org/packages/38/7d/f457626e3cd3c29b3a49ca115a304cebb8cc6f31b04678f03b216899d3c6/scipy-1.15.3-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:1c832e1bd78dea67d5c16f786681b28dd695a8cb1fb90af2e27580d3d0967e92", size = 25121732, upload-time = "2025-05-08T16:04:36.596Z" }, { url = "https://files.pythonhosted.org/packages/db/0a/92b1de4a7adc7a15dcf5bddc6e191f6f29ee663b30511ce20467ef9b82e4/scipy-1.15.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:263961f658ce2165bbd7b99fa5135195c3a12d9bef045345016b8b50c315cb82", size = 35547617, upload-time = "2025-05-08T16:04:43.546Z" }, { url = "https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9e2abc762b0811e09a0d3258abee2d98e0c703eee49464ce0069590846f31d40", size = 37662964, upload-time = "2025-05-08T16:04:49.431Z" }, { url = "https://files.pythonhosted.org/packages/25/e1/3df8f83cb15f3500478c889be8fb18700813b95e9e087328230b98d547ff/scipy-1.15.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:ed7284b21a7a0c8f1b6e5977ac05396c0d008b89e05498c8b7e8f4a1423bba0e", size = 37238749, upload-time = "2025-05-08T16:04:55.215Z" }, { url = "https://files.pythonhosted.org/packages/93/3e/b3257cf446f2a3533ed7809757039016b74cd6f38271de91682aa844cfc5/scipy-1.15.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5380741e53df2c566f4d234b100a484b420af85deb39ea35a1cc1be84ff53a5c", size = 40022383, upload-time = "2025-05-08T16:05:01.914Z" }, { url = "https://files.pythonhosted.org/packages/d1/84/55bc4881973d3f79b479a5a2e2df61c8c9a04fcb986a213ac9c02cfb659b/scipy-1.15.3-cp310-cp310-win_amd64.whl", hash = "sha256:9d61e97b186a57350f6d6fd72640f9e99d5a4a2b8fbf4b9ee9a841eab327dc13", size = 41259201, upload-time = "2025-05-08T16:05:08.166Z" }, + { url = "https://files.pythonhosted.org/packages/96/ab/5cc9f80f28f6a7dff646c5756e559823614a42b1939d86dd0ed550470210/scipy-1.15.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:993439ce220d25e3696d1b23b233dd010169b62f6456488567e830654ee37a6b", size = 38714255, upload-time = "2025-05-08T16:05:14.596Z" }, + { url = "https://files.pythonhosted.org/packages/4a/4a/66ba30abe5ad1a3ad15bfb0b59d22174012e8056ff448cb1644deccbfed2/scipy-1.15.3-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:34716e281f181a02341ddeaad584205bd2fd3c242063bd3423d61ac259ca7eba", size = 30111035, upload-time = "2025-05-08T16:05:20.152Z" }, + { url = "https://files.pythonhosted.org/packages/4b/fa/a7e5b95afd80d24313307f03624acc65801846fa75599034f8ceb9e2cbf6/scipy-1.15.3-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:3b0334816afb8b91dab859281b1b9786934392aa3d527cd847e41bb6f45bee65", size = 22384499, upload-time = "2025-05-08T16:05:24.494Z" }, + { url = "https://files.pythonhosted.org/packages/17/99/f3aaddccf3588bb4aea70ba35328c204cadd89517a1612ecfda5b2dd9d7a/scipy-1.15.3-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:6db907c7368e3092e24919b5e31c76998b0ce1684d51a90943cb0ed1b4ffd6c1", size = 25152602, upload-time = "2025-05-08T16:05:29.313Z" }, { url = "https://files.pythonhosted.org/packages/56/c5/1032cdb565f146109212153339f9cb8b993701e9fe56b1c97699eee12586/scipy-1.15.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:721d6b4ef5dc82ca8968c25b111e307083d7ca9091bc38163fb89243e85e3889", size = 35503415, upload-time = "2025-05-08T16:05:34.699Z" }, { url = "https://files.pythonhosted.org/packages/bd/37/89f19c8c05505d0601ed5650156e50eb881ae3918786c8fd7262b4ee66d3/scipy-1.15.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39cb9c62e471b1bb3750066ecc3a3f3052b37751c7c3dfd0fd7e48900ed52982", size = 37652622, upload-time = "2025-05-08T16:05:40.762Z" }, { url = "https://files.pythonhosted.org/packages/7e/31/be59513aa9695519b18e1851bb9e487de66f2d31f835201f1b42f5d4d475/scipy-1.15.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:795c46999bae845966368a3c013e0e00947932d68e235702b5c3f6ea799aa8c9", size = 37244796, upload-time = "2025-05-08T16:05:48.119Z" }, { url = "https://files.pythonhosted.org/packages/10/c0/4f5f3eeccc235632aab79b27a74a9130c6c35df358129f7ac8b29f562ac7/scipy-1.15.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:18aaacb735ab38b38db42cb01f6b92a2d0d4b6aabefeb07f02849e47f8fb3594", size = 40047684, upload-time = "2025-05-08T16:05:54.22Z" }, { url = "https://files.pythonhosted.org/packages/ab/a7/0ddaf514ce8a8714f6ed243a2b391b41dbb65251affe21ee3077ec45ea9a/scipy-1.15.3-cp311-cp311-win_amd64.whl", hash = "sha256:ae48a786a28412d744c62fd7816a4118ef97e5be0bee968ce8f0a2fba7acf3bb", size = 41246504, upload-time = "2025-05-08T16:06:00.437Z" }, + { url = "https://files.pythonhosted.org/packages/37/4b/683aa044c4162e10ed7a7ea30527f2cbd92e6999c10a8ed8edb253836e9c/scipy-1.15.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6ac6310fdbfb7aa6612408bd2f07295bcbd3fda00d2d702178434751fe48e019", size = 38766735, upload-time = "2025-05-08T16:06:06.471Z" }, + { url = "https://files.pythonhosted.org/packages/7b/7e/f30be3d03de07f25dc0ec926d1681fed5c732d759ac8f51079708c79e680/scipy-1.15.3-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:185cd3d6d05ca4b44a8f1595af87f9c372bb6acf9c808e99aa3e9aa03bd98cf6", size = 30173284, upload-time = "2025-05-08T16:06:11.686Z" }, + { url = "https://files.pythonhosted.org/packages/07/9c/0ddb0d0abdabe0d181c1793db51f02cd59e4901da6f9f7848e1f96759f0d/scipy-1.15.3-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:05dc6abcd105e1a29f95eada46d4a3f251743cfd7d3ae8ddb4088047f24ea477", size = 22446958, upload-time = "2025-05-08T16:06:15.97Z" }, + { url = "https://files.pythonhosted.org/packages/af/43/0bce905a965f36c58ff80d8bea33f1f9351b05fad4beaad4eae34699b7a1/scipy-1.15.3-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:06efcba926324df1696931a57a176c80848ccd67ce6ad020c810736bfd58eb1c", size = 25242454, upload-time = "2025-05-08T16:06:20.394Z" }, { url = "https://files.pythonhosted.org/packages/56/30/a6f08f84ee5b7b28b4c597aca4cbe545535c39fe911845a96414700b64ba/scipy-1.15.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c05045d8b9bfd807ee1b9f38761993297b10b245f012b11b13b91ba8945f7e45", size = 35210199, upload-time = "2025-05-08T16:06:26.159Z" }, { url = "https://files.pythonhosted.org/packages/0b/1f/03f52c282437a168ee2c7c14a1a0d0781a9a4a8962d84ac05c06b4c5b555/scipy-1.15.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:271e3713e645149ea5ea3e97b57fdab61ce61333f97cfae392c28ba786f9bb49", size = 37309455, upload-time = "2025-05-08T16:06:32.778Z" }, { url = "https://files.pythonhosted.org/packages/89/b1/fbb53137f42c4bf630b1ffdfc2151a62d1d1b903b249f030d2b1c0280af8/scipy-1.15.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6cfd56fc1a8e53f6e89ba3a7a7251f7396412d655bca2aa5611c8ec9a6784a1e", size = 36885140, upload-time = "2025-05-08T16:06:39.249Z" }, { url = "https://files.pythonhosted.org/packages/2e/2e/025e39e339f5090df1ff266d021892694dbb7e63568edcfe43f892fa381d/scipy-1.15.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0ff17c0bb1cb32952c09217d8d1eed9b53d1463e5f1dd6052c7857f83127d539", size = 39710549, upload-time = "2025-05-08T16:06:45.729Z" }, { url = "https://files.pythonhosted.org/packages/e6/eb/3bf6ea8ab7f1503dca3a10df2e4b9c3f6b3316df07f6c0ded94b281c7101/scipy-1.15.3-cp312-cp312-win_amd64.whl", hash = "sha256:52092bc0472cfd17df49ff17e70624345efece4e1a12b23783a1ac59a1b728ed", size = 40966184, upload-time = "2025-05-08T16:06:52.623Z" }, + { url = "https://files.pythonhosted.org/packages/73/18/ec27848c9baae6e0d6573eda6e01a602e5649ee72c27c3a8aad673ebecfd/scipy-1.15.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2c620736bcc334782e24d173c0fdbb7590a0a436d2fdf39310a8902505008759", size = 38728256, upload-time = "2025-05-08T16:06:58.696Z" }, + { url = "https://files.pythonhosted.org/packages/74/cd/1aef2184948728b4b6e21267d53b3339762c285a46a274ebb7863c9e4742/scipy-1.15.3-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:7e11270a000969409d37ed399585ee530b9ef6aa99d50c019de4cb01e8e54e62", size = 30109540, upload-time = "2025-05-08T16:07:04.209Z" }, + { url = "https://files.pythonhosted.org/packages/5b/d8/59e452c0a255ec352bd0a833537a3bc1bfb679944c4938ab375b0a6b3a3e/scipy-1.15.3-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:8c9ed3ba2c8a2ce098163a9bdb26f891746d02136995df25227a20e71c396ebb", size = 22383115, upload-time = "2025-05-08T16:07:08.998Z" }, + { url = "https://files.pythonhosted.org/packages/08/f5/456f56bbbfccf696263b47095291040655e3cbaf05d063bdc7c7517f32ac/scipy-1.15.3-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:0bdd905264c0c9cfa74a4772cdb2070171790381a5c4d312c973382fc6eaf730", size = 25163884, upload-time = "2025-05-08T16:07:14.091Z" }, { url = "https://files.pythonhosted.org/packages/a2/66/a9618b6a435a0f0c0b8a6d0a2efb32d4ec5a85f023c2b79d39512040355b/scipy-1.15.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:79167bba085c31f38603e11a267d862957cbb3ce018d8b38f79ac043bc92d825", size = 35174018, upload-time = "2025-05-08T16:07:19.427Z" }, { url = "https://files.pythonhosted.org/packages/b5/09/c5b6734a50ad4882432b6bb7c02baf757f5b2f256041da5df242e2d7e6b6/scipy-1.15.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c9deabd6d547aee2c9a81dee6cc96c6d7e9a9b1953f74850c179f91fdc729cb7", size = 37269716, upload-time = "2025-05-08T16:07:25.712Z" }, { url = "https://files.pythonhosted.org/packages/77/0a/eac00ff741f23bcabd352731ed9b8995a0a60ef57f5fd788d611d43d69a1/scipy-1.15.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:dde4fc32993071ac0c7dd2d82569e544f0bdaff66269cb475e0f369adad13f11", size = 36872342, upload-time = "2025-05-08T16:07:31.468Z" }, { url = "https://files.pythonhosted.org/packages/fe/54/4379be86dd74b6ad81551689107360d9a3e18f24d20767a2d5b9253a3f0a/scipy-1.15.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f77f853d584e72e874d87357ad70f44b437331507d1c311457bed8ed2b956126", size = 39670869, upload-time = "2025-05-08T16:07:38.002Z" }, { url = "https://files.pythonhosted.org/packages/87/2e/892ad2862ba54f084ffe8cc4a22667eaf9c2bcec6d2bff1d15713c6c0703/scipy-1.15.3-cp313-cp313-win_amd64.whl", hash = "sha256:b90ab29d0c37ec9bf55424c064312930ca5f4bde15ee8619ee44e69319aab163", size = 40988851, upload-time = "2025-05-08T16:08:33.671Z" }, + { url = "https://files.pythonhosted.org/packages/1b/e9/7a879c137f7e55b30d75d90ce3eb468197646bc7b443ac036ae3fe109055/scipy-1.15.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3ac07623267feb3ae308487c260ac684b32ea35fd81e12845039952f558047b8", size = 38863011, upload-time = "2025-05-08T16:07:44.039Z" }, + { url = "https://files.pythonhosted.org/packages/51/d1/226a806bbd69f62ce5ef5f3ffadc35286e9fbc802f606a07eb83bf2359de/scipy-1.15.3-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:6487aa99c2a3d509a5227d9a5e889ff05830a06b2ce08ec30df6d79db5fcd5c5", size = 30266407, upload-time = "2025-05-08T16:07:49.891Z" }, + { url = "https://files.pythonhosted.org/packages/e5/9b/f32d1d6093ab9eeabbd839b0f7619c62e46cc4b7b6dbf05b6e615bbd4400/scipy-1.15.3-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:50f9e62461c95d933d5c5ef4a1f2ebf9a2b4e83b0db374cb3f1de104d935922e", size = 22540030, upload-time = "2025-05-08T16:07:54.121Z" }, + { url = "https://files.pythonhosted.org/packages/e7/29/c278f699b095c1a884f29fda126340fcc201461ee8bfea5c8bdb1c7c958b/scipy-1.15.3-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:14ed70039d182f411ffc74789a16df3835e05dc469b898233a245cdfd7f162cb", size = 25218709, upload-time = "2025-05-08T16:07:58.506Z" }, { url = "https://files.pythonhosted.org/packages/24/18/9e5374b617aba742a990581373cd6b68a2945d65cc588482749ef2e64467/scipy-1.15.3-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a769105537aa07a69468a0eefcd121be52006db61cdd8cac8a0e68980bbb723", size = 34809045, upload-time = "2025-05-08T16:08:03.929Z" }, { url = "https://files.pythonhosted.org/packages/e1/fe/9c4361e7ba2927074360856db6135ef4904d505e9b3afbbcb073c4008328/scipy-1.15.3-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9db984639887e3dffb3928d118145ffe40eff2fa40cb241a306ec57c219ebbbb", size = 36703062, upload-time = "2025-05-08T16:08:09.558Z" }, { url = "https://files.pythonhosted.org/packages/b7/8e/038ccfe29d272b30086b25a4960f757f97122cb2ec42e62b460d02fe98e9/scipy-1.15.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:40e54d5c7e7ebf1aa596c374c49fa3135f04648a0caabcb66c52884b943f02b4", size = 36393132, upload-time = "2025-05-08T16:08:15.34Z" }, @@ -4835,36 +4840,60 @@ dependencies = [ ] sdist = { url = "https://files.pythonhosted.org/packages/4c/3b/546a6f0bfe791bbb7f8d591613454d15097e53f906308ec6f7c1ce588e8e/scipy-1.16.2.tar.gz", hash = "sha256:af029b153d243a80afb6eabe40b0a07f8e35c9adc269c019f364ad747f826a6b", size = 30580599, upload-time = "2025-09-11T17:48:08.271Z" } wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/ef/37ed4b213d64b48422df92560af7300e10fe30b5d665dd79932baebee0c6/scipy-1.16.2-cp311-cp311-macosx_10_14_x86_64.whl", hash = "sha256:6ab88ea43a57da1af33292ebd04b417e8e2eaf9d5aa05700be8d6e1b6501cd92", size = 36619956, upload-time = "2025-09-11T17:39:20.5Z" }, + { url = "https://files.pythonhosted.org/packages/85/ab/5c2eba89b9416961a982346a4d6a647d78c91ec96ab94ed522b3b6baf444/scipy-1.16.2-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:c95e96c7305c96ede73a7389f46ccd6c659c4da5ef1b2789466baeaed3622b6e", size = 28931117, upload-time = "2025-09-11T17:39:29.06Z" }, + { url = "https://files.pythonhosted.org/packages/80/d1/eed51ab64d227fe60229a2d57fb60ca5898cfa50ba27d4f573e9e5f0b430/scipy-1.16.2-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:87eb178db04ece7c698220d523c170125dbffebb7af0345e66c3554f6f60c173", size = 20921997, upload-time = "2025-09-11T17:39:34.892Z" }, + { url = "https://files.pythonhosted.org/packages/be/7c/33ea3e23bbadde96726edba6bf9111fb1969d14d9d477ffa202c67bec9da/scipy-1.16.2-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:4e409eac067dcee96a57fbcf424c13f428037827ec7ee3cb671ff525ca4fc34d", size = 23523374, upload-time = "2025-09-11T17:39:40.846Z" }, { url = "https://files.pythonhosted.org/packages/96/0b/7399dc96e1e3f9a05e258c98d716196a34f528eef2ec55aad651ed136d03/scipy-1.16.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e574be127bb760f0dad24ff6e217c80213d153058372362ccb9555a10fc5e8d2", size = 33583702, upload-time = "2025-09-11T17:39:49.011Z" }, { url = "https://files.pythonhosted.org/packages/1a/bc/a5c75095089b96ea72c1bd37a4497c24b581ec73db4ef58ebee142ad2d14/scipy-1.16.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f5db5ba6188d698ba7abab982ad6973265b74bb40a1efe1821b58c87f73892b9", size = 35883427, upload-time = "2025-09-11T17:39:57.406Z" }, { url = "https://files.pythonhosted.org/packages/ab/66/e25705ca3d2b87b97fe0a278a24b7f477b4023a926847935a1a71488a6a6/scipy-1.16.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ec6e74c4e884104ae006d34110677bfe0098203a3fec2f3faf349f4cb05165e3", size = 36212940, upload-time = "2025-09-11T17:40:06.013Z" }, { url = "https://files.pythonhosted.org/packages/d6/fd/0bb911585e12f3abdd603d721d83fc1c7492835e1401a0e6d498d7822b4b/scipy-1.16.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:912f46667d2d3834bc3d57361f854226475f695eb08c08a904aadb1c936b6a88", size = 38865092, upload-time = "2025-09-11T17:40:15.143Z" }, { url = "https://files.pythonhosted.org/packages/d6/73/c449a7d56ba6e6f874183759f8483cde21f900a8be117d67ffbb670c2958/scipy-1.16.2-cp311-cp311-win_amd64.whl", hash = "sha256:91e9e8a37befa5a69e9cacbe0bcb79ae5afb4a0b130fd6db6ee6cc0d491695fa", size = 38687626, upload-time = "2025-09-11T17:40:24.041Z" }, { url = "https://files.pythonhosted.org/packages/68/72/02f37316adf95307f5d9e579023c6899f89ff3a051fa079dbd6faafc48e5/scipy-1.16.2-cp311-cp311-win_arm64.whl", hash = "sha256:f3bf75a6dcecab62afde4d1f973f1692be013110cad5338007927db8da73249c", size = 25503506, upload-time = "2025-09-11T17:40:30.703Z" }, + { url = "https://files.pythonhosted.org/packages/b7/8d/6396e00db1282279a4ddd507c5f5e11f606812b608ee58517ce8abbf883f/scipy-1.16.2-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:89d6c100fa5c48472047632e06f0876b3c4931aac1f4291afc81a3644316bb0d", size = 36646259, upload-time = "2025-09-11T17:40:39.329Z" }, + { url = "https://files.pythonhosted.org/packages/3b/93/ea9edd7e193fceb8eef149804491890bde73fb169c896b61aa3e2d1e4e77/scipy-1.16.2-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:ca748936cd579d3f01928b30a17dc474550b01272d8046e3e1ee593f23620371", size = 28888976, upload-time = "2025-09-11T17:40:46.82Z" }, + { url = "https://files.pythonhosted.org/packages/91/4d/281fddc3d80fd738ba86fd3aed9202331180b01e2c78eaae0642f22f7e83/scipy-1.16.2-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:fac4f8ce2ddb40e2e3d0f7ec36d2a1e7f92559a2471e59aec37bd8d9de01fec0", size = 20879905, upload-time = "2025-09-11T17:40:52.545Z" }, + { url = "https://files.pythonhosted.org/packages/69/40/b33b74c84606fd301b2915f0062e45733c6ff5708d121dd0deaa8871e2d0/scipy-1.16.2-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:033570f1dcefd79547a88e18bccacff025c8c647a330381064f561d43b821232", size = 23553066, upload-time = "2025-09-11T17:40:59.014Z" }, { url = "https://files.pythonhosted.org/packages/55/a7/22c739e2f21a42cc8f16bc76b47cff4ed54fbe0962832c589591c2abec34/scipy-1.16.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ea3421209bf00c8a5ef2227de496601087d8f638a2363ee09af059bd70976dc1", size = 33336407, upload-time = "2025-09-11T17:41:06.796Z" }, { url = "https://files.pythonhosted.org/packages/53/11/a0160990b82999b45874dc60c0c183d3a3a969a563fffc476d5a9995c407/scipy-1.16.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f66bd07ba6f84cd4a380b41d1bf3c59ea488b590a2ff96744845163309ee8e2f", size = 35673281, upload-time = "2025-09-11T17:41:15.055Z" }, { url = "https://files.pythonhosted.org/packages/96/53/7ef48a4cfcf243c3d0f1643f5887c81f29fdf76911c4e49331828e19fc0a/scipy-1.16.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5e9feab931bd2aea4a23388c962df6468af3d808ddf2d40f94a81c5dc38f32ef", size = 36004222, upload-time = "2025-09-11T17:41:23.868Z" }, { url = "https://files.pythonhosted.org/packages/49/7f/71a69e0afd460049d41c65c630c919c537815277dfea214031005f474d78/scipy-1.16.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:03dfc75e52f72cf23ec2ced468645321407faad8f0fe7b1f5b49264adbc29cb1", size = 38664586, upload-time = "2025-09-11T17:41:31.021Z" }, { url = "https://files.pythonhosted.org/packages/34/95/20e02ca66fb495a95fba0642fd48e0c390d0ece9b9b14c6e931a60a12dea/scipy-1.16.2-cp312-cp312-win_amd64.whl", hash = "sha256:0ce54e07bbb394b417457409a64fd015be623f36e330ac49306433ffe04bc97e", size = 38550641, upload-time = "2025-09-11T17:41:36.61Z" }, { url = "https://files.pythonhosted.org/packages/92/ad/13646b9beb0a95528ca46d52b7babafbe115017814a611f2065ee4e61d20/scipy-1.16.2-cp312-cp312-win_arm64.whl", hash = "sha256:2a8ffaa4ac0df81a0b94577b18ee079f13fecdb924df3328fc44a7dc5ac46851", size = 25456070, upload-time = "2025-09-11T17:41:41.3Z" }, + { url = "https://files.pythonhosted.org/packages/c1/27/c5b52f1ee81727a9fc457f5ac1e9bf3d6eab311805ea615c83c27ba06400/scipy-1.16.2-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:84f7bf944b43e20b8a894f5fe593976926744f6c185bacfcbdfbb62736b5cc70", size = 36604856, upload-time = "2025-09-11T17:41:47.695Z" }, + { url = "https://files.pythonhosted.org/packages/32/a9/15c20d08e950b540184caa8ced675ba1128accb0e09c653780ba023a4110/scipy-1.16.2-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5c39026d12edc826a1ef2ad35ad1e6d7f087f934bb868fc43fa3049c8b8508f9", size = 28864626, upload-time = "2025-09-11T17:41:52.642Z" }, + { url = "https://files.pythonhosted.org/packages/4c/fc/ea36098df653cca26062a627c1a94b0de659e97127c8491e18713ca0e3b9/scipy-1.16.2-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:e52729ffd45b68777c5319560014d6fd251294200625d9d70fd8626516fc49f5", size = 20855689, upload-time = "2025-09-11T17:41:57.886Z" }, + { url = "https://files.pythonhosted.org/packages/dc/6f/d0b53be55727f3e6d7c72687ec18ea6d0047cf95f1f77488b99a2bafaee1/scipy-1.16.2-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:024dd4a118cccec09ca3209b7e8e614931a6ffb804b2a601839499cb88bdf925", size = 23512151, upload-time = "2025-09-11T17:42:02.303Z" }, { url = "https://files.pythonhosted.org/packages/11/85/bf7dab56e5c4b1d3d8eef92ca8ede788418ad38a7dc3ff50262f00808760/scipy-1.16.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7a5dc7ee9c33019973a470556081b0fd3c9f4c44019191039f9769183141a4d9", size = 33329824, upload-time = "2025-09-11T17:42:07.549Z" }, { url = "https://files.pythonhosted.org/packages/da/6a/1a927b14ddc7714111ea51f4e568203b2bb6ed59bdd036d62127c1a360c8/scipy-1.16.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c2275ff105e508942f99d4e3bc56b6ef5e4b3c0af970386ca56b777608ce95b7", size = 35681881, upload-time = "2025-09-11T17:42:13.255Z" }, { url = "https://files.pythonhosted.org/packages/c1/5f/331148ea5780b4fcc7007a4a6a6ee0a0c1507a796365cc642d4d226e1c3a/scipy-1.16.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:af80196eaa84f033e48444d2e0786ec47d328ba00c71e4299b602235ffef9acb", size = 36006219, upload-time = "2025-09-11T17:42:18.765Z" }, { url = "https://files.pythonhosted.org/packages/46/3a/e991aa9d2aec723b4a8dcfbfc8365edec5d5e5f9f133888067f1cbb7dfc1/scipy-1.16.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9fb1eb735fe3d6ed1f89918224e3385fbf6f9e23757cacc35f9c78d3b712dd6e", size = 38682147, upload-time = "2025-09-11T17:42:25.177Z" }, { url = "https://files.pythonhosted.org/packages/a1/57/0f38e396ad19e41b4c5db66130167eef8ee620a49bc7d0512e3bb67e0cab/scipy-1.16.2-cp313-cp313-win_amd64.whl", hash = "sha256:fda714cf45ba43c9d3bae8f2585c777f64e3f89a2e073b668b32ede412d8f52c", size = 38520766, upload-time = "2025-09-11T17:43:25.342Z" }, { url = "https://files.pythonhosted.org/packages/1b/a5/85d3e867b6822d331e26c862a91375bb7746a0b458db5effa093d34cdb89/scipy-1.16.2-cp313-cp313-win_arm64.whl", hash = "sha256:2f5350da923ccfd0b00e07c3e5cfb316c1c0d6c1d864c07a72d092e9f20db104", size = 25451169, upload-time = "2025-09-11T17:43:30.198Z" }, + { url = "https://files.pythonhosted.org/packages/09/d9/60679189bcebda55992d1a45498de6d080dcaf21ce0c8f24f888117e0c2d/scipy-1.16.2-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:53d8d2ee29b925344c13bda64ab51785f016b1b9617849dac10897f0701b20c1", size = 37012682, upload-time = "2025-09-11T17:42:30.677Z" }, + { url = "https://files.pythonhosted.org/packages/83/be/a99d13ee4d3b7887a96f8c71361b9659ba4ef34da0338f14891e102a127f/scipy-1.16.2-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:9e05e33657efb4c6a9d23bd8300101536abd99c85cca82da0bffff8d8764d08a", size = 29389926, upload-time = "2025-09-11T17:42:35.845Z" }, + { url = "https://files.pythonhosted.org/packages/bf/0a/130164a4881cec6ca8c00faf3b57926f28ed429cd6001a673f83c7c2a579/scipy-1.16.2-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:7fe65b36036357003b3ef9d37547abeefaa353b237e989c21027b8ed62b12d4f", size = 21381152, upload-time = "2025-09-11T17:42:40.07Z" }, + { url = "https://files.pythonhosted.org/packages/47/a6/503ffb0310ae77fba874e10cddfc4a1280bdcca1d13c3751b8c3c2996cf8/scipy-1.16.2-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:6406d2ac6d40b861cccf57f49592f9779071655e9f75cd4f977fa0bdd09cb2e4", size = 23914410, upload-time = "2025-09-11T17:42:44.313Z" }, { url = "https://files.pythonhosted.org/packages/fa/c7/1147774bcea50d00c02600aadaa919facbd8537997a62496270133536ed6/scipy-1.16.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ff4dc42bd321991fbf611c23fc35912d690f731c9914bf3af8f417e64aca0f21", size = 33481880, upload-time = "2025-09-11T17:42:49.325Z" }, { url = "https://files.pythonhosted.org/packages/6a/74/99d5415e4c3e46b2586f30cdbecb95e101c7192628a484a40dd0d163811a/scipy-1.16.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:654324826654d4d9133e10675325708fb954bc84dae6e9ad0a52e75c6b1a01d7", size = 35791425, upload-time = "2025-09-11T17:42:54.711Z" }, { url = "https://files.pythonhosted.org/packages/1b/ee/a6559de7c1cc710e938c0355d9d4fbcd732dac4d0d131959d1f3b63eb29c/scipy-1.16.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:63870a84cd15c44e65220eaed2dac0e8f8b26bbb991456a033c1d9abfe8a94f8", size = 36178622, upload-time = "2025-09-11T17:43:00.375Z" }, { url = "https://files.pythonhosted.org/packages/4e/7b/f127a5795d5ba8ece4e0dce7d4a9fb7cb9e4f4757137757d7a69ab7d4f1a/scipy-1.16.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:fa01f0f6a3050fa6a9771a95d5faccc8e2f5a92b4a2e5440a0fa7264a2398472", size = 38783985, upload-time = "2025-09-11T17:43:06.661Z" }, { url = "https://files.pythonhosted.org/packages/3e/9f/bc81c1d1e033951eb5912cd3750cc005943afa3e65a725d2443a3b3c4347/scipy-1.16.2-cp313-cp313t-win_amd64.whl", hash = "sha256:116296e89fba96f76353a8579820c2512f6e55835d3fad7780fece04367de351", size = 38631367, upload-time = "2025-09-11T17:43:14.44Z" }, { url = "https://files.pythonhosted.org/packages/d6/5e/2cc7555fd81d01814271412a1d59a289d25f8b63208a0a16c21069d55d3e/scipy-1.16.2-cp313-cp313t-win_arm64.whl", hash = "sha256:98e22834650be81d42982360382b43b17f7ba95e0e6993e2a4f5b9ad9283a94d", size = 25787992, upload-time = "2025-09-11T17:43:19.745Z" }, + { url = "https://files.pythonhosted.org/packages/8b/ac/ad8951250516db71619f0bd3b2eb2448db04b720a003dd98619b78b692c0/scipy-1.16.2-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:567e77755019bb7461513c87f02bb73fb65b11f049aaaa8ca17cfaa5a5c45d77", size = 36595109, upload-time = "2025-09-11T17:43:35.713Z" }, + { url = "https://files.pythonhosted.org/packages/ff/f6/5779049ed119c5b503b0f3dc6d6f3f68eefc3a9190d4ad4c276f854f051b/scipy-1.16.2-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:17d9bb346194e8967296621208fcdfd39b55498ef7d2f376884d5ac47cec1a70", size = 28859110, upload-time = "2025-09-11T17:43:40.814Z" }, + { url = "https://files.pythonhosted.org/packages/82/09/9986e410ae38bf0a0c737ff8189ac81a93b8e42349aac009891c054403d7/scipy-1.16.2-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:0a17541827a9b78b777d33b623a6dcfe2ef4a25806204d08ead0768f4e529a88", size = 20850110, upload-time = "2025-09-11T17:43:44.981Z" }, + { url = "https://files.pythonhosted.org/packages/0d/ad/485cdef2d9215e2a7df6d61b81d2ac073dfacf6ae24b9ae87274c4e936ae/scipy-1.16.2-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:d7d4c6ba016ffc0f9568d012f5f1eb77ddd99412aea121e6fa8b4c3b7cbad91f", size = 23497014, upload-time = "2025-09-11T17:43:49.074Z" }, { url = "https://files.pythonhosted.org/packages/a7/74/f6a852e5d581122b8f0f831f1d1e32fb8987776ed3658e95c377d308ed86/scipy-1.16.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9702c4c023227785c779cba2e1d6f7635dbb5b2e0936cdd3a4ecb98d78fd41eb", size = 33401155, upload-time = "2025-09-11T17:43:54.661Z" }, { url = "https://files.pythonhosted.org/packages/d9/f5/61d243bbc7c6e5e4e13dde9887e84a5cbe9e0f75fd09843044af1590844e/scipy-1.16.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d1cdf0ac28948d225decdefcc45ad7dd91716c29ab56ef32f8e0d50657dffcc7", size = 35691174, upload-time = "2025-09-11T17:44:00.101Z" }, { url = "https://files.pythonhosted.org/packages/03/99/59933956331f8cc57e406cdb7a483906c74706b156998f322913e789c7e1/scipy-1.16.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:70327d6aa572a17c2941cdfb20673f82e536e91850a2e4cb0c5b858b690e1548", size = 36070752, upload-time = "2025-09-11T17:44:05.619Z" }, { url = "https://files.pythonhosted.org/packages/c6/7d/00f825cfb47ee19ef74ecf01244b43e95eae74e7e0ff796026ea7cd98456/scipy-1.16.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5221c0b2a4b58aa7c4ed0387d360fd90ee9086d383bb34d9f2789fafddc8a936", size = 38701010, upload-time = "2025-09-11T17:44:11.322Z" }, { url = "https://files.pythonhosted.org/packages/e4/9f/b62587029980378304ba5a8563d376c96f40b1e133daacee76efdcae32de/scipy-1.16.2-cp314-cp314-win_amd64.whl", hash = "sha256:f5a85d7b2b708025af08f060a496dd261055b617d776fc05a1a1cc69e09fe9ff", size = 39360061, upload-time = "2025-09-11T17:45:09.814Z" }, { url = "https://files.pythonhosted.org/packages/82/04/7a2f1609921352c7fbee0815811b5050582f67f19983096c4769867ca45f/scipy-1.16.2-cp314-cp314-win_arm64.whl", hash = "sha256:2cc73a33305b4b24556957d5857d6253ce1e2dcd67fa0ff46d87d1670b3e1e1d", size = 26126914, upload-time = "2025-09-11T17:45:14.73Z" }, + { url = "https://files.pythonhosted.org/packages/51/b9/60929ce350c16b221928725d2d1d7f86cf96b8bc07415547057d1196dc92/scipy-1.16.2-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:9ea2a3fed83065d77367775d689401a703d0f697420719ee10c0780bcab594d8", size = 37013193, upload-time = "2025-09-11T17:44:16.757Z" }, + { url = "https://files.pythonhosted.org/packages/2a/41/ed80e67782d4bc5fc85a966bc356c601afddd175856ba7c7bb6d9490607e/scipy-1.16.2-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:7280d926f11ca945c3ef92ba960fa924e1465f8d07ce3a9923080363390624c4", size = 29390172, upload-time = "2025-09-11T17:44:21.783Z" }, + { url = "https://files.pythonhosted.org/packages/c4/a3/2f673ace4090452696ccded5f5f8efffb353b8f3628f823a110e0170b605/scipy-1.16.2-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:8afae1756f6a1fe04636407ef7dbece33d826a5d462b74f3d0eb82deabefd831", size = 21381326, upload-time = "2025-09-11T17:44:25.982Z" }, + { url = "https://files.pythonhosted.org/packages/42/bf/59df61c5d51395066c35836b78136accf506197617c8662e60ea209881e1/scipy-1.16.2-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:5c66511f29aa8d233388e7416a3f20d5cae7a2744d5cee2ecd38c081f4e861b3", size = 23915036, upload-time = "2025-09-11T17:44:30.527Z" }, { url = "https://files.pythonhosted.org/packages/91/c3/edc7b300dc16847ad3672f1a6f3f7c5d13522b21b84b81c265f4f2760d4a/scipy-1.16.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:efe6305aeaa0e96b0ccca5ff647a43737d9a092064a3894e46c414db84bc54ac", size = 33484341, upload-time = "2025-09-11T17:44:35.981Z" }, { url = "https://files.pythonhosted.org/packages/26/c7/24d1524e72f06ff141e8d04b833c20db3021020563272ccb1b83860082a9/scipy-1.16.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7f3a337d9ae06a1e8d655ee9d8ecb835ea5ddcdcbd8d23012afa055ab014f374", size = 35790840, upload-time = "2025-09-11T17:44:41.76Z" }, { url = "https://files.pythonhosted.org/packages/aa/b7/5aaad984eeedd56858dc33d75efa59e8ce798d918e1033ef62d2708f2c3d/scipy-1.16.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bab3605795d269067d8ce78a910220262711b753de8913d3deeaedb5dded3bb6", size = 36174716, upload-time = "2025-09-11T17:44:47.316Z" },