Skip to content

[model] Refactor Qwen3-VL and Ministral3 fine-tuning scripts #2735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kamran-nvidia merged 13 commits into from
Mar 12, 2026
Merged

[model] Refactor Qwen3-VL and Ministral3 fine-tuning scripts #2735

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
75bd3fb
Refactor and enhance Qwen3-VL and Ministral3 scripts for improved fin…
kamran-nvidia Mar 10, 2026
3fe84d5
Merge branch 'main' into kamran/variuos_fixes
kamran-nvidia Mar 10, 2026
bd61712
[model, recipe] Add Qwen3-VL 8B PEFT config with Energon dataset support
kamran-nvidia Mar 11, 2026
a53e192
[model] Update parallelism configurations for LoRA finetuning in peft…
kamran-nvidia Mar 11, 2026
0a8e330
[model] Remove redundant parallelism configuration for MoE finetuning…
kamran-nvidia Mar 11, 2026
b7b51a1
[model] Remove dataset_type argument from recipe loading functions
kamran-nvidia Mar 11, 2026
ae822c8
Fix validation parameter reference in SFT script
kamran-nvidia Mar 12, 2026
2a6a857
[docs] Add expected training dynamics section with W&B report links i…
kamran-nvidia Mar 12, 2026
37d2baa
Merge branch 'main' into kamran/variuos_fixes
kamran-nvidia Mar 12, 2026
2c1797c
fix: Linting in qwen3_vl.py
kamran-nvidia Mar 12, 2026
4011f62
Merge branch 'main' into kamran/variuos_fixes
kamran-nvidia Mar 12, 2026
b61857f
fix: Update README and add assertions in EnergonProvider; enhance Qwe…
kamran-nvidia Mar 12, 2026
2299ea6
fix: Linting
kamran-nvidia Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions examples/models/vlm/ministral3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,13 +101,13 @@ Pretraining is not verified for this model.

### Supervised Fine-Tuning (SFT)

See the [sft.sh](sft.sh) script for full parameter fine-tuning with configurable model parallelisms.
See the [sft_unpacked.sh](sft_unpacked.sh) script for full parameter fine-tuning with configurable model parallelisms.

W&B report coming soon.

### Parameter-Efficient Fine-Tuning (PEFT) with LoRA

See the [peft.sh](peft.sh) script for LoRA fine-tuning with configurable tensor and pipeline parallelism.
See the [peft_unpacked.sh](peft_unpacked.sh) script for LoRA fine-tuning with configurable tensor and pipeline parallelism.
Comment thread
kamran-nvidia marked this conversation as resolved.

W&B report coming soon.
Comment thread
kamran-nvidia marked this conversation as resolved.
Outdated

Expand Down
15 changes: 8 additions & 7 deletions examples/models/vlm/ministral3/peft.sh → ...es/models/vlm/ministral3/peft_unpacked.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Ministral-3-3B-Instruct-2512-BF16
MODEL_NAME=ministral3_3b
DATASET_NAME=cord_v2
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=1
EVAL_ITERS=10
LR=0.0002
MIN_LR=0.00002
EVAL_ITERS=20
LR=0.00005
MIN_LR=0.000005
LR_WARMUP_ITERS=10
LOG_INTERVAL=1
WANDB_PROJECT=megatron-bridge-${DATASET_NAME}
Expand All @@ -47,15 +47,15 @@ for config in "${PARALLELISM_CONFIGS[@]}"; do

echo "Running LoRA finetuning with TP=$TP, PP=$PP"
uv run --no-sync python -m torch.distributed.run --nproc_per_node=8 scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_peft_config \
--step_func vlm_step \
--peft_scheme lora \
checkpoint.pretrained_checkpoint=$PRETRAINED_CHECKPOINT \
model.seq_length=$SEQ_LENGTH \
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
Expand All @@ -65,6 +65,7 @@ for config in "${PARALLELISM_CONFIGS[@]}"; do
logger.wandb_exp_name=${MODEL_NAME}_${DATASET_NAME}_lora_tp${TP}_pp${PP} \
dataset.maker_name=make_${DATASET_NAME}_dataset \
dataset.seq_length=$SEQ_LENGTH \
dataset.pack_sequences_in_batch=False \
model.tensor_model_parallel_size=$TP \
model.pipeline_model_parallel_size=$PP
done
15 changes: 8 additions & 7 deletions examples/models/vlm/ministral3/sft.sh → ...les/models/vlm/ministral3/sft_unpacked.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Ministral-3-3B-Instruct-2512-BF16
MODEL_NAME=ministral3_3b
DATASET_NAME=cord_v2
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=1
EVAL_ITERS=10
LR=0.00005
MIN_LR=0.000005
EVAL_ITERS=20
LR=0.00001
MIN_LR=0.000001
LR_WARMUP_ITERS=10
LOG_INTERVAL=1
WANDB_PROJECT=megatron-bridge-${DATASET_NAME}
Expand All @@ -47,14 +47,14 @@ for config in "${PARALLELISM_CONFIGS[@]}"; do

echo "Running full finetuning with TP=$TP, PP=$PP"
uv run --no-sync python -m torch.distributed.run --nproc_per_node=8 scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_sft_config \
--step_func vlm_step \
checkpoint.pretrained_checkpoint=$PRETRAINED_CHECKPOINT \
model.seq_length=$SEQ_LENGTH \
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
Expand All @@ -64,6 +64,7 @@ for config in "${PARALLELISM_CONFIGS[@]}"; do
logger.wandb_exp_name=${MODEL_NAME}_${DATASET_NAME}_sft_tp${TP}_pp${PP} \
dataset.maker_name=make_${DATASET_NAME}_dataset \
dataset.seq_length=$SEQ_LENGTH \
dataset.pack_sequences_in_batch=False \
model.tensor_model_parallel_size=$TP \
model.pipeline_model_parallel_size=$PP
done
2 changes: 1 addition & 1 deletion examples/models/vlm/qwen3_vl/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ field_map:
conversation: json
```

Then, update the dataset path (`dataset.path=/path/to/energon/dataset`) in [sft_energon.sh](sft_energon.sh) and run the script.
Then, update the dataset path (`dataset.path=/path/to/energon/dataset`) in [peft_energon.sh](peft_energon.sh) and run the script.

## Evaluation

Expand Down
24 changes: 14 additions & 10 deletions examples/models/vlm/qwen3_vl/peft.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Qwen3-VL-8B-Instruct
MODEL_NAME=qwen3_vl_8b
DATASET_NAME=cord_v2
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=2
EVAL_ITERS=10
EVAL_ITERS=20
EVAL_INTERVAL=20
LR=0.00005
MIN_LR=0.000005
LR_WARMUP_ITERS=10
Expand All @@ -45,15 +46,16 @@ for pack_config in "${SEQ_PACKING_CONFIGS[@]}"; do
IFS=',' read -r EP TP PP CP <<< "$par_config"
echo "Running LoRA finetuning pack_sequences_in_batch=$pack_config with EP=$EP TP=$TP PP=$PP CP=$CP"
uv run python -m torch.distributed.run --nproc_per_node=8 scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_peft_config \
--step_func qwen3_vl_step \
--peft_scheme lora \
checkpoint.pretrained_checkpoint=$PRETRAINED_CHECKPOINT \
model.seq_length=$SEQ_LENGTH \
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
validation.eval_interval=$EVAL_INTERVAL \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
Expand All @@ -80,10 +82,11 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Qwen3-VL-30B-A3B-Instruct
MODEL_NAME=qwen3_vl_30b_a3b
DATASET_NAME=cord_v2
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=2
EVAL_ITERS=10
EVAL_ITERS=20
EVAL_INTERVAL=20
LR=0.00005
MIN_LR=0.000005
LR_WARMUP_ITERS=10
Expand All @@ -100,15 +103,16 @@ for pack_config in "${SEQ_PACKING_CONFIGS[@]}"; do
IFS=',' read -r EP TP PP CP <<< "$par_config"
echo "Running LoRA finetuning pack_sequences_in_batch=$pack_config with EP=$EP TP=$TP PP=$PP CP=$CP"
uv run python -m torch.distributed.run --nproc_per_node=8 scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_peft_config \
--step_func qwen3_vl_step \
--peft_scheme lora \
checkpoint.pretrained_checkpoint=$PRETRAINED_CHECKPOINT \
model.seq_length=$SEQ_LENGTH \
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
validation.eval_interval=$EVAL_INTERVAL \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
Expand Down
14 changes: 8 additions & 6 deletions examples/models/vlm/qwen3_vl/sft_energon.sh → examples/models/vlm/qwen3_vl/peft_energon.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Qwen3-VL-8B-Instruct
MODEL_NAME=qwen3_vl_8b
DATASET_NAME=energon
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=2
EVAL_ITERS=10
EVAL_ITERS=20
EVAL_INTERVAL=20
LR=0.00005
MIN_LR=0.000005
LR_WARMUP_ITERS=10
Expand All @@ -47,7 +48,7 @@ for pack_config in "${SEQ_PACKING_CONFIGS[@]}"; do
IFS=',' read -r EP TP PP CP N_PROC <<< "$par_config"
echo "Running LoRA finetuning pack_sequences_in_batch=$pack_config with EP=$EP TP=$TP PP=$PP CP=$CP N_PROC=$N_PROC"
uv run python -m torch.distributed.run --nproc_per_node=$N_PROC scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_peft_config \
--step_func qwen3_vl_step \
--peft_scheme lora \
--dataset_type energon \
Expand All @@ -56,11 +57,12 @@ for pack_config in "${SEQ_PACKING_CONFIGS[@]}"; do
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
validation.eval_interval=$EVAL_INTERVAL \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
checkpoint.save=${WORKSPACE}/results/${MODEL_NAME}_lora_seq_pack_${pack_config}_cp${CP} \
checkpoint.save=${WORKSPACE}/results/${MODEL_NAME}_energon_lora_seq_pack_${pack_config}_cp${CP} \
logger.log_interval=$LOG_INTERVAL \
logger.wandb_project=$WANDB_PROJECT \
logger.wandb_exp_name=${MODEL_NAME}_${DATASET_NAME}_lora_seq_pack_${pack_config}_cp${CP} \
Expand Down
30 changes: 17 additions & 13 deletions examples/models/vlm/qwen3_vl/peft_unpacked.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Qwen3-VL-8B-Instruct
MODEL_NAME=qwen3_vl_8b
DATASET_NAME=cord_v2
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
MICRO_BATCH_SIZE=1
EVAL_ITERS=10
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=2
EVAL_ITERS=20
EVAL_INTERVAL=20
LR=0.00005
MIN_LR=0.000005
LR_WARMUP_ITERS=10
Expand All @@ -43,15 +44,16 @@ for config in "${PARALLELISM_CONFIGS[@]}"; do

echo "Running LoRA finetuning with TP=$TP, PP=$PP"
uv run python -m torch.distributed.run --nproc_per_node=2 scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_peft_config \
--step_func qwen3_vl_step \
--peft_scheme lora \
checkpoint.pretrained_checkpoint=$PRETRAINED_CHECKPOINT \
model.seq_length=$SEQ_LENGTH \
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
validation.eval_interval=$EVAL_INTERVAL \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
Expand All @@ -71,33 +73,35 @@ PRETRAINED_CHECKPOINT=${WORKSPACE}/models/Qwen3-VL-30B-A3B-Instruct
MODEL_NAME=qwen3_vl_30b_a3b
DATASET_NAME=cord_v2
SEQ_LENGTH=4096
TRAIN_ITERS=50
GLOBAL_BATCH_SIZE=32
MICRO_BATCH_SIZE=1
EVAL_ITERS=10
TRAIN_ITERS=100
GLOBAL_BATCH_SIZE=16
MICRO_BATCH_SIZE=2
EVAL_ITERS=20
EVAL_INTERVAL=20
LR=0.00005
MIN_LR=0.000005
LR_WARMUP_ITERS=10
LOG_INTERVAL=1
WANDB_PROJECT=megatron-bridge-${DATASET_NAME}

# EP/TP/PP combinations: "EP,TP,PP" configurations
PARALLELISM_CONFIGS=("8,1,1" "4,1,1" "2,1,1")
PARALLELISM_CONFIGS=("8,1,1" "4,1,1" "2,2,1")

for config in "${PARALLELISM_CONFIGS[@]}"; do
IFS=',' read -r EP TP PP <<< "$config"

echo "Running LoRA finetuning with EP=$EP, TP=$TP, PP=$PP"
uv run python -m torch.distributed.run --nproc_per_node=8 scripts/training/run_recipe.py \
--recipe ${MODEL_NAME}_finetune_config \
--recipe ${MODEL_NAME}_peft_config \
--step_func qwen3_vl_step \
--peft_scheme lora \
checkpoint.pretrained_checkpoint=$PRETRAINED_CHECKPOINT \
model.seq_length=$SEQ_LENGTH \
train.train_iters=$TRAIN_ITERS \
train.global_batch_size=$GLOBAL_BATCH_SIZE \
train.micro_batch_size=$MICRO_BATCH_SIZE \
train.eval_iters=$EVAL_ITERS \
validation.eval_iters=$EVAL_ITERS \
validation.eval_interval=$EVAL_INTERVAL \
optimizer.lr=$LR \
optimizer.min_lr=$MIN_LR \
scheduler.lr_warmup_iters=$LR_WARMUP_ITERS \
Expand Down
Loading