LLAMA3 70B: LoRa enabled in all modules instead of only LinearQKV

[rhmukundan]

Disable LoRa in linear_proj, linear_fc1, and linear_fc2, retaining it solely in linear_qkv.

Summary by CodeRabbit

• Chores
  • Refined Llama 3 70B LoRA fine-tuning configuration to target specific model components, improving precision and efficiency during model adaptation.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

This change adds a LoRA configuration override to three Llama 3 70B model configuration functions, constraining PEFT target_modules to QKV (Query, Key, Value) modules. The modification is a straightforward configuration addition with no control flow changes.

Changes

Cohort / File(s)	Summary
LoRA Configuration Override scripts/performance/configs/llama/llama3_llm_finetune.py	Adds target_modules constraint to QKV across gb300, gb200, and h100 Llama 3 70B LoRA configuration functions, refining PEFT application scope.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title claims LoRa is being enabled in all modules, but the description and changes clarify it's actually being constrained to only linear_qkv, contradicting the title's assertion.	Revise the title to accurately reflect the change, such as: 'LLAMA3 70B: LoRa configuration constrained to only linear_qkv modules' or similar phrasing that matches the actual implementation.
Test Results For Major Changes	⚠️ Warning	PR contains significant LoRA configuration change but lacks documentation of test results, convergence metrics, or performance validation.	Add test results to PR description including convergence behavior, before-and-after loss curves, and performance impact with specific testing configuration details.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch rhmukundan/llama3_70b_qkv_only_lora

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[rhmukundan]

/ok to test b22d740

[rhmukundan]

/ok to test 04a9849

[rhmukundan]

/ok to test 71db322