feat: Support PEFT weight mapping and merge LoRA adapters when export to hf

[HollowMan6]

What does this PR do ?

Now we support PEFT weight mapping (to_warp), and ignore those adapters export for now, but merge LoRA adapters to the base weights when export to hf. The linear_in tensors for LoRA adapters needs to be gathered before we merge lora using the transformation.

Changelog

• Support PEFT weight mapping
• Merge LoRA adapters when export to hf

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

• Related to [peft] Add support for LoRA merge #418
• Resolves Support load_hf_weights and export_hf_weights for PEFT/LoRA enabled models #1272

_{✨ Presented to you with Mind Lab - A Lab for Experiential Intelligence.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

@HollowMan6 : we have a plan to do this in 2 steps. We are going to provide a merging script, then you can use regular export to export the merged weights.
Mixing together will reduce the readability of the code. Is there a strong reason you need to do it in single step?

[HollowMan6]

Is there a strong reason you need to do it in single step?

@yaoyu-33 Yes, in RL training, it’s better to do it in a single step so that merged model weights can be updated to rollout engines in a streaming manner without any hassle. Please check verl-project/verl#4063 (comment)

[yaoyu-33]

@HollowMan6 : we understand the need now, can help you merge this.
Please check our comments, see if it makes sense to you.

[cuichenx]

Thanks for your contribution! The overall logic and structure look good to me. Some comments below. Could you also add a unit test that covers the newly added code in model bridge?

[HollowMan6]

Thank you for reviewing @cuichenx and @yaoyu-33 ! I have just addressed all your change requests and added straightforward unit tests for this feature. Please let me know if you still have any other concerns or suggestions, thanks!

[cuichenx]

LGTM

[cuichenx]

/ok to test 01599fc

[cuichenx]

/ok to test 33eeac6