layer selection for ia3 (NVIDIA#7417)

* layer selection for ia3 Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Sasha Meister <[email protected]>
ssh-meister · Oct 5, 2023 · 2147faa · 2147faa
1 parent f6fc39a
commit 2147faa
Showing 1 changed file with 5 additions and 0 deletions.
diff --git a/nemo/collections/nlp/models/language_modeling/megatron_gpt_peft_models.py b/nemo/collections/nlp/models/language_modeling/megatron_gpt_peft_models.py
@@ -434,6 +434,11 @@ def __init__(self, cfg: DictConfig, trainer: Trainer):
                 self.name_key_to_cfg[k] = infused_adapter_cfg
             else:
                 raise ValueError(f"PEFT Key {k} is unknown.")
+
+        self.layer_selection = cfg.peft.ia3_tuning.get("layer_selection", None)
+        if self.layer_selection is None:
+            self.layer_selection = list(range(1, cfg.num_layers + 1))
+
         super().__init__(cfg, trainer)