Skip to content

Commit

Permalink
fixed code quality
Browse files Browse the repository at this point in the history
  • Loading branch information
julian-fong committed Jan 6, 2025
1 parent 3aa2493 commit f2ecd3c
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions src/adapters/configuration/adapter_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -374,13 +374,13 @@ class ParBnConfig(BnConfig):
class AdapterPlusConfig(BnConfig):
"""
The AdapterPlus config architecture proposed by Jan-Martin O, Steitz and Stefan Roth. See https://arxiv.org/pdf/2406.06820
Please note that some configurations of the adapters parameters `original_ln_after`, `original_ln_before`, and
`residual_before_ln` may result in performance issues when training.
Please note that some configurations of the adapters parameters `original_ln_after`, `original_ln_before`, and
`residual_before_ln` may result in performance issues when training.
In the general case:
1) At least one of `original_ln_before` or `original_ln_after` should be set to True in order to ensure that the original residual
connection from pre-training is preserved.
connection from pre-training is preserved.
2) If `original_ln_after` is set to `False`, `residual_before_ln` must also be set to `False` to ensure convergence during training.
"""

Expand Down

0 comments on commit f2ecd3c

Please sign in to comment.