Skip to content

Fix torch.compile for text generation#811

Merged
regisss merged 1 commit into
mainfrom
fix_torch_compile
Mar 18, 2024
Merged

Fix torch.compile for text generation#811
regisss merged 1 commit into
mainfrom
fix_torch_compile

Conversation

@regisss
Copy link
Copy Markdown
Collaborator

@regisss regisss commented Mar 17, 2024

What does this PR do?

Text-generation tests with torch.compile currently fail, this PR fixes this issue.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@regisss regisss requested a review from a user March 17, 2024 00:01
@regisss
Copy link
Copy Markdown
Collaborator Author

regisss commented Mar 17, 2024

@vivekgoe @hlahkar The two torch.compile tests for text generation (1x and 8x) now fail after #806. It seems reverting from has_fused_rms_norm to FusedRMSNorm is enough to fix it, let me know if that's okay for you.

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@regisss regisss merged commit 007e920 into main Mar 18, 2024
@regisss regisss deleted the fix_torch_compile branch March 18, 2024 17:11
@vivekgoe
Copy link
Copy Markdown
Collaborator

@regisss has_fused_rms_norm will work with Synapse 1.15.0 release content. We need to revert this change back once 1.15.0 is available, without this there will be a performance drop for Llama-FSDP fine-tuning. FYI @libinta @hlahkar

hlahkar pushed a commit to hlahkar/optimum-habana that referenced this pull request Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants