Skip to content

Add ControlNet Pipeline#585

Merged
regisss merged 12 commits into
huggingface:mainfrom
nngokhale:controlnet
Jan 30, 2024
Merged

Add ControlNet Pipeline#585
regisss merged 12 commits into
huggingface:mainfrom
nngokhale:controlnet

Conversation

@nngokhale
Copy link
Copy Markdown
Contributor

@nngokhale nngokhale commented Dec 6, 2023 •
edited
Loading

What does this PR do?

Adds ControlNet pipeline.
The port is based on GaudiStableDiffusionPipeline.

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@nngokhale nngokhale requested a review from regisss as a code owner December 6, 2023 13:44
libinta
libinta reviewed Dec 15, 2023
View reviewed changes
Comment thread examples/controlnet/example_canny2image.py Outdated
libinta
libinta reviewed Dec 15, 2023
View reviewed changes
Comment thread examples/controlnet/example_canny2image.py Outdated
libinta
libinta reviewed Dec 15, 2023
View reviewed changes
Comment thread examples/controlnet/example_canny2image.py Outdated
libinta
libinta reviewed Dec 15, 2023
View reviewed changes
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
libinta
libinta reviewed Dec 15, 2023
View reviewed changes
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
libinta
libinta reviewed Dec 15, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@libinta libinta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you rebase to latest?

vidyasiv
vidyasiv reviewed Dec 15, 2023
View reviewed changes
Comment thread examples/controlnet/example_canny2image.py Outdated
@nngokhale nngokhale force-pushed the controlnet branch 2 times, most recently from e22fb1d to 56da112 Compare December 16, 2023 06:06
@nngokhale
Copy link
Copy Markdown
Contributor Author

nngokhale commented Dec 16, 2023

can you rebase to latest?

Done

@nngokhale nngokhale force-pushed the controlnet branch 6 times, most recently from cf5d5a4 to e87c4ed Compare December 20, 2023 07:22
@nngokhale nngokhale requested a review from libinta December 21, 2023 05:59
libinta
libinta reviewed Dec 21, 2023
View reviewed changes
Comment thread examples/controlnet/README.md Outdated
Comment thread examples/controlnet/README.md Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py
from diffusers.utils.torch_utils import is_compiled_module
from ....utils import speed_metrics, HabanaProfile
from ..pipeline_utils import GaudiDiffusionPipeline

Copy link
Copy Markdown
Collaborator

@libinta libinta Dec 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use #611 as guideline to port this file. also move the pipeline_controlnet.py under pipelines

Copy link
Copy Markdown
Collaborator

@regisss regisss Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there should be things that are not necessary anymore. I'm going to try to point them out below.

Comment thread examples/controlnet/text_to_image_generation_canny.py Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated

# Workaround for Synapse 1.11 for full bf16
if bf16_full_eval:
unet.conv_in.float()
Copy link
Copy Markdown
Collaborator

@libinta libinta Dec 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove line141-142

Copy link
Copy Markdown
Collaborator

@regisss regisss Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@libinta We don't need this anymore?

Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
libinta
libinta reviewed Dec 21, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@libinta libinta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, please check https://github.com/huggingface/optimum-habana/blob/main/tests/test_diffusers.py to add test case for regression protection

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Jan 3, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

regisss
regisss reviewed Jan 3, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many comments related to #611 that should be quick to address.

Can you also run the following to make sure the code format is right please?

pip install --upgrade ruff
make style

Comment thread examples/controlnet/text_to_image_generation_canny.py Outdated
Comment thread examples/controlnet/text_to_image_generation_canny.py Outdated
Comment thread examples/controlnet/README.md Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py
from diffusers.utils.torch_utils import is_compiled_module
from ....utils import speed_metrics, HabanaProfile
from ..pipeline_utils import GaudiDiffusionPipeline

Copy link
Copy Markdown
Collaborator

@regisss regisss Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there should be things that are not necessary anymore. I'm going to try to point them out below.

Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
@nngokhale nngokhale force-pushed the controlnet branch 2 times, most recently from 200c908 to e0dab26 Compare January 12, 2024 14:52
@regisss
Copy link
Copy Markdown
Collaborator

regisss commented Jan 15, 2024

@nngokhale Can you integrate the new ControlNet example into the existing Stable Diffusion example please? It will be much easier to maintain as there is a lot of duplicated code.

@nngokhale
Copy link
Copy Markdown
Contributor Author

nngokhale commented Jan 16, 2024

@nngokhale Can you integrate the new ControlNet example into the existing Stable Diffusion example please? It will be much easier to maintain as there is a lot of duplicated code.
Yes, working on it and all other comments. Sorry for the delay, lost access to gaudi systems for couple of weeks.

@nngokhale
Copy link
Copy Markdown
Contributor Author

nngokhale commented Jan 16, 2024

There are many comments related to #611 that should be quick to address.

Can you also run the following to make sure the code format is right please?

pip install --upgrade ruff
make style

Style fixed

@nngokhale
Copy link
Copy Markdown
Contributor Author

nngokhale commented Jan 18, 2024

also, please check https://github.com/huggingface/optimum-habana/blob/main/tests/test_diffusers.py to add test case for regression protection

Added tests for ControlNet pipeline.

@nngokhale nngokhale requested a review from regisss January 19, 2024 18:51
@nngokhale
Copy link
Copy Markdown
Contributor Author

nngokhale commented Jan 19, 2024

All comments / issues have been resolved, please review.

@vidyasiv vidyasiv mentioned this pull request Jan 19, 2024
Merged
3 tasks
@nngokhale nngokhale requested a review from libinta January 22, 2024 18:08
regisss
regisss reviewed Jan 23, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few comments.

Can you also run make style please?

Do you have any throughput number to share?

Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py
Comment thread optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated
Comment thread tests/test_diffusers.py
Comment thread examples/stable-diffusion/text_to_image_generation.py
Comment thread examples/stable-diffusion/text_to_image_generation.py
@nngokhale
Copy link
Copy Markdown
Contributor Author

nngokhale commented Jan 24, 2024

I left a few comments.

Can you also run make style please?

Do you have any throughput number to share?

Resolved latest comments, ran 'make style'.

Here are my throughput numbers:

For 512x512, 1 prompt, 50 inf steps, 20 generations per prompt, 4 samples per batch, BF16 + HPU Graphs,

on Gaudi2 HL-SMI: hl-1.13.0-rc-fw-47.0.0.0, Habana Driver: 1.13.0-ee32e42, Installed Habana Torch plug-in: 1.13.0.463, Installed Optimum-Habana: 1.10.0.dev0

GaudiStableDiffusionControlNetPipeline with lllyasviel/sd-controlnet-canny and runwayml/stable-diffusion-v1-5
Speed metrics: {'generation_runtime': 112.9972, 'generation_samples_per_second': 0.693, 'generation_steps_per_second': 0.289}

For comparison
GaudiStableDiffusionPipeline with runwayml/stable-diffusion-v1-5
Speed metrics: {'generation_runtime': 83.9506, 'generation_samples_per_second': 0.975, 'generation_steps_per_second': 0.406}

@nngokhale nngokhale requested a review from regisss January 25, 2024 10:53
@regisss regisss added the run-test Run CI for PRs from external contributors label Jan 30, 2024
regisss
regisss approved these changes Jan 30, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@regisss regisss merged commit 937a411 into huggingface:main Jan 30, 2024
regisss pushed a commit that referenced this pull request Feb 12, 2024
@nngokhale @regisss
Add ControlNet Pipeline (#585)
58c4ea1
jychen21 pushed a commit to jychen21/optimum-habana that referenced this pull request Feb 27, 2024
@nngokhale
Add ControlNet Pipeline (huggingface#585)
7dcfab9
dudilester pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Feb 29, 2024
@nngokhale @dudilester
Add ControlNet Pipeline (huggingface#585)
6ebb46e
dudilester pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Feb 29, 2024
@nngokhale @dudilester
Add ControlNet Pipeline (huggingface#585)
829a2d8
HolyFalafel pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Mar 11, 2024
@nngokhale @HolyFalafel
Add ControlNet Pipeline (huggingface#585)
23da729
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@regisss regisss regisss approved these changes

@libinta libinta Awaiting requested review from libinta

+1 more reviewer

@vidyasiv vidyasiv vidyasiv left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

run-test Run CI for PRs from external contributors

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nngokhale @HuggingFaceDocBuilderDev @regisss @libinta @vidyasiv