Skip to content

Conversation

@mar-muel
Copy link
Contributor

@mar-muel mar-muel commented Jul 18, 2023

What does this PR do?

Adds Flax implementation for SD-XL pipeline 🚀

Related issue: #4007

This is still WIP unfortunately 😅

  • I've managed to convert the weights using from_pt=True, however this might currently not work out of the box for
    text_encoder_2 (as transformers currently doesn't support CLIPTextModelWithProjection for Flax). I've pushed my version of the converted flax weights to this repo for now: https://huggingface.co/nyxai-lab/sdxl-0.9-flax/tree/main
  • I've confirmed forward passes for UNet and VAE to work correctly. It seems the text encoders currently yield correct results for hidden states, but still getting different results for pooled layers EDIT: Upon checking again the outputs seem close enough
  • Refiner stage and part of txt2img is still WIP
  • I haven't looked into safety checker, watermark, docs or tests yet

Hope this can serve as a starting point :)

EDIT: Adding some outputs here (based on this PR):

a beautiful stack of rocks sitting on top of a beach, a picture, red black white golden colors, chakras, packshot, stock photo
image

photo of a rhino dressed suit and tie sitting at a table in a bar with a bar stools, award winning photography, Elke vogelsang
image

@pcuenca
Copy link
Member

pcuenca commented Jul 18, 2023

Awesome @mar-muel, many thanks! We'll take a look soon!

Comment on lines 75 to 76
for tokenizer in enumerate(tokenizers):
text_inputs = self.tokenizer(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for tokenizer in enumerate(tokenizers):
text_inputs = self.tokenizer(
for tokenizer in tokenizers:
text_inputs = tokenizer(

Fixed prompt embedding shapes so they work in parallel mode. Assuming we
always have both text encoders for now, for simplicity.
@pcuenca pcuenca mentioned this pull request Jul 25, 2023
6 tasks
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label Aug 19, 2023
@github-actions github-actions bot closed this Aug 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Issues that haven't received updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants