Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove upblock3 workaround from UNet Shallow forward pass #16601

Open
esmalTT opened this issue Jan 10, 2025 · 2 comments
Open

Remove upblock3 workaround from UNet Shallow forward pass #16601

esmalTT opened this issue Jan 10, 2025 · 2 comments
Assignees
Labels

Comments

@esmalTT
Copy link
Contributor

esmalTT commented Jan 10, 2025

Summary

Since 282a7b2, a temporary workaround was required get the model working correctly:

c2_residual = ttnn.to_memory_config(c2_residual, ttnn.L1_MEMORY_CONFIG)

The temporary workaround was to move upblock3 inputs to interleaved, which is a small performance regression. The larger issue is that this is a band-aid for a different issue, which will likely resurface when making any additional changes to the model.

Without the workaround, the model first has bad PCC on upblock3 ttnn.concat. Interestingly it seems that each board reset causes this failing op PCC to change, which might indicate some kind of out of bounds memory access. Enabling TT_METAL_CLEAR_L1=1 makes it consistent.

@pavlejosipovic
Copy link
Contributor

Model doesn't "crash" right it just fails with PCC error

@esmalTT
Copy link
Contributor Author

esmalTT commented Jan 10, 2025

Running the model using comparison mode turns up an issue with to_layout:

models/experimental/functional_unet/tt/unet_shallow_ttnn.py:518: in __call__
    x = self.upblock1(x, c4_residual)
models/experimental/functional_unet/tt/unet_shallow_ttnn.py:304: in __call__
    residual = ttnn.to_layout(residual, ttnn.ROW_MAJOR_LAYOUT)
RuntimeError: ttnn.to_layout: Comparing output tensor 0 against CPU locally failed: pcc is 0.9840820378995527 but should be >=0.9990000128746033

This may or may not be related, need to investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants