Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FIx Nondeterministic Failures of PCC #1795

Closed
ctodTT opened this issue Jan 15, 2025 · 0 comments · Fixed by #1796
Closed

FIx Nondeterministic Failures of PCC #1795

ctodTT opened this issue Jan 15, 2025 · 0 comments · Fixed by #1796
Assignees
@ctodTT
Labels
bug Something isn't working

Comments

@ctodTT
Copy link
Contributor

ctodTT commented Jan 15, 2025

Nondeterministically, ops will fail their PCC checks. This is most likely due to an error during the process of copying over output tensors from device, or the processing done on said tensors prior to the PCC check, as it is nondeterministic and thus most likely due to some introduction of uninitialized memory
@ctodTT ctodTT added the bug Something isn't working label Jan 15, 2025
@ctodTT ctodTT self-assigned this Jan 15, 2025
ctodTT added a commit that referenced this issue Jan 15, 2025
@ctodTT
Fix tensor post processing
0b18496 
This change fixes an issue with processing output tensors after they are
pulled off the device when checking golden-ness. The resulting tensor
had the first 32 bytes overwritten with seemingly uninitialized memory,
resulting in nondeterministic failures during similarity checks. Now,
the unadulterated tensor is returned.

Fixes #1795
@ctodTT ctodTT mentioned this issue Jan 15, 2025
Merged
ctodTT added a commit that referenced this issue Jan 16, 2025
@ctodTT
Fix tensor post processing
e61cd66 
This change fixes an issue with processing output tensors after they are
pulled off the device when checking golden-ness. The resulting tensor
had the first 32 bytes overwritten with seemingly uninitialized memory,
resulting in nondeterministic failures during similarity checks. Now,
the unadulterated tensor is returned.

Fixes #1795
@ctodTT ctodTT closed this as completed in 9278f5d Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ctodTT ctodTT

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Post From-Device Copy Processing Issue tenstorrent/tt-mlir
1 participant
@ctodTT