Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

verify now returns outputs from forward pass + tests rework #1348

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

vkovinicTT
Copy link
Member

Ticket

Fix #1331

Problem description

verify() needed to return outputs from the forward pass, as there are some use cases where we need outputs of forward pass.

What's changed

Changed verify() to return outputs from framework and compiled model and also updated tests.

@vkovinicTT
Copy link
Member Author

vkovinicTT commented Feb 28, 2025

For me personally it's easier to just have:

# Forward pass (prediction) on device
pred = tt_model(data)[0]
golden_pred = framework_model(data)
assert golden_pred.dtype == dtype
assert compare_with_golden(golden_pred, pred, pcc=0.95)

then what I have currently set it to:

golden_pred, pred = verify(inputs=[data], framework_model=framework_model, compiled_model=tt_model, verify_cfg=VerifyConfig(value_checker=AutomaticValueChecker(pcc=0.95)))
pred = pred[0]

as it is not really clear based on the method name verify (it becomes clear when you read method's description though) that it returns outputs and it feels easier to understand code with the former approach even though you have to do things explicitly.
What do you guys think? @nvukobratTT @pglusacTT

@vkovinicTT vkovinicTT force-pushed the vkovinic/verification_outputs branch from 2045bb5 to f08ac0d Compare February 28, 2025 13:47
Copy link

TestsPassed ✅Skipped ⚠️Failed
TT-Forge-FE Tests675 ran543 passed132 skipped0 failed
TestResult
No test annotations available

1 similar comment
Copy link

TestsPassed ✅Skipped ⚠️Failed
TT-Forge-FE Tests675 ran543 passed132 skipped0 failed
TestResult
No test annotations available

Copy link

TestsPassed ✅Skipped ⚠️Failed
TT-Forge-FE Tests617 ran483 passed134 skipped0 failed
TestResult
No test annotations available

1 similar comment
Copy link

TestsPassed ✅Skipped ⚠️Failed
TT-Forge-FE Tests617 ran483 passed134 skipped0 failed
TestResult
No test annotations available

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Expand verify to return golden and device calculated values (optional)
1 participant