Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] Reciprocal unary op causes crash after all_reduce #16646

Open
rfurko-tt opened this issue Jan 10, 2025 · 1 comment
Open

[Bug Report] Reciprocal unary op causes crash after all_reduce #16646

rfurko-tt opened this issue Jan 10, 2025 · 1 comment
Assignees
Labels
bug Something isn't working P0 TT-train

Comments

@rfurko-tt
Copy link
Contributor

Describe the bug

Crash while running this test. Substituting ttnn::reciprocal with any other unary op doesn't result in crash.

TEST_F(N300UtilsTest, TestXTensorReplicateAllReduce_96_768) {
    auto* device = &ttml::autograd::ctx().get_device();
    auto mesh_shape = device->shape();
    std::cout << "I AM HERE IN TEST 0\n";
    xt::xarray<float> xtensor = xt::random::rand({96 * 768}, -0.05, 0.05).reshape({1, 1, 96, 768});

    ttml::core::XTensorToMeshVariant<float> replicate_composer = ttml::core::ReplicateXTensorToMesh<float>(mesh_shape);
    auto tensor = ttml::core::from_xtensor(xtensor, device, replicate_composer);
    std::cout << "I AM HERE IN TEST 1\n";
    tensor = ttnn::add(tensor, 100.0F);
    std::cout << "I AM HERE IN TEST 2\n";
    tensor = ttnn::reciprocal(tensor);
    std::cout << "I AM HERE IN TEST 3\n";
}

To Reproduce

  1. Run specified test above
  2. See crash and prints before last print

Expected behavior
Should work without crash.

Please complete the following environment information:

  • OS: Ubuntu 20.04
  • Version of software: ca2c867

Additional context
There might be a deeper problem, so we would like to understand what's going on with this random operation. Thanks in advance.

@rfurko-tt rfurko-tt added bug Something isn't working P0 TT-train labels Jan 10, 2025
@rfurko-tt
Copy link
Contributor Author

@dmakoviichuk-tt @davorchap FYI ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P0 TT-train
Projects
None yet
Development

No branches or pull requests

2 participants