Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Aug 31, 2025

Fixes #116133

Core Fix

  • Introduced SumOfSquaredAbsoluteDifferences<T> operation that computes T.Abs(x - mean) before squaring
  • Updated Tensor.StdDev to use this new operation, ensuring mathematically correct results for complex numbers

Test Improvements

  • Converted TensorStdDevTests to TensorStdDevFloatTests Theory test with multiple test cases including sequential numbers, same values, and mixed positive/negative values
  • Converted TensorStdDevComplexTests to Theory test with test cases for complex numbers from the original issue, purely imaginary numbers, and real complex numbers
  • Added separate TensorStdDevNonContiguousTests to ensure non-contiguous tensor calculations work correctly
  • Added a TestComplex struct that throws NotImplementedException for interface methods not essential for StdDev calculations

Test Coverage

The new Theory tests cover:

  • Cases where Abs doesn't matter: Real numbers (float values) where |x|² = x² for the standard deviation calculation
  • Cases where Abs is critical: Complex numbers where |z|² ≠ z², ensuring the fix works correctly
  • Edge cases: All same values (should return 0), mixed positive/negative values, purely imaginary numbers, and real complex numbers

Both TensorPrimitives.StdDev and Tensor.StdDev now return identical mathematically correct results for all test cases.

Copilot AI changed the title [WIP] Tensor.StdDev vs TensorPrimitivers.StdDev differences for Complex input Fix Tensor.StdDev vs TensorPrimitives.StdDev differences for Complex input Aug 31, 2025
Copilot AI requested a review from jeffhandley August 31, 2025 07:16
Copilot finished work on behalf of jeffhandley August 31, 2025 07:16
…tion

- Fix typo: "TensorPrimitivers" -> "TensorPrimitives" in test comment
- Replace custom SumOfSquaredMagnitudeDifferences operation with direct use of TensorPrimitives.StdDev
- Leverage TensorPrimitives vectorization optimizations where T.Abs is only called when necessary
- Remove unnecessary custom operation that always called T.Abs for all types
- Use TryGetSpan for contiguous tensors and fallback to FlattenTo for non-contiguous tensors

Co-authored-by: jeffhandley <[email protected]>
[Fact]
public static void TensorStdDevComplexTests()
{
// Test case from issue: Tensor.StdDev vs TensorPrimitivers.StdDev differences for Complex input
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the typo in commit 4585313.

Copilot AI requested a review from jeffhandley August 31, 2025 10:22
Copilot finished work on behalf of jeffhandley August 31, 2025 10:22
Copilot AI requested a review from jeffhandley August 31, 2025 11:18
Copilot finished work on behalf of jeffhandley August 31, 2025 11:18
…or non-essential methods and convert StdDev tests to Theory tests

Co-authored-by: jeffhandley <[email protected]>
Copilot AI requested a review from jeffhandley August 31, 2025 13:02
Copilot finished work on behalf of jeffhandley August 31, 2025 13:02
@jeffhandley
Copy link
Member

@tannergooding I explored a couple options on this and this seems best to me. For my own reference/learning, I looked into how NumPy handles complex numbers for Euclidean Distance and Standard Deviation of complex numbers and found it also resorts to an absolute value of the diff for StdDev but not Distance, increasing my confidence this was the correct approach.

I do worry slightly about introducing the Abs call purely to support complex numbers, but I couldn't find a way to distinguish between real and (custom) complex numbers reliably from the interfaces. If you have a trick up your sleeve here, that could change the implementation.

@jeffhandley jeffhandley marked this pull request as ready for review August 31, 2025 14:01
Copilot AI review requested due to automatic review settings August 31, 2025 14:01
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes mathematical correctness issues between Tensor.StdDev and TensorPrimitives.StdDev when calculating standard deviation for complex numbers. The core problem was that the tensor implementation was squaring complex differences directly instead of taking the absolute value before squaring, which is required for mathematically correct standard deviation calculation.

Key changes:

  • Introduced SumOfSquaredMagnitudeDifferences<T> operation that properly computes T.Abs(x - mean) before squaring
  • Updated Tensor.StdDev to use the new operation for consistent results with TensorPrimitives.StdDev
  • Enhanced test coverage with Theory tests covering both real and complex number scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/libraries/System.Numerics.Tensors/tests/TensorTests.cs Converts existing tests to Theory tests with comprehensive test cases for both float and complex numbers, adds TestComplex struct for testing
src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/netcore/TensorOperation.cs Adds new SumOfSquaredMagnitudeDifferences operation that correctly applies absolute value before squaring
src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/netcore/Tensor.cs Updates StdDev implementation to use the new mathematically correct operation

@jeffhandley jeffhandley enabled auto-merge (squash) September 16, 2025 19:20
@jeffhandley jeffhandley merged commit 0c5f13b into main Sep 17, 2025
88 checks passed
@jeffhandley
Copy link
Member

/backport to release/10.0

@github-actions
Copy link
Contributor

@jkotas jkotas deleted the copilot/fix-116133 branch September 22, 2025 23:28
@github-actions github-actions bot locked and limited conversation to collaborators Oct 23, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tensor.StdDev vs TensorPrimitives.StdDev differences for Complex input

4 participants