Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

normalize implementation is not numerically stable #1343

Open
jordanwalker98 opened this issue Jan 26, 2025 · 0 comments
Open

normalize implementation is not numerically stable #1343

jordanwalker98 opened this issue Jan 26, 2025 · 0 comments

Comments

@jordanwalker98
Copy link

jordanwalker98 commented Jan 26, 2025

I found that the current implementation of normalize:
return v * inversesqrt(dot(v, v));
is less numerically stable than:
return v / sqrt(dot(v, v));
even for double precision vectors.

By this I mean that if you repeatedly call the original implementation, the result will perpetually oscillate between two values. The division version will converge and is very stable with double precision. A simple test is to generate a series of random vectors, normalize, then normalize again and compare the first and second results. This can cause (in my case) hard-to-diagnose issues when using glm to solve computational geometry problems that require high precision.

EDIT: there are still rare cases where normalizing double precision vectors doesn't converge but division does appear to be more stable

I also found this link that points out the same issue:
https://stackoverflow.com/questions/23303598/3d-vector-normalization-issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant