Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Signed and unsigned values comparison in PoolingKernelImpl #23

Open
dvorotnev opened this issue May 20, 2020 · 0 comments
Open

Signed and unsigned values comparison in PoolingKernelImpl #23

dvorotnev opened this issue May 20, 2020 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@dvorotnev
Copy link

dvorotnev commented May 20, 2020

Hello! In the function PoolingKernelImpl resulting value is calculated as follows:

result = CLAMP(result / (size_x * size_y), getMinValue(fmt), getMaxValue(fmt));

But the result variable has a type int32_t, the variables size_x and size_y have a type size_t and, therefore, a result of a dividing operation has a type size_t. This value is compared in the CLAMP macro with value INT16_MIN of type int_fast32_t which is converted to a big positive value of a type size_t. And so almost all positive resulting values will be clamped to INT16_MIN.

You can try it here.

I think that the correct way to calculate the resulting value is to convert it to the type int32_t before comparison:

result = result / (size_x * size_y);
result = CLAMP(result, getMinValue(fmt), getMaxValue(fmt));
dvorotnev added a commit to dvorotnev/OpenVX-sample-impl that referenced this issue May 20, 2020
@kiritigowda kiritigowda added the bug Something isn't working label Jul 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants