You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello! In the function PoolingKernelImpl resulting value is calculated as follows:
result = CLAMP(result / (size_x * size_y), getMinValue(fmt), getMaxValue(fmt));
But the result variable has a type int32_t, the variables size_x and size_y have a type size_t and, therefore, a result of a dividing operation has a type size_t. This value is compared in the CLAMP macro with value INT16_MIN of type int_fast32_t which is converted to a big positive value of a type size_t. And so almost all positive resulting values will be clamped to INT16_MIN.
Hello! In the function
PoolingKernelImpl
resulting value is calculated as follows:But the
result
variable has a typeint32_t
, the variablessize_x
andsize_y
have a typesize_t
and, therefore, a result of a dividing operation has a typesize_t
. This value is compared in theCLAMP
macro with valueINT16_MIN
of typeint_fast32_t
which is converted to a big positive value of a typesize_t
. And so almost all positive resulting values will be clamped toINT16_MIN
.You can try it here.
I think that the correct way to calculate the resulting value is to convert it to the type
int32_t
before comparison:The text was updated successfully, but these errors were encountered: