jl_calloc is subject to unchecked unsigned integer wrapping

[mkitti]

There is the opportunity for an unchecked unsigned integer wrapping
in the exported jl_calloc:

julia/src/gc.c

Lines 3464 to 3466 in 0ea61fa

	JL_DLLEXPORT void *jl_calloc(size_t nm, size_t sz)
	{
	size_t nmsz = nm*sz;

julia> Libc.calloc(0xffffffffffffffff,  0xffffffffffffffff)
Ptr{Nothing} @0x0000000000000000

julia> ccall(:jl_calloc,Ptr{Nothing},(Csize_t, Csize_t), 0xffffffffffffffff, 0xffffffffffffffff)
Ptr{Nothing} @0x000000001ebd1180

My expectation is that jl_calloc should return C_NULL like Libc.calloc.

[mkitti]

My proposed fix is

//_unchecked_calloc does not check for potential overflow of nm*sz
static inline void *_unchecked_calloc(size_t nm, size_t sz) {
    size_t nmsz = nm*sz;
    int64_t *p = (int64_t *)jl_gc_counted_calloc(nmsz + JL_SMALL_BYTE_ALIGNMENT, 1);
    if (p == NULL)
        return NULL;
    p[0] = nmsz;
    return (void *)(p + 2); // assumes JL_SMALL_BYTE_ALIGNMENT == 16
}

JL_DLLEXPORT void *jl_calloc(size_t nm, size_t sz)
{
    if (nm > SIZE_MAX/sz)
        return NULL;
    return _unchecked_calloc(nm, sz);
}

This is implemented in #42704, but we may want to apply a smaller patch.