Vendor in types and datamodel for CUDA-specific changes

[VijayKandiah]

This is a large PR that vendors in types and datamodel from Numba for future CUDA-specific customizations. This PR includes a convert_to_cuda_type API in numba.cuda.core.sigutils which is called by default inside normalize_signature to map numba.core.types (if any) to equivalent numba.cuda.types if possible. This mapping enables the typing system in numba-cuda to work on numba.core.types.

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[VijayKandiah]

/ok to test

[VijayKandiah]

/ok to test

[VijayKandiah]

Working on the cuDF failures.

[VijayKandiah]

/ok to test

[VijayKandiah]

/ok to test

[VijayKandiah]

/ok to test

[VijayKandiah]

/ok to test

[VijayKandiah]

/ok to test

[brandon-b-miller]

Not blocking for this PR but have we considered determining the presence of numba in a central way up front that we can just reference instead of proliferating try/catches? if HAVE_NUMBA: etc

[VijayKandiah]

I like that idea. Yes, we should be doing that. A centralized HAVE_NUMBA is much cleaner than all the try imports we have in typing and other modules. This can be in a future PR.

[VijayKandiah]

Also, since we have the redirector in place, this type conversion/mapping in sigutils is not really needed. If we have numba in the env, the redirector will ensure that numba.core.types are used everywhere. So we would not encounter a case where there'd be a mix of numba.cuda.types and numba.core.types needing this conversion. Should we just remove this?

[VijayKandiah]

I removed the type mapping.

[VijayKandiah]

/ok to test

[gmarkall]

This redirection method does not work - if I change this to simulate what would happen if Numba were not found, like:

diff --git a/numba_cuda/numba/cuda/types.py b/numba_cuda/numba/cuda/types.py
index 94e30c17..ba81757e 100644
--- a/numba_cuda/numba/cuda/types.py
+++ b/numba_cuda/numba/cuda/types.py
@@ -5,7 +5,7 @@ import importlib
 import sys
 from numba.cuda.utils import _RedirectSubpackage
 
-if importlib.util.find_spec("numba.core.types"):
+if importlib.util.find_spec("numba.core.types") and False:
     sys.modules[__name__] = _RedirectSubpackage(locals(), "numba.core.types")
 else:
     sys.modules[__name__] = _RedirectSubpackage(

then importing cuda from numba no longer works:

$ python -c "from numba import cuda"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/gmarkall/numbadev/numba-cuda/numba_cuda/numba/cuda/__init__.py", line 11, in <module>
    import numba.cuda.types as types
  File "/home/gmarkall/numbadev/numba-cuda/numba_cuda/numba/cuda/types.py", line 11, in <module>
    sys.modules[__name__] = _RedirectSubpackage(
                            ^^^^^^^^^^^^^^^^^^^^
  File "/home/gmarkall/numbadev/numba-cuda/numba_cuda/numba/cuda/utils.py", line 592, in __init__
    new_mod_obj = import_module(new_module)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gmarkall/miniforge3/envs/numbadev/lib/python3.12/importlib/__init__.py", line 98, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gmarkall/numbadev/numba-cuda/numba_cuda/numba/cuda/cuda_types/__init__.py", line 10, in <module>
    from .containers import *
  File "/home/gmarkall/numbadev/numba-cuda/numba_cuda/numba/cuda/cuda_types/containers.py", line 26, in <module>
    from .misc import Undefined, unliteral, Optional, NoneType
  File "/home/gmarkall/numbadev/numba-cuda/numba_cuda/numba/cuda/cuda_types/misc.py", line 4, in <module>
    from numba.cuda.types.abstract import Callable, Literal, Type, Hashable
ModuleNotFoundError: No module named 'numba.cuda.types.abstract'; 'numba.cuda.types' is not a package

[gmarkall]

Just realised I forgot to clarify: the problem is you can't use a module to redirect a package, only another module - this is why when I prototyped this I had individual redirectors like cuda_abstract.py -> abstract.py, etc.

[gmarkall]

I don't think any of the globals in here (e.g. Dim3, GridGroup), etc., have been used like public API under numba.cuda.types before, so this should be a safe move. If this turns out to be incorrect we could always move the ones that have been used publicly into types/__init__.py.

[gmarkall]

I think in general the changes here are good, apart from the specific issue with the redirector not being able to support using a module to redirect to a package. I believe I have a fix for this in #545; with that change, the CI seems to pass, apart from a couple of runners getting stuck with network connection issues. When I locally apply:

diff --git a/numba_cuda/numba/cuda/utils.py b/numba_cuda/numba/cuda/utils.py
index 13203103..b5bf83ca 100644
--- a/numba_cuda/numba/cuda/utils.py
+++ b/numba_cuda/numba/cuda/utils.py
@@ -615,7 +615,7 @@ class _RedirectSubpackage(ModuleType):
 
 
 def redirect_numba_module(old_module_locals, numba_module, numba_cuda_module):
-    if find_spec("numba"):
+    if find_spec("numba") and False:
         return _RedirectSubpackage(old_module_locals, numba_module)
     else:
         return _RedirectSubpackage(old_module_locals, numba_cuda_module)

to that, to simulate if Numba were not found, then I see that the redirection works in the sense that the tests run, but they don't all pass because we still aren't fully isolated between Numba types and Numba-CUDA types (the errors we've seen before like TypeError: cannot augment Function(<built-in function truth>) with Function(<built-in function truth>) appear).

For the purposes of this PR I think that is OK - if the code were exclusively using Numba-CUDA code / types and not also including Numba types, there would be no duplication, but we're not at that point until subsequent PRs have gone in.

[VijayKandiah]

/ok to test