Split Triton library in two: libtriton.so and libtritoncore.so

[abrown]

Over time, Triton has accumulated dependencies from its core files
toward the plugins that implement vendor-specific functionality. This is
not ideal: (a) Triton can only be compiled with vendor-specific plugins
enabled and (b) this could encourage Triton extensions to depend on
vendor-specific symbols. This change proposes a split into two
libraries:

• libtritoncore.so which would (eventually) contain only Triton code,
  nothing vendor-specific; all of these symbols are exported
• libtriton.so which RPATH-depends on libtritoncore.so and
  includes all of vendor-specific code with the Python bindings; these
  symbols are all hidden (see #9922).

This split ensures Triton extensions have a clear, minimal target to
link to: the exported symbols in libtritoncore.so. It also avoids (a)
relying on hacks to avoid the missing pybind symbols for triton-opt
and other binaries used with extensions (e.g., #51) and (b) eliminates
the TRITON_EXT_ENABLED configuration entirely. This last result
normalizes how Triton is built: the built artifacts are now the same
whether Triton extensions are enabled or not.

What this PR does not do:

• it doesn't yet fix all of the vendor dependencies: this only sets
  things up to make these decisions. A next step for sorting out what
  libraries go where is to add a CORE argument to add_triton_library
  to split dependent libraries into TRITON_CORE_LIBRARIES or
  TRITON_PYTHON_LIBRARIES
• it does not fully resolve globally-scoped symbol conflicts: when
  Triton is used with extensions, the extensions need access to exported
  symbols. This change limits the number of exported symbols to those in
  libtritoncore.so (and the single symbol that #9922 exports from
  libtriton.so), but does not fully resolve the opportunity for
  conflicts. I had hoped that libtriton.so depending on
  libtritoncore.so would prevent the core symbols from polluting the
  global symbol scope but, IIUC, system loaders will just dump
  everything there. I would be happy to be corrected on this by someone
  more familiar; maybe there is something additional that can be done at
  the system loading level to do the equivalent of RTLD_LOCAL (?).

This change should not alter any Triton functionality, but the change in
symbol location requires careful review. I can provide dumps of symbol
lists if that helps.

[neildhar]

Splitting the libraries in this way has some significant downsides:

1. It forces us to export the symbols needed for extensions, which we had decided against doing by default because of the potential for conflicts.
2. It requires us to be careful to avoid ODR conflicts by including the same file or static library in both dynamic libraries.
3. It complicates the build and runtime loading.

(a) relying on hacks to avoid the missing pybind symbols for triton-opt and other binaries used with extensions

I recall the key issue here is that triton-opt has its own copy of LLVM. I have previously proposed using an empty libtriton.so to work around this problem (and the python issue).

[CRobeck]

I think @ThomasRaoux is also going to have issues with eliminating TRITON_EXT_ENABLED. One of agreements was to allow for enforcing layering between Triton core and the plugins mechanism. We agreed to have this enabled for the release by default but for upstream builds it had to explicitly be set by the user.

[abrown]

1. It forces us to export the symbols needed for extensions, ...

@neildhar, @CRobeck: removing TRITON_EXT_ENABLED and exporting all core symbols here is orthogonal to the main thrust of this PR (i.e., splitting out a core subset of symbols). I only removed TRITON_EXT_ENABLED because exporting all the symbols from a core shared library made it irrelevant; one could imagine removing that commit and retaining TRITON_EXT_ENABLED, which would either export all the core symbols in libtritoncore.so when TRITON_EXT_ENABLED=ON or statically provide core symbols in libtritoncore.a when TRITON_EXT_ENABLED=OFF. That would keep the symbol scope as it is today and still accomplish the main benefit of this PR, which is to (eventually separate "core" Triton from the plugin-related code in a clean way.

2. It requires us to be careful to avoid ODR conflicts...

I agree. This split could have a follow-on change to check that no symbol in libtritoncore.so is also defined in the libtriton.so. But the split also helps with other kinds of conflicts: as we decide what "core" means, we will start finding dependencies "pointing the wrong way" like those in 3685fb6. This split would give the mechanism and motivation to fix these dependency conflicts.

3. It complicates the build and runtime loading.

This change isn't a terribly huge one to the build configuration (?) and depending on a library via RPATH is pretty conventional (though, again, we could statically link a libtritoncore.a and get some of the benefit). But it is a change... I guess what I'm asking with this proposal is whether we want to carve out the "core" subset of Triton, which will invariably require some change to the codebase.

[sjw36]

This looks like a clean structural separation of Triton MLIR and the python front-end, harder to hide wrong-way dependencies. And simplifies lit testing in triton-ext.
Bonus: the bin/ tools stop each statically linking their own copy of LLVM+Triton, which should help link time and binary size.

+1 to keeping the TRITON_EXT_ENABLED build flag and default to static linkage as @abrown suggests.

[ThomasRaoux]

I don't understand what this is trying to solve.
I don't think we want two static library, do any other changes help with the problem you are trying to solve?