custreamz oauth callback for kafka (librdkafka)

[jdye64]

Previously it was impossible to use custreamz with oauth enabled Kafka brokers. This PR adds a feature so that the user can supply a Python function which is invoked to get the oauth token, from a http endpoint for example, and then supply that token to librdkafka to be used in both the initial connection to kafka and also subsequently as the token becomes stale.

This closes #9410

[robertmaynard]

CMake changes 👍

[ajschmidt8]

We'll also need to update the librdkafka version in the integration repo below. @jdye64, can you open a PR for that?

• https://github.com/rapidsai/integration/blob/b71620b926b2f1a7024e67fafb071ad92a3dc825/conda/recipes/versions.yaml#L89-L90

[jdye64]

We'll also need to update the librdkafka version in the integration repo below. @jdye64, can you open a PR for that?

• https://github.com/rapidsai/integration/blob/b71620b926b2f1a7024e67fafb071ad92a3dc825/conda/recipes/versions.yaml#L89-L90

Done - rapidsai/integration#386

[jdye64]

rerun tests

[ajschmidt8]

rerun tests

[jdye64]

rerun tests

[jdye64]

rerun tests

[vyasr]

The new version looks so much better! I'm happy that you were able to avoid the Python C API to this extent. I have a few minor inline comments, but hopefully nothing too painful left to deal with. I also have a few questions that didn't fit inline anywhere:

• There are a lot of build script changes in this PR that seem unrelated to enabling the kafka callbacks, what's the reason for those?
• A .swo file got committed, can you remove it? I assume you have .swp in your .gitignore but had multiple copies opened at different times (or had a vim crash) so it created a .swo file.
• Can we add a Python test of the callback as well as the C++ test? The only Python test change that I see seems completely unrelated.

[ramakrishnap-nv]

Looks good, need to update year in files since its 2022. And if possible resolve the following issue https://github.com/rapidsai/cudf/pull/9486/files#r751707416.

Rest looks good

Seems that all requested changes have been addressed.

[ajschmidt8]

@gpucibot merge

[bdice]

@jdye64 It looks like this PR accidentally added a 12KB binary temp file from vim, in commit jdye64@7932468. This was missed during review and is now in the main repo: https://github.com/rapidsai/cudf/blob/76f89db80a64a2aa49b618aad80fe80e34e0332f/python/cudf_kafka/cudf_kafka/_lib/.kafka.pxd.swo

Can you please open a PR to remove this file? Thank you!

This gitignore may also be helpful: https://github.com/github/gitignore/blob/main/Global/Vim.gitignore

[vyasr]

Hmm I specifically requested to get rid of that in some round of review and I thought I saw it deleted in one of the subsequent PRs. Unfortunate that it crept back in. Replacing *.swp with a broader gitignore rule like *.sw[a-p] would definitely be helpful to avoid this kind of issue.