chore(recordings): use node-librdkafka for ingester production

[hazzadous]

Previously we've been using the KafkaJS Producer with a wrapper around
it to handle batching. There are a number of issues with the batching
implementation e.g. not having a way to provide guarantees on delivery
and rather than fix that, we can simply use the librdkafka Producer
which is a lot more mature and battle-tested.

Ultimately we should probably move over to librdkafka for all the plugin server
operations, but I'm starting with recordings first and just the producer which is a
little less critical than the other workloads.

Problem

Changes

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

[mariusandra]

We switched away from librdkafka because it was flakey and caused the app to crash in unpredicatble ways :). @Twixes even had to submit a C++ patch to them.

Might be better now, and the early days of the plugin server were different (more async stuff flying around), but don't assume it's a silver bullet.

[hazzadous]

@mariusandra do you happen to have a link to examples of issues you'd seen? I've only migrated the functional tests to use the librdkafka producer so far, the seems ok so far. But would love to fail fast here rather than move everything over.

Is it more than just the patch that @Twixes submitted?

[mariusandra]

I tried finding something the old archive, but couldn't. I also don't remember the specifics, but more of a general feeling of things being odd/sluggish/flakey/crashy whenever we used it.

Sorry, I know I'm not very helpful :(

[hazzadous]

I tried finding something the old archive, but couldn't. I also don't remember the specifics, but more of a general feeling of things being odd/sluggish/flakey/crashy whenever we used it.

Sorry, I know I'm not very helpful :(

No it's all useful data. I'll have a look through that repo to see how it was being used.

[posthog-bot]

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

[posthog-bot]

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

[posthog-bot]

This PR was closed due to lack of activity. Feel free to reopen if it's still relevant.

[hazzadous]

@xvello running the tests with master merged in 🤞