Proposal: Drop old rdkafka support and stop patching Rdkafka::Producer#close #629

Background

out_rdkafka and out_rdkafka2 patch Rdkafka::Producer#close to add a timeout. The original close waits until all queued messages are delivered or expire. When the broker is down, this takes message.timeout.ms (5 minutes by default), and Fluentd cannot stop in the meantime.

rdkafka changed its internals several times, so we now keep five versions of the patch in lib/fluent/plugin/rdkafka_patch/. Each one depends on the internals of rdkafka, and they have caused bugs:

• With rdkafka 0.16 or later, the worker can crash or hang on shutdown with queued messages (rdkafka2: Worker crashes with SIGSEGV or hangs on shutdown when messages are still queued #611, fixed by out_rdkafka, out_rdkafka2: Fix crash or hang on shutdown with queued messages #624).
• With rdkafka older than 0.12, the patch never destroys the librdkafka handle, so the process can crash with SIGSEGV on exit. This sometimes breaks the CI job for these versions (out_rdkafka, out_rdkafka2: Fix crash on exit with rdkafka older than 0.12 #626).

Proposal

Stop the producer with the public API of rdkafka instead of the patch:

1. Call Producer#flush(timeout_ms) (rdkafka 0.13 or later).
2. If it returns false, call Producer#purge (rdkafka 0.14 or later) to drop the remaining messages.
3. Call the original Producer#close. It returns soon because the queue is empty.

I tried this with rdkafka 0.31.0 and an unreachable broker. flush(2000) returned false after 2 seconds, and purge and close finished right after. With close alone, it did not return within 30 seconds. The original close also takes the lock of rdkafka, so it is safe against a concurrent produce.

If this does not work well, we can ask rdkafka to add a timeout to close. Note that a similar request was closed in karafka/rdkafka-ruby#268. The maintainer suggested tuning message.timeout.ms instead.

Supported versions

To remove all the patches, we need rdkafka 0.14 or later.

No fluent-package release ships rdkafka older than 0.12, and td-agent v4 reached its end of life in 2023. So we can drop these versions first.

fluent-package v5.0.x ships rdkafka 0.12.0. It has also reached its end of life, but many users may still use it. We can keep the patches for 0.12 and 0.13 for a while, or drop them at the same time.