fix: bind durable DLQ consumer so retry drains on workqueue stream - #220
Conversation
The DLQ retry controller created an ephemeral pull consumer per batch on the ACTIVITY_DEAD_LETTER stream, which uses workqueue retention. Workqueue streams permit only one consumer per non-overlapping subject filter, so concurrent replicas and the per-policy narrow filter collided with "filtered consumer not unique on workqueue stream" and the DLQ never drained (totalSucceeded: 0 every run; prod backlog flat for 42h). Bind a single shared durable consumer instead, provisioned declaratively like every other stream's consumer. Per-policy filtering stays client-side, so no second overlapping consumer is created. Defer redelivery of backed-off events with NakWithDelay so a periodic run reaches processed==0 and stops instead of re-fetching the same events until the run deadline. Key changes: - Add dlq-retry-consumer.yaml durable pull consumer (filter activity.dlq.>) and wire it into the nats-streams kustomization - Bind the durable in processRetryBatch; drop the ephemeral consumer and the overlapping per-policy server-side filter - NakWithDelay backed-off events instead of immediate Nak Fixes #216
Add in-process JetStream tests covering the #216 failure modes: draining a workqueue-retention DLQ stream, two replicas binding the shared durable consumer without a "filtered consumer not unique" collision, and a backed-off event not being redelivered within the same run. Adds nats-server/v2 as a test dependency for the embedded server.
Binding one shared durable consumer means the policy-triggered retry path can no longer ask the server for its own subject, so it has to scan past the events of every other policy queued ahead of it. It ran a single batch and NAKed non-matching events outright, so a backlog larger than one batch consumed the whole fetch and the policy reached none of its own events. Production currently holds 420 dead-lettered events against a batch size of 100, so an ActivityPolicy update would have retried nothing. Scan forward until the policy's events are exhausted, the context is cancelled, or a 30s cap is reached, and defer a skipped event past the end of the scan so the loop never re-fetches what it just passed over. Key changes: - Replace the single policy batch with a bounded scan loop, capped by maxPolicyRunDuration and reporting per-batch durations like the periodic run - Defer skipped events with NakWithDelay(policySkipRedeliveryDelay) instead of an immediate Nak, which would redeliver them into the same scan - Raise the durable's maxAckPending to twice the batch size: two replicas share the consumer, and at one batch's worth a single replica can hold every pending slot while the other reads its empty fetch as a drained DLQ and ends its run early - Cover both behaviours with JetStream tests: a policy event behind a full batch of another policy's events is still retried, and a skipped event is not redelivered inside the same scan
87dfb53 to
5456c2d
Compare
|
Production reports 420 dead-lettered events and zero consumers on that stream, so nothing has attached since this was filed and nobody has drained it by hand. Two choices worth a second look. The policy-triggered path scans rather than filters. One shared consumer carries one fixed subject filter, so that path cannot ask the server for its own subject any more and has to match client-side against whatever sits at the head of the queue. A single batch is useless there: with 120 other-policy events queued ahead and a batch size of 100, it processes 100 and republishes none of its own. Production is at 420 against that same batch size. It now scans until its own events run out, the context is cancelled, or 30 seconds pass, and defers a skipped event past the end of the scan so the loop cannot re-fetch what it just passed over. The durable's pending-ack ceiling is twice the batch size because two replicas share the consumer. At one batch's worth, one replica can hold every slot while the other reads its empty fetch as a drained queue and ends its run early. |
Summary
The dead-letter stream allows only one consumer per subject filter, and the retry loop created a fresh overlapping one on every batch, so it usually could not attach and no failed activity event was ever retried.
The backlog has grown from 177 events to 420 with no consumer attached at all, and each one is a user-visible activity that will never render.
One durable consumer is now provisioned for that stream, both processor replicas bind it and compete for messages, and events waiting on backoff are deferred rather than redelivered immediately.
Policy-triggered retry can no longer narrow its fetch server-side, so it scans forward under a time cap and defers the events it skips.
Test plan
Fixes #216
Related to #241