Skip to content

fix: bind durable DLQ consumer so retry drains on workqueue stream - #220

Merged
ecv merged 3 commits into
mainfrom
fix/dlq-retry-durable-consumer
Sep 15, 2026
Merged

ecv merged 3 commits into
mainfrom
fix/dlq-retry-durable-consumer

Conversation

@ecv

@ecv ecv commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

The dead-letter stream allows only one consumer per subject filter, and the retry loop created a fresh overlapping one on every batch, so it usually could not attach and no failed activity event was ever retried.

The backlog has grown from 177 events to 420 with no consumer attached at all, and each one is a user-visible activity that will never render.

One durable consumer is now provisioned for that stream, both processor replicas bind it and compete for messages, and events waiting on backoff are deferred rather than redelivered immediately.

Policy-triggered retry can no longer narrow its fetch server-side, so it scans forward under a time cap and defers the events it skips.

Test plan

  • Five JetStream tests cover each failure mode, and each one fails when its fix is reverted
  • Two replicas draining 250 queued events concurrently reach zero, with every event republished exactly once
  • Build, vet, and the component render all pass on the rebased branch
  • After release, the production backlog drains from 420 and the stream reports a live consumer

Fixes #216
Related to #241

@ecv
ecv requested review from JoseSzycho and scotwells July 9, 2026 13:48
@ecv ecv self-assigned this Aug 6, 2026
ecv added 3 commits September 14, 2026 14:35
The DLQ retry controller created an ephemeral pull consumer per batch on
the ACTIVITY_DEAD_LETTER stream, which uses workqueue retention. Workqueue
streams permit only one consumer per non-overlapping subject filter, so
concurrent replicas and the per-policy narrow filter collided with
"filtered consumer not unique on workqueue stream" and the DLQ never
drained (totalSucceeded: 0 every run; prod backlog flat for 42h).

Bind a single shared durable consumer instead, provisioned declaratively
like every other stream's consumer. Per-policy filtering stays client-side,
so no second overlapping consumer is created. Defer redelivery of
backed-off events with NakWithDelay so a periodic run reaches processed==0
and stops instead of re-fetching the same events until the run deadline.

Key changes:
- Add dlq-retry-consumer.yaml durable pull consumer (filter activity.dlq.>)
  and wire it into the nats-streams kustomization
- Bind the durable in processRetryBatch; drop the ephemeral consumer and
  the overlapping per-policy server-side filter
- NakWithDelay backed-off events instead of immediate Nak

Fixes #216
Add in-process JetStream tests covering the #216 failure modes: draining a
workqueue-retention DLQ stream, two replicas binding the shared durable
consumer without a "filtered consumer not unique" collision, and a
backed-off event not being redelivered within the same run.

Adds nats-server/v2 as a test dependency for the embedded server.
Binding one shared durable consumer means the policy-triggered retry path
can no longer ask the server for its own subject, so it has to scan past
the events of every other policy queued ahead of it. It ran a single
batch and NAKed non-matching events outright, so a backlog larger than
one batch consumed the whole fetch and the policy reached none of its own
events. Production currently holds 420 dead-lettered events against a
batch size of 100, so an ActivityPolicy update would have retried nothing.

Scan forward until the policy's events are exhausted, the context is
cancelled, or a 30s cap is reached, and defer a skipped event past the end
of the scan so the loop never re-fetches what it just passed over.

Key changes:
- Replace the single policy batch with a bounded scan loop, capped by
  maxPolicyRunDuration and reporting per-batch durations like the
  periodic run
- Defer skipped events with NakWithDelay(policySkipRedeliveryDelay)
  instead of an immediate Nak, which would redeliver them into the same
  scan
- Raise the durable's maxAckPending to twice the batch size: two replicas
  share the consumer, and at one batch's worth a single replica can hold
  every pending slot while the other reads its empty fetch as a drained
  DLQ and ends its run early
- Cover both behaviours with JetStream tests: a policy event behind a
  full batch of another policy's events is still retried, and a skipped
  event is not redelivered inside the same scan
@ecv
ecv force-pushed the fix/dlq-retry-durable-consumer branch from 87dfb53 to 5456c2d Compare September 14, 2026 18:40
@ecv

ecv commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Production reports 420 dead-lettered events and zero consumers on that stream, so nothing has attached since this was filed and nobody has drained it by hand.

Two choices worth a second look.

The policy-triggered path scans rather than filters. One shared consumer carries one fixed subject filter, so that path cannot ask the server for its own subject any more and has to match client-side against whatever sits at the head of the queue. A single batch is useless there: with 120 other-policy events queued ahead and a batch size of 100, it processes 100 and republishes none of its own. Production is at 420 against that same batch size. It now scans until its own events run out, the context is cancelled, or 30 seconds pass, and defers a skipped event past the end of the scan so the loop cannot re-fetch what it just passed over.

The durable's pending-ack ceiling is twice the batch size because two replicas share the consumer. At one batch's worth, one replica can hold every slot while the other reads its empty fetch as a drained queue and ends its run early.

@ecv
ecv merged commit dedcbec into main Sep 15, 2026
6 checks passed
@ecv
ecv deleted the fix/dlq-retry-durable-consumer branch September 15, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The DLQ retry loop cannot attach its consumer, so the backlog stays

2 participants