Propagate LLM Observability context across service boundaries

[ncybul]

What Does This Do

Carries LLM Observability context (ML app, session ID, agent attribution, and the parent span id) across process boundaries, so an LLM trace stays connected when work crosses a queue or a service call.

Approach

Register an LLMObsContextPropagator as a propagation concern (AgentPropagation.LLMOBS_CONCERN). This propagator stages _dd.p.llmobs_* onto the span context and the tracing propagator serializes them like any other propagation tag. Every boundary auto-instrumentation already covers (SQS, HTTP, gRPC, Kafka) is handled at once.

Tag	Source
_dd.p.llmobs_ml_app	innermost active LLMObs span's ml_app
_dd.p.llmobs_sid	effective session_id
_dd.p.llmobs_pagent_span_id	nearest agent-kind ancestor's span id
_dd.p.llmobs_pagent_name	that ancestor's name
_dd.p.llmobs_parent_id	innermost active LLMObs span's span id

Java doesn't need dd-trace-py's _dd.p.llmobs_trace_id — it reuses the APM trace id.

Values are resolved from the ambient LLMObsContext at injection time, not written when the span starts, and injection is gated on trace-id consistency (the same gate DDLLMObsSpan already applies in-process).

Receive side

There are two possible parents: the ambient in-process LLMObsContext, and the _dd.p.llmobs_* values on the extracted span context. DDLLMObsSpan prefers the in-process one, but only when it belongs to the same trace. Otherwise (no ambient context, or one from a different trace) it falls back to the extracted values. An explicit value passed by the caller still wins over both.

Falling back on the trace mismatch, and not only when the ambient context is absent, is what makes a queue consumer correct: a scope leaked from a previously handled message would otherwise shadow the attribution that arrived with the current one.

"The extracted values" means specifically what arrived on the inbound headers, never what a local injection staged for an outbound call. PTags keeps both: a mutable bundle the propagator rewrites on every injection, and a final baseline of what was decoded. They have to be separate because CoreTracer hands the extracted context's PropagationTags to the local root, so the two live in one object. Two behaviours fall out of that split:

• A service that forwards a request without opening an LLMObs span of its own keeps passing its caller's context along, rather than clearing it. What arrived is the floor; an active LLMObs span on the trace overrides it.
• A peer LLMObs span opened after a sibling has finished doesn't read the sibling's leftover staged tags as though they were upstream. Nothing clears staged tags when a span ends — only the next injection does — so the receive-side getters read the baseline instead.

Other Changes

ml_app precedence

The service default now applies last, in DDLLMObsSpan, instead of eagerly in the span factory. Substituting it at the factory left no way to tell "caller named no application" from an explicit value, so every ml_app inheritance step — in-process and propagated alike — would have been unreachable.

Order is explicit > in-process LLMObs parent > propagated > DD_LLMOBS_ML_APP > DD_SERVICE, matching dd-trace-py.

The resolved value is unchanged for a span that names its own ml_app, and for one that names none with no LLMObs parent — it still falls back to DD_LLMOBS_ML_APP/DD_SERVICE. What does change is that a nested span naming no ml_app now inherits its enclosing span's value rather than the default. That subtree inheritance is required by propagation rather than incidental to it: injection reads the innermost active span's ml_app, and on the receiving side only the first LLMObs span reads the propagated value — without it, the wire value would be wrong at the producer and lost one level down at the consumer.

Validating application-supplied values

Unlike every other _dd.p.* tag, these five come from the application rather than the tracer, so they are checked before they reach the wire. A value carrying a character the receiving codec mis-parses doesn't just lose itself — it fails the whole tagset with decoding_error and takes _dd.p.tid with it, leaving the two services disagreeing about the upper 64 bits of the trace id. Dropping the one tag is the cheaper loss.

Rejected: anything outside printable ASCII, , (the x-datadog-tags separator), " and \ (AWS messaging concatenates these headers into a _datadog JSON attribute that is written and parsed without escaping), and anything the tracestate conversion rewrites irreversibly — ; and ~, which both become _. The last of those asks TagValue's own conversion table rather than restating it, so it stays correct if the table changes:

static boolean survivesW3CRoundTrip(char c) {
  return convertW3CtoDD(convertDDtoW3C(c)) == c;
}

= is deliberately kept — it round-trips as ~, and base64-ish session ids carry it routinely.

Rejecting rather than sanitizing, matching dd-trace-py: a _-mangled ml_app can silently collide two distinct applications into one bucket, which is worse than an absent tag falling back to the receiver's own DD_LLMOBS_ML_APP/DD_SERVICE.

Note the unescaped JSON concatenation in MessageAttributeInjector / TextMapInjectAdapter is a property of those carriers, not of LLMObs — any product's _dd.p.* value containing a quote hits it. Rejecting here closes the LLMObs route to it; escaping at the injectors is the broader fix and is left to agent-bootstrap.

Also fixes: propagation tags dropped at every AWS messaging boundary

DatadogAttributeParser — the shared _datadog message attribute parser used by SQS, SNS, EventBridge and Step Functions — never read x-datadog-tags, so every _dd.p.* tag was silently dropped crossing those boundaries. Inject was always correct; the raw attribute carries the full set.

Pre-existing bug, not introduced here, but needed for LLMObs to cross a queue. It means _dd.p.* now survives AWS messaging for all products — intended, but a change in what goes over the wire downstream of a queue.

Follow-ups (not in this PR)

• Restoring extracted context into the ambient LLMObsContext. Covered here: tags onto outbound carriers, and parsed back into the extracted span context, which DDLLMObsSpan reads. Not covered: the fully-automatic path where an inbound trace is followed by an auto-instrumented call with no manual LLMObs span — OpenAiDecorator reads LLMObsContext.current(), still null there. Separate because someone has to own attaching and closing a scope around the consumer's work, and that owner differs per integration.
• A length cap on application-supplied values. ml_app and session_id are charset-checked before the wire but not length-checked; an oversized value drops the whole header via inject_max_size. Left as-is because dd-trace-py behaves identically — no per-value guard there either. Worth revisiting cross-language.
• Step Functions. The parser fix above covers SQS, SNS and EventBridge, whose extract adapters all route through DatadogAttributeParser. Step Functions doesn't: it writes trace context into the request input body rather than into message attributes, so it needs its own extract path before LLMObs context can cross it. A gap rather than a regression — nothing about it changes in this PR.
• Per-injection tags on shared trace state — a cross-language design question. These five tags are staged on the root span context's propagation tags, which the whole local trace shares. Two threads injecting from different LLMObs scopes on one trace therefore race: each write is a whole immutable bundle behind a volatile, so a reader always sees a coherent set, but it may be the other call's. dd-trace-py has the same property — LLMObs._inject_llmobs_context writes into span_context._meta, and Span.context shares trace-level _meta across the trace — so this isn't a Java-specific defect and shouldn't be fixed Java-only. Worth noting the exposure differs: a JVM service fanning out parallel calls from sibling agents hits it more readily than Python's usual shapes. The real question is broader than a lock — everything else staged there (sampling priority, org marker, _dd.p.tid) is a trace-level fact identical for every destination, whereas these are per-injection and vary by destination. So: how should a propagation concern contribute per-injection tags? Same question in every language.

Testing

• LLMObsContextPropagatorTest — 10 tests, none calling any LLMObs propagation API: injection carries all five tags; nothing added with no active span; tags stop once the scope closes; no leak into a later injection on the same trace; producer → worker round trip with zero application plumbing; a worker naming no ml_app inherits the producer's; a worker with no upstream inherits nothing; a pass-through service forwards its caller's context; an injection before the local LLMObs span opens doesn't destroy the extracted context; a peer span doesn't inherit a finished sibling's staged tags.
• DDLLMObsSpanMlAppTest — 5 tests pinning the precedence chain (explicit, inherited, override, empty-as-absent, service default). Verified load-bearing by neutralising both fallbacks (4 of 5 fail). Only the propagated case overlaps with the propagator test.
• DatadogPropagationTagsTest / W3CPropagationTagsTest — the five tags through both codecs: round trip, per-value rejection of the characters listed above, and decoding on read (a tracestate value written as app~v1 reads back as app=v1).
• DatadogAttributeParserTest — 1 test covering only what this branch changes: x-datadog-tags reaches the extractor. Fails without the one-line parser fix.

Each fix landed in this PR was checked to be load-bearing by reverting it in isolation and confirming the intended test — and only that test — fails.

End to End Testing

Before

HTTP calls are split across more than one LLMObs trace (agent trace and tool trace) whereas the APM trace is connected.

SQS messages are split across more than one LLMObs trace (agent trace and tool trace). The APM trace is connected.

After

LLMObs trace with HTTP call and associated APM trace.

LLMObs trace with SQS message queueing and associated APM trace.

Claude session: 15543c2c-2abe-408e-b16e-05ddbe972287
Resume: claude --resume 15543c2c-2abe-408e-b16e-05ddbe972287

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.73 s	14.73 s	[-0.7%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent	13.62 s	13.66 s	[-1.0%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	17.63 s	17.33 s	[+0.7%; +2.9%] (maybe worse)
startup:petclinic:iast:Agent	17.50 s	17.66 s	[-1.7%; -0.2%] (maybe better)
startup:petclinic:profiling:Agent	17.54 s	17.43 s	[-0.5%; +1.7%] (no difference)
startup:petclinic:sca:Agent	17.42 s	17.45 s	[-1.2%; +0.8%] (no difference)
startup:petclinic:tracing:Agent	16.46 s	16.62 s	[-1.9%; -0.0%] (maybe better)

Commit: 8a4a3793 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[ncybul]

@codex

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77546ab8f7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bfa7670f8a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[datadog-prod-us1-6]

Parallel calls on one trace use the same LLMObs tag state. One call can send the session and parent data from another call.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit bfa7670 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[dougqh]

Design concern beyond the specific bugs above: putting these 5 tags on x-datadog-tags/tracestate shares fate with every other propagation tag through the size cap in PTagsCodec.fillTagMap — once the combined size exceeds getxDatadogTagsLimit(), the entire outbound header is dropped, not just the LLMObs fields, taking _dd.p.tid (upper 64 bits of the trace id) and sampling priority down with it. The PR description already flags this failure mode for per-value corruption (the whole rationale for rejecting unrepresentable characters), but the same shared budget is exposed to plain size growth too: ml_app + session_id + agent name are application-supplied and can be long, and now compete with every other product's propagation tags for the same limited space.

Given that, would baggage (already a first-class, separately-propagated mechanism here — its own W3C header, own context key, independent of PropagationTags) be a better carrier for these five? It would decouple LLMObs's fate from trace-id/sampling propagation entirely, and avoid adding cost to the ptags hot path (PropagationTags is parsed/serialized on every request) for traffic that never touches LLMObs. The counterargument is cross-language consistency — dd-trace-py already uses _dd.p.llmobs_* propagation tags for this, so matching it gets automatic joining with Python/Go/JS services without needing agreement on a baggage key scheme — but that tradeoff seems worth calling out explicitly rather than assumed.

[dougqh]

Following up on the size-cap comment above with a narrower point: the wire header names themselves are long — _dd.p.llmobs_ml_app, _dd.p.llmobs_sid, _dd.p.llmobs_pagent_span_id, _dd.p.llmobs_pagent_name, _dd.p.llmobs_parent_id — and every byte of key name eats into the same x-datadog-tags budget as the value itself. Given we have a hard cap on that header, I'd have preferred short wire keys mapped to longer, readable internal tag names — a translation layer at the codec boundary, rather than requiring the header key and the tag name to be the same string. That also would have sidestepped the character-escaping problem this PR spends real effort on: a mapping only needs to be reversible, not identical to something a human reads.

In retrospect, though — dd-trace-py/js/go have already shipped these exact key names on the wire, so changing them now isn't a Java-only decision, it's a cross-language migration. Given that commitment is already made, I don't think it's fair to hold this PR to fixing it. Noting it here mainly so it's on record for whenever the format does get revisited (e.g. if a versioned propagation-tag scheme ever gets designed).

[ncybul]

Following up on the size-cap comment above with a narrower point: the wire header names themselves are long — _dd.p.llmobs_ml_app, _dd.p.llmobs_sid, _dd.p.llmobs_pagent_span_id, _dd.p.llmobs_pagent_name, _dd.p.llmobs_parent_id — and every byte of key name eats into the same x-datadog-tags budget as the value itself. Given we have a hard cap on that header, I'd have preferred short wire keys mapped to longer, readable internal tag names — a translation layer at the codec boundary, rather than requiring the header key and the tag name to be the same string. That also would have sidestepped the character-escaping problem this PR spends real effort on: a mapping only needs to be reversible, not identical to something a human reads.

In retrospect, though — dd-trace-py/js/go have already shipped these exact key names on the wire, so changing them now isn't a Java-only decision, it's a cross-language migration. Given that commitment is already made, I don't think it's fair to hold this PR to fixing it. Noting it here mainly so it's on record for whenever the format does get revisited (e.g. if a versioned propagation-tag scheme ever gets designed).

@dougqh We have plans to shift from propagation tags to baggage across all of the SDKs. However, given that we still need propagation tags for backwards compatibility as of right now, we're thinking of implementing both mechanisms and then removing _dd.p.llmobs_ml_app, _dd.p.llmobs_sid, _dd.p.llmobs_pagent_span_id, _dd.p.llmobs_pagent_name, _dd.p.llmobs_parent_id in the next major release. So this is more of a temporary / stop-gap solution to get this working for customers who have already requested distributed tracing support for java.