fix: flush buffered OTEL logs by yashrajshuklaaa · Pull Request #2772 · kagent-dev/kagent

yashrajshuklaaa · 2026-09-09T15:20:49Z

What

Fix ForceFlush() so it flushes both the tracer and logger providers.

Why

GenAI audit logs are buffered by the OTEL batch log processor. When an Agent Substrate actor checkpoints right after the A2A response closes, those buffered logs can be left behind and never exported.

Changes

Flush the global LoggerProvider in ForceFlush().
Keep tracer flushing working as before.
Use the same timeout for both providers.
Added a regression test to make sure buffered logs are exported, even when the request context is already cancelled.
Updated the related comment from spans to telemetry.

Validation

go test ./adk/pkg/telemetry ./adk/pkg/a2a/server
go build ./adk/pkg/telemetry ./adk/pkg/a2a/server

All passed.

Fixes #2759

EItanya · 2026-09-09T19:10:20Z

@krisztianfekete

krisztianfekete

Thanks for the PR! Would you mind also fixing this on the Python side, address the comments and add test coverage for both runtimes and both signals?

krisztianfekete · 2026-09-10T10:11:15Z

 		return
 	}
+
 	flushCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), flushTimeout())


Can we give each provider its own budget, or flush logs first?

krisztianfekete · 2026-09-10T10:13:43Z

+// session's last message).
 // Uses its own detached timeout because the request context is typically
 // already canceled by the time deferred cleanup runs. The timeout defaults to
 // 3s and is configurable via KAGENT_TRACE_FLUSH_TIMEOUT_MS.


We probably want to rename this now that it covers both traces and logs.

yashrajshuklaaa · 2026-09-10T16:06:54Z

@krisztianfekete Thanks for the review. I’ve addressed the comments and updated the PR. fix now covers both Go and Python, flushing logs first while keeping a shared timeout budget. also added test coverage for both logs and traces in both runtimes and renamed the timeout setting to KAGENT_TELEMETRY_FLUSH_TIMEOUT_MS
Validation is passing locally and the changes are pushed to the PR.

yashrajshuklaaa · 2026-09-19T09:30:41Z

@mesutoezdil resolved all comments :) PTAL

yashrajshuklaaa · 2026-09-19T09:31:02Z

@krisztianfekete pls review this

krisztianfekete · 2026-09-21T09:20:11Z

This is nested under if tracing_enabled, so with OTEL_LOGGING_ENABLED=true and tracing off the new log flush is never called. Can you move out of the tracing branch?

Fixed. the post-response flush hook now installs when tracing_enabled or metrics_enabled or logging_enabled. Added test_configure_installs_post_response_flush_with_logs_only to cover the logs-only case.

krisztianfekete · 2026-09-21T09:21:09Z

+	flushCtx, cancel := context.WithTimeout(
+		context.WithoutCancel(ctx),
+		flushTimeout(),
+	)


Both implementations share this one deadline, while Python splits the budget with remaining_millis, can you please match the two?

Matched now. each provider in Go gets its own full FlushTimeout, the same way Python already passed the full timeout_millis to each provider's force_flush individually. No more shared/split deadline in either runtime.

krisztianfekete · 2026-09-21T09:23:00Z


 func flushTimeout() time.Duration {
-	if value := strings.TrimSpace(os.Getenv("KAGENT_TRACE_FLUSH_TIMEOUT_MS")); value != "" {
+	if value := strings.TrimSpace(os.Getenv("KAGENT_TELEMETRY_FLUSH_TIMEOUT_MS")); value != "" {


The rename drops any existing KAGENT_TRACE_FLUSH_TIMEOUT_MS override to the 3s default with no warning. Can we keep the old name as a fallback for a release and log a notice for the rename?

Moot after the rebase. the telemetry refactor removed the configurable timeout env var entirely in favor of a fixed constant so there's no override or fallback to preserve on either name.

yashrajshuklaaa · 2026-09-25T01:28:11Z

@krisztianfekete all review comments have been addressed. PTAL : )

krisztianfekete

Thanks for sticking with this, one last rebase needed. #2929 merged after your push and changed Providers.ForceFlush.

On the Go side, let's keep main's version and only add the logger. Drop the per-provider flush and use the one shared timeout. Flush logs first, then tracer, then meter, and include p.logger == nil in the early return.

Python changes should be fine after the rebase.

yashrajshuklaaa · 2026-09-25T20:18:09Z

Thanks for sticking with this, one last rebase needed. #2929 merged after your push and changed Providers.ForceFlush.

On the Go side, let's keep main's version and only add the logger. Drop the per-provider flush and use the one shared timeout. Flush logs first, then tracer, then meter, and include p.logger == nil in the early return.

Python changes should be fine after the rebase.

Rebased onto main after #2929. Kept the flushRecordKey/WithFlushRecord skip-on-failure logic and single shared FlushTimeout, dropped the per-provider timeout split, and added the logger flush on top, keeping the logs to tracer to meter order via errors.Join. Also added p.logger == nil to the early-return check. Go and Python tests all passing on my end, PTAL 🙂

krisztianfekete · 2026-09-25T20:21:10Z

+	flush := func(f func(context.Context) error) error {
+		return f(ctx)
+	}


This wrapper does nothing now, please call p.logger.ForceFlush(ctx) and the others directly.

Signed-off-by: Yashraj Shukla <shuklayashraj68@gmail.com>

yashrajshuklaaa · 2026-09-26T13:51:15Z

thanks @krisztianfekete @EItanya 🎉☺️

yashrajshuklaaa requested a review from supreme-gg-gg as a code owner September 9, 2026 15:20

github-actions Bot added the bug Something isn't working label Sep 9, 2026

krisztianfekete requested changes Sep 10, 2026

View reviewed changes

yashrajshuklaaa requested a review from a team as a code owner September 10, 2026 16:04

moezdil reviewed Sep 14, 2026

View reviewed changes

Comment thread go/adk/pkg/telemetry/tracing.go Outdated

Comment thread go/adk/pkg/telemetry/tracing.go Outdated

Comment thread python/packages/kagent-core/tests/test_tracing_configure.py Outdated

moezdil mentioned this pull request Sep 14, 2026

feat(observability): add OTEL traces and logs to harnesses #2805

Merged

yashrajshuklaaa force-pushed the fix/flush-otel-logger-provider branch from 8dc13f1 to bbc2055 Compare September 19, 2026 09:24

krisztianfekete reviewed Sep 21, 2026

View reviewed changes

yashrajshuklaaa force-pushed the fix/flush-otel-logger-provider branch from 8d34cf6 to c70733d Compare September 25, 2026 01:19

krisztianfekete reviewed Sep 25, 2026

View reviewed changes

yashrajshuklaaa force-pushed the fix/flush-otel-logger-provider branch from c70733d to b6eb9b7 Compare September 25, 2026 20:16

krisztianfekete reviewed Sep 25, 2026

View reviewed changes

fix: flush buffered OTEL logs on ForceFlush

f6f0e0e

Signed-off-by: Yashraj Shukla <shuklayashraj68@gmail.com>

yashrajshuklaaa force-pushed the fix/flush-otel-logger-provider branch from b6eb9b7 to f6f0e0e Compare September 25, 2026 20:33

krisztianfekete approved these changes Sep 25, 2026

View reviewed changes

EItanya added this pull request to the merge queue Sep 26, 2026

Merged via the queue into kagent-dev:main with commit 271c72e Sep 26, 2026
29 checks passed

Rachael-Graham mentioned this pull request Sep 29, 2026

Telemetry docs: rewrite for SDK-spec OTEL variables and semconv metrics (targets 1.0.0-alpha4) kagent-dev/website#531

Open

6 tasks

Conversation

yashrajshuklaaa commented Sep 9, 2026

What

Why

Changes

Validation

Uh oh!

EItanya commented Sep 9, 2026

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yashrajshuklaaa commented Sep 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yashrajshuklaaa commented Sep 19, 2026

Uh oh!

yashrajshuklaaa commented Sep 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yashrajshuklaaa commented Sep 25, 2026

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

yashrajshuklaaa commented Sep 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yashrajshuklaaa commented Sep 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants