Repository navigation
fix(text): keep binary tool output out of the text pipeline (NUL / invalid UTF-8) - #460
Merged
Merged
Conversation
…valid UTF-8) Issue #457 (follow-up): Read on a binary file killed the turn's persistence with 'ERROR: invalid byte sequence for encoding UTF8: 0x00 (SQLSTATE 22021)'. NUL is valid UTF-8 (U+0000), so the existing SanitizeUTF8-based pgText passed it through — but PostgreSQL text cannot store 0x00 at all. SQLite stores it silently, which is why this only surfaced on the PG backend. Three layers: - tools/read.go: reject binary/non-UTF-8 files (NUL in the first 8 KiB, or >25% invalid UTF-8 bytes) for both the local and sandbox cat paths, with an actionable hint (Shell iconv/xxd/head -c/base64) instead of dumping bytes into the context. - textutil.SanitizeText: single scan repairing invalid sequences AND NUL to U+FFFD (zero-alloc for clean input); applied at the agent tool-result boundary (success, error and synthetic tool pairs) so context, SSE, offload files and DB all agree. - storage.pgText and cmd/xbot-migrate use SanitizeText as the last line of defence (migrated rows stay consistent with runtime writes). Tests: textutil, migrate, read binary detection, agent boundary sanitisation, shared contract case (both backends) and a real-PG test asserting a NUL-carrying message is storable and reads back clean.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
Author
|
@codex preview |
|
To use Codex here, create an environment for this repo. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题（issue #457 后续，生产日志实证）
Read读到二进制文件后，该轮对话持久化失败：日志里共 8 次（如
xbot-2026-10-24.log04:41:58，sessionweb:chat_18E098316EE7ACF5turn 30，Read返回后立即报错）。关键点：
0x00（NUL）是合法 UTF-8（U+0000），所以现有的pgText → textutil.SanitizeUTF8（只修非法序列）原样放行；但 PG 的text类型不能存 NUL，服务端拒绝整批 INSERT。SQLite 把 TEXT 当裸字节存，所以本地完全看不见，切到 PG 才爆。现场 pgx 事务探针（BEGIN + SELECT + ROLLBACK）实测：
修复（三层）
tools/read.go：新增detectBinary（本地 + sandboxcat两条路径共用）。前 8 KiB 内出现 NUL 或 >25% 非法 UTF-8 字节 ⇒ 直接返回binary or non-UTF-8 file (NUL byte at offset N): <path> — Read returns UTF-8 text only; use Shell (iconv / xxd / head -c / base64) to inspect or convert it不再把原始字节当文本塞进上下文（同时省掉几 MB 垃圾 token）。单字节损坏的文本文件仍在阈值内，正常读取。
textutil.SanitizeText= 非法序列 + NUL → U+FFFD（干净输入零分配）；在agent.updateToolResultLine（成功/失败）与newSyntheticToolPair统一执行 ⇒ 模型上下文 / SSE / offload 文件 / DB 四处同源干净。SanitizeUTF8保持"只修编码"语义不变。storage.pgText改用SanitizeText；cmd/xbot-migrate的sanitizeUTF8同样委托它（否则 SQLite 历史行里的 NUL 会让迁移失败）。验证
internal/textutil：NUL/非法序列/合法串；SanitizeUTF8仍保留 NUL（语义不变）cmd/xbot-migrate：含 NUL 的历史行可修复tools/read_binary_test.go：二进制（含 NUL / 纯 0xff）必须拒绝；单字节损坏的文本仍可读；8 KiB 探针边界agent/tool_output_sanitize_test.go：工具结果成功/失败/注入三条路径storage：共享契约用例（SQLite+PG）+ 真实 PGTestPGSessionStoreNULMessageStoredSanitized（写入成功、读回无 NUL 且是合法 UTF-8）文档
docs/agent/gotchas-tools.md、docs/agent/storage.md、AGENTS.md一行索引；docs-site/content/{en,zh-cn}/features/tools.md的Read说明同步（二进制会被拒绝并提示用 Shell）。