Skip to content

Preserve Unicode characters in PowerShell path completions - #58

Draft
warp-factories[bot] wants to merge 2 commits into
masterfrom
factory/preserve-unicode-path-completions
Draft

warp-factories[bot] wants to merge 2 commits into
masterfrom
factory/preserve-unicode-path-completions

Conversation

@warp-factories

@warp-factories warp-factories Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Description

Fixes non-ASCII characters (e.g. Turkish ü, ş, ğ, İ) being corrupted when a path suggested by autocomplete is accepted in PowerShell.

Root cause. Warp renders its own input editor, so the suggestion displays correctly, but the accepted command is written to the ConPTY input pipe as UTF-8 bytes and the console session decodes those bytes with the console input code page — which Windows PowerShell 5.1 defaults to the system's legacy OEM/ANSI code page (e.g. 857/1254 on Turkish Windows), not UTF-8. The multi-byte characters therefore arrive mangled in the shell (the cd Masa??st? signature from the issue) and the path no longer resolves. In addition, Windows PowerShell 5.1's $OutputEncoding defaults to strict ASCII, which turns every non-ASCII character into a literal ? when piping text to a native program. This explains why it only happens in Warp: terminals like Windows Terminal host the console through the Unicode console API instead of translating ConPTY channel bytes through a legacy code page.

Fix. When warpifying PowerShell, pin the console input/output encodings and $OutputEncoding to BOM-less UTF-8 (guarded to Windows hosts, best-effort in a try/catch). The pin runs in Warp-Finish-Bootstrap, i.e. after the user's profiles, so a profile cannot reset it. A BOM-less encoding instance also avoids Windows PowerShell 5.1 prepending a UTF-8 BOM to native stdin once the input code page is 65001. All wire-protocol payloads (hooks, native completions) are hex-encoded ASCII, so they are unaffected by the code page change.

Client-side hardening/tests. Added regression tests proving that accepting a completion containing non-ASCII characters inserts them verbatim — with multi-byte characters both left of the completed token (byte-span safety) and inside the candidate — and that a shell-reported replacement span is floored to UTF-8 char boundaries rather than slicing a multi-byte character (the client half of the UTF-16-code-unit → byte-offset contract; see the comment above Warp-Run-GeneratorCommand-NativeCompletion).

Linked Issue

Testing

  • I have manually tested my changes locally with ./script/run — not possible here (the fix targets Windows PowerShell 5.1 + ConPTY; this sandbox is Linux without a Rust toolchain). CI will exercise the added Rust tests. On a Windows machine, the change can be verified by warpifying a PowerShell 5.1 session and confirming [Console]::InputEncoding.CodePage reports 65001, then running the repro from the issue (mkdir Masaüstü, type cd Mas, accept the suggestion, press Enter).
  • pwsh.ps1: parse-checked with the PowerShell 7.4 language parser, dot-sourced cleanly on a non-Windows host (the new Windows-only guard is correctly skipped), the encoding block was executed and verified to set input/output code page 65001 with a zero-length BOM preamble, and the file passes script/lint_powershell (PSScriptAnalyzer with this repo's settings) with zero findings.
  • app/src/terminal/input_tests.rs: added completion_acceptance_preserves_non_ascii_characters, accepting_a_native_completion_preserves_non_ascii_characters (native-channel reply → menu → confirm → buffer), and native_shell_replacement_span_is_floored_to_char_boundaries, mirroring the existing tab-completion/native-completions test patterns. The editor-level tests cannot observe the ConPTY encoding pin itself — that half is only observable on Windows PS 5.1 + ConPTY, which CI does not exercise; the tests pin the client-side contract (acceptance preserves non-ASCII text verbatim and byte spans stay char-boundary safe).

Screenshots / Videos

Not applicable (no UI change; Windows-only runtime behavior).

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Warp renders its own input editor, so a path suggestion containing
non-ASCII characters (e.g. Turkish 'ü' in 'Masaüstü') displays correctly
in the completions menu. But when the accepted command is executed, it is
written to the ConPTY input pipe as UTF-8 bytes, and the console session
decodes those bytes with the console input code page - which Windows
PowerShell 5.1 defaults to the system's legacy OEM/ANSI code page, not
UTF-8. The non-ASCII characters therefore arrive mangled in the shell
(the 'cd Masa??st?' signature from warpdotdev#9000) and the path no longer
resolves. Windows PowerShell 5.1's $OutputEncoding also defaults to
strict ASCII, turning every non-ASCII character into a literal '?' when
piping to a native program.

Pin the console input/output encodings and $OutputEncoding to BOM-less
UTF-8 when warpifying PowerShell, so accepted path suggestions survive
the round trip. The BOM-less encoding instance also avoids Windows
PowerShell 5.1 prepending a UTF-8 BOM to native stdin once the input
code page is 65001.

Also add client-side regression tests proving completion acceptance
preserves non-ASCII characters verbatim - with multi-byte characters
both left of the completed token (byte-span safety) and inside it -
and that a shell-reported replacement span is floored to char
boundaries instead of slicing a multi-byte character.

Fixes warpdotdev#9000
@warp-factories

Copy link
Copy Markdown
Author

This PR was generated with Warp.

Comment @warp on this PR to send it follow-up work.

View run View conversation

Two review findings on de02f5b:

- The native-completions test used a single candidate, which the
  completer auto-inserts (InsertSingle) instead of opening the menu, so
  the menu assertion timed out; and it typed 'cd Mäs', whose bundled
  specs can preempt the native dispatch. Use a spec-less command and two
  candidates sharing only a short common prefix so the menu opens, then
  confirm the selected item and assert the buffer.

- Move the UTF-8 encoding pin into Warp-Finish-Bootstrap so it runs
  after the user's profiles, which could otherwise reset the encodings
  back to the OEM/ANSI code page.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants