Repository navigation
Preserve Unicode characters in PowerShell path completions - #58
Draft
warp-factories[bot] wants to merge 2 commits into
Draft
warp-factories[bot] wants to merge 2 commits into
warp-factories[bot] wants to merge 2 commits into
Conversation
Warp renders its own input editor, so a path suggestion containing non-ASCII characters (e.g. Turkish 'ü' in 'Masaüstü') displays correctly in the completions menu. But when the accepted command is executed, it is written to the ConPTY input pipe as UTF-8 bytes, and the console session decodes those bytes with the console input code page - which Windows PowerShell 5.1 defaults to the system's legacy OEM/ANSI code page, not UTF-8. The non-ASCII characters therefore arrive mangled in the shell (the 'cd Masa??st?' signature from warpdotdev#9000) and the path no longer resolves. Windows PowerShell 5.1's $OutputEncoding also defaults to strict ASCII, turning every non-ASCII character into a literal '?' when piping to a native program. Pin the console input/output encodings and $OutputEncoding to BOM-less UTF-8 when warpifying PowerShell, so accepted path suggestions survive the round trip. The BOM-less encoding instance also avoids Windows PowerShell 5.1 prepending a UTF-8 BOM to native stdin once the input code page is 65001. Also add client-side regression tests proving completion acceptance preserves non-ASCII characters verbatim - with multi-byte characters both left of the completed token (byte-span safety) and inside it - and that a shell-reported replacement span is floored to char boundaries instead of slicing a multi-byte character. Fixes warpdotdev#9000
Author
|
This PR was generated with Warp. Comment |
Two review findings on de02f5b: - The native-completions test used a single candidate, which the completer auto-inserts (InsertSingle) instead of opening the menu, so the menu assertion timed out; and it typed 'cd Mäs', whose bundled specs can preempt the native dispatch. Use a spec-less command and two candidates sharing only a short common prefix so the menu opens, then confirm the selected item and assert the buffer. - Move the UTF-8 encoding pin into Warp-Finish-Bootstrap so it runs after the user's profiles, which could otherwise reset the encodings back to the OEM/ANSI code page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description
Fixes non-ASCII characters (e.g. Turkish
ü,ş,ğ,İ) being corrupted when a path suggested by autocomplete is accepted in PowerShell.Root cause. Warp renders its own input editor, so the suggestion displays correctly, but the accepted command is written to the ConPTY input pipe as UTF-8 bytes and the console session decodes those bytes with the console input code page — which Windows PowerShell 5.1 defaults to the system's legacy OEM/ANSI code page (e.g. 857/1254 on Turkish Windows), not UTF-8. The multi-byte characters therefore arrive mangled in the shell (the
cd Masa??st?signature from the issue) and the path no longer resolves. In addition, Windows PowerShell 5.1's$OutputEncodingdefaults to strict ASCII, which turns every non-ASCII character into a literal?when piping text to a native program. This explains why it only happens in Warp: terminals like Windows Terminal host the console through the Unicode console API instead of translating ConPTY channel bytes through a legacy code page.Fix. When warpifying PowerShell, pin the console input/output encodings and
$OutputEncodingto BOM-less UTF-8 (guarded to Windows hosts, best-effort in atry/catch). The pin runs inWarp-Finish-Bootstrap, i.e. after the user's profiles, so a profile cannot reset it. A BOM-less encoding instance also avoids Windows PowerShell 5.1 prepending a UTF-8 BOM to native stdin once the input code page is 65001. All wire-protocol payloads (hooks, native completions) are hex-encoded ASCII, so they are unaffected by the code page change.Client-side hardening/tests. Added regression tests proving that accepting a completion containing non-ASCII characters inserts them verbatim — with multi-byte characters both left of the completed token (byte-span safety) and inside the candidate — and that a shell-reported replacement span is floored to UTF-8 char boundaries rather than slicing a multi-byte character (the client half of the UTF-16-code-unit → byte-offset contract; see the comment above
Warp-Run-GeneratorCommand-NativeCompletion).Linked Issue
ready-to-implement.Testing
./script/run— not possible here (the fix targets Windows PowerShell 5.1 + ConPTY; this sandbox is Linux without a Rust toolchain). CI will exercise the added Rust tests. On a Windows machine, the change can be verified by warpifying a PowerShell 5.1 session and confirming[Console]::InputEncoding.CodePagereports65001, then running the repro from the issue (mkdir Masaüstü, typecd Mas, accept the suggestion, press Enter).pwsh.ps1: parse-checked with the PowerShell 7.4 language parser, dot-sourced cleanly on a non-Windows host (the new Windows-only guard is correctly skipped), the encoding block was executed and verified to set input/output code page 65001 with a zero-length BOM preamble, and the file passesscript/lint_powershell(PSScriptAnalyzer with this repo's settings) with zero findings.app/src/terminal/input_tests.rs: addedcompletion_acceptance_preserves_non_ascii_characters,accepting_a_native_completion_preserves_non_ascii_characters(native-channel reply → menu → confirm → buffer), andnative_shell_replacement_span_is_floored_to_char_boundaries, mirroring the existing tab-completion/native-completions test patterns. The editor-level tests cannot observe the ConPTY encoding pin itself — that half is only observable on Windows PS 5.1 + ConPTY, which CI does not exercise; the tests pin the client-side contract (acceptance preserves non-ASCII text verbatim and byte spans stay char-boundary safe).Screenshots / Videos
Not applicable (no UI change; Windows-only runtime behavior).
Agent Mode