Repository navigation
Encode Unicode tables into each solver query instead of as global constraints - #547
Merged
Merged
Conversation
…straints Mask predicates expand inline via CharMask.smt_matches, and integer maps become IntTransforms: segments of key+offset or constant, applied as a balanced If tree with the offset outside the tree. Nothing is asserted globally, so solver checks no longer reprocess the tables on every call. UnicodeMaskCache is replaced by module-level smt_* functions, since the encodings no longer depend on a solver. Co-authored-by: wtfyd-agent[bot] <328989958+wtfyd-agent[bot]@users.noreply.github.com>
Co-authored-by: wtfyd-agent[bot] <328989958+wtfyd-agent[bot]@users.noreply.github.com>
Co-authored-by: wtfyd-agent[bot] <328989958+wtfyd-agent[bot]@users.noreply.github.com>
Co-authored-by: wtfyd-agent[bot] <328989958+wtfyd-agent[bot]@users.noreply.github.com>
pschanely
approved these changes
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a symbolic string method that needs Unicode data (
upper,lower,casefold,title,isalpha,isspace,strip,unicodedata.digit, ...) is first used on a path,UnicodeMaskCacheadds that table to the solver as a global constraint. Masks are added as a quantified axiom (ForAll c. is_<name>(c) == <ranges>), and case and numeric maps as one big conjunction of ground equations (toupper_1st(k) == vfor each of ~1,550 code points).make_default_solverbuilds the solver from thesmttactic, which is not incremental: everycheck()processes every assertion from scratch. So once a table is loaded, every later check on that path pays for it, including checks that have nothing to do with strings:toupper_exists+spacemaskstoupper_1stmaptoupper_exists+toupper_1st+spaceThe slowest single queries were inverse lookups such as
97 == toupper_1st(s)(frompartition/replaceafter.upper()), at hundreds of milliseconds each. The table above and the benchmarks below are on z3 5.3.1.Change
CharMask.smt_matches, which builds a disjunction of ranges. Nothing is asserted globally.relibalready used masks this way.decimal/digitvalues, numeric numerator and denominator) becomeIntTransforms. Sorted keys are grouped into segments where every key maps tokey + offsetor to a constant. Keys outside the map never reach a transform, because every call is guarded by a mask fork, so segments can span gaps between keys.toupper_1stgoes from 1,552 entries to 187 segments. Each use builds a balancedIftree over segment starts (about 8 deep).c + If(..., k1, k2)), not at each leaf (If(..., c + k1, c + k2)). With offsets at the leaves, z3's auto-config can classify a query as difference logic, and the difference-logic solver then gives up (unknown, "incomplete (theory difference-logic)") on atoms likec - (c + 205) <= 0. A 10-Ifreduction reproduces this, andtest_int_transform_offsets_are_decidablecovers it.UnicodeMaskCacheandspace.extra(UnicodeMaskCache)are replaced by module-level functions (smt_is_alpha,smt_upper_1st,smt_upper_has_2nd, ...). Each builds its table lazily once and is applied directly:smt_is_alpha(smt_codepoint)instead ofspace.extra(UnicodeMaskCache).alpha()(smt_codepoint).decimal_exists,digit_existsandnumeric_existswere the same sets asdecimal,digitandnumeric(isdecimal()matchesunicodedata.decimal(ch, None) is not Noneon every code point, and likewise for digit and numeric).CharMask.interpret_smt_functionand_INTERPRETATION_CACHE.Results
All numbers are on z3 5.3.1 (the dev pin) and Python 3.14, with
PYTHONHASHSEED=0. End to end, 40 iterations, two runs each:(s.upper() + t.strip()).replace("a", "bb")s.lower() == s.casefold() or s.isalpha() or s.title().isupper()s.title().isupper()int(s) if s.isdigit() and len(s) < 4 else len(s.split())sorted(x % 7 for x in xs)(no strings)Every solver check in these runs returned sat or unsat. There were no
unknowns on either branch.In isolation, a trivial check after loading the upper tables drops from 28 ms back to 0.14 ms, since nothing is loaded globally any more. The inverse lookup
upper(c) == 97(given thatcchanges underupper) drops from 474 ms to 37 ms.Every transform was checked against its source table on every key (
z3.simplify(smt_fn(k)) == table[k]), and every mask at each range boundary plus 300 random code points.Testing
unicode_categories_test,unicodedatalib_test,builtinslib_test,builtinslib_ch_test,relib_test,relib_ch_test,encodings_ch_test: 820 passed, 1 failed. The failure,check_divmod, also fails on main in this environment (z3 5.3.1, Python 3.14), taking 316 s alone. On this branch it passes alone in 3 s.strmethod andunicodedata.decimal/digit/numericleave a symbolic input symbolic (catches a method silently falling back to a realizing implementation),IntTransformsegmenting and lookup, exact match against every case map, and the difference-logic regression. That test fails when offsets are moved back to the leaves.-n auto, macOS, z3 5.3.1, Python 3.14): 15,011 passed, 25 xfailed, 816 skipped, 12 failed. Eleven failures were caused by my local environment and are unrelated to this change: tenfuzz_core_testturtle.*ops, andtest_uncategorized_ops_probe_cleanly, which runs the same ops (it passes on main in isolation). The last is thecheck_divmodfailure on main described above.pre-commit run --all-filespasses (black, isort, flake8, mypy, doctest, smoke tests).Open questions
alphahas several hundred ranges, so a query over many symbolic characters builds larger expressions. The benchmarks above don't show a cost, but a long string run throughisalpha()would be the case to watch.