Separate parser token inspection from consumption by mununki · Pull Request #8633 · rescript-lang/rescript

mununki · 2026-09-06T16:19:38Z

Resolves: #8629

This PR separates token inspection from consumption, simplifying parser state management and allowing Diamond mode and the separate prev_end_pos field to be removed.

Correct rollback also uncovered invalid fixtures that previously passed because diagnostics were suppressed. Local benchmarks show roughly 5% lower parsing time and 2.9% less memory allocated.

Previously, make scanned the first token immediately, and next scanned its successor. Inspecting the following token required temporarily advancing the parser through lookahead:

let parser = Parser.make "let x = 1" "Example.res"
let token = parser.token
let following =
  Parser.lookahead parser (fun parser ->
    Parser.next parser;
    parser.token)

let () = Parser.next parser
let consumed_end = parser.prev_end_pos

Now, make does not scan tokens. peek and peek2 read and cache tokens without advancing the consumed position. next consumes the current token without scanning its successor:

let parser = Parser.make "let x = 1" "Example.res"
let token = Parser.peek parser
let following = Parser.peek2 parser

let () = Parser.next parser
let consumed_end = Parser.position parser

Main changes:

Add lazy token inspection with a bounded two-token cache.
Interpret angle-bracket operators in parser context, removing Diamond mode.
Consolidate consumed source positions through Parser.position.
Centralize speculative parsing and rollback, including diagnostics and error suppression.
Improve shared delimiter recovery, preserve useful correction hints, and correct invalid fixtures.

Parsing time change compared with pre-refactor master (5357fe4ac), where negative means faster:

Input	Compiler parsing	Formatter parsing
Real sources, 239 files combined	−4.65%	−4.96%
Runtime	−3.99%	−3.83%
Belt	−3.32%	−3.52%
Nested generics (synthetic)	−8.43%	−8.52%
Operators (synthetic)	−10.13%	−10.32%

Measured locally with 500 paired runs per input and mode. Memory allocated for the combined real sources decreased by 2.92% in both modes. Measurements cover parsing only, excluding type checking, code generation, and formatted output.

Validation: make test, make test-syntax-roundtrip, and added cursor/recovery regression coverage.

chatgpt-codex-connector · 2026-09-06T16:19:50Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T12:41:20.084844Z	`b1102fa`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1fb9db09a7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mununki · 2026-09-06T16:30:24Z

@@ -1,5 +1,5 @@
 @val
-external defaults: {x: int} = "defaults"
+external defaults: {"x": int} = "defaults"


These fixtures previously passed because speculative lookahead left error reporting disabled. Quoting the fields makes these non-function external types valid while preserving the object-type AST the old parser produced.

mununki · 2026-09-06T16:31:34Z

+
+
+  Syntax error!
+  syntax_tests/data/parsing/errors/typexpr/typeConstructorArgs.res:9:16


Line 9 was already invalid, but this diagnostic was previously missing. The parser now reports the invalid opening parenthesis separately from the preceding line's error and suggests option<node<int>>.

mununki · 2026-09-06T16:32:35Z

 type nonrec 'a t = private 'a Belt.Map.t
-type nonrec t = option
-;;node < (int >> ([%rescript.exprhole ]))
+type nonrec t = int node option


The extra < is still reported as an error. Previously, << was read as a shift operator, so recovery split the type declaration into option and a malformed expression. Recovery now preserves the type structure as option<node<int>>.

…-parser-token-cursor Signed-off-by: mununki <woonki.moon@gmail.com>

codecov · 2026-09-06T18:03:55Z

Codecov Report

❌ Patch coverage is 98.25000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.91%. Comparing base (0b4bd0a) to head (b1102fa).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
compiler/syntax/src/res_parser.ml	96.36%	4 Missing ⚠️
tests/ounit_tests/ounit_parser_cursor_tests.ml	99.13%	2 Missing ⚠️
tests/ounit_tests/ounit_parser_recovery_tests.ml	97.67%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8633      +/-   ##
==========================================
+ Coverage   77.58%   77.91%   +0.33%     
==========================================
  Files         475      477       +2     
  Lines       64051    64609     +558     
==========================================
+ Hits        49693    50343     +650     
+ Misses      14358    14266      -92

Files with missing lines	Coverage Δ
compiler/syntax/src/res_core.ml	`91.85% <ø> (+0.25%)`	⬆️
compiler/syntax/src/res_diagnostics.ml	`80.43% <100.00%> (+2.65%)`	⬆️
compiler/syntax/src/res_driver.ml	`78.43% <ø> (ø)`
compiler/syntax/src/res_grammar.ml	`42.56% <100.00%> (+0.51%)`	⬆️
compiler/syntax/src/res_scanner.ml	`91.08% <100.00%> (+0.52%)`	⬆️
...s/ounit_tests/ounit_constructor_arguments_tests.ml	`86.66% <100.00%> (+0.36%)`	⬆️
tests/ounit_tests/ounit_tests_main.ml	`100.00% <ø> (ø)`
tests/ounit_tests/ounit_parser_recovery_tests.ml	`97.67% <97.67%> (ø)`
tests/ounit_tests/ounit_parser_cursor_tests.ml	`99.13% <99.13%> (ø)`
compiler/syntax/src/res_parser.ml	`94.95% <96.36%> (+1.73%)`	⬆️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pkg-pr-new · 2026-09-06T18:04:15Z

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript-lang/rescript@8633

@rescript/belt

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/belt@8633

@rescript/darwin-arm64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/darwin-arm64@8633

@rescript/darwin-x64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/darwin-x64@8633

@rescript/linux-arm64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/linux-arm64@8633

@rescript/linux-x64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/linux-x64@8633

@rescript/runtime

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/runtime@8633

@rescript/win32-x64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/win32-x64@8633

commit: b1102fa

github-actions · 2026-09-06T18:12:45Z

Developer playground preview: https://rescript-lang.github.io/rescript/dev-playground/?version=pr-8633

mununki · 2026-09-07T00:03:10Z

/codex review

cknitt · 2026-09-07T06:26:00Z

@codex review

chatgpt-codex-connector

💡 Codex Review

rescript/tests/syntax_benchmarks/benchmark.ml

Lines 276 to 278 in 6b03ec6

    
           |> Seq.iteri (fun i json -> 
        
               print_endline (if i == 0 then "[" else ","); 
        
               print_string (Yojson.to_string json));

Emit an opening bracket for empty manifests

When --parse-manifest contains an empty JSON object, benchmarks and the flattened sequence are empty, so this callback never runs and the unconditional closing bracket produces only \n]. That is invalid JSON and breaks automation consuming the benchmark output; print [ independently of the first result or serialize the completed result list.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mununki · 2026-09-07T12:32:56Z

@codex review

chatgpt-codex-connector · 2026-09-07T12:41:18Z

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: b1102fa23d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Separate parser token inspection from consumption

1fb9db0

chatgpt-codex-connector Bot reviewed Sep 6, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

mununki commented Sep 6, 2026

View reviewed changes

mununki added 2 commits September 7, 2026 01:35

Replace changelog placeholders with the PR link

c0bcd99

Merge remote-tracking branch 'upstream/master' into refactor/simplify…

6b03ec6

…-parser-token-cursor Signed-off-by: mununki <woonki.moon@gmail.com>

chatgpt-codex-connector Bot reviewed Sep 7, 2026

View reviewed changes

Fix benchmark JSON output for empty manifests

b1102fa



		Syntax error!
		syntax_tests/data/parsing/errors/typexpr/typeConstructorArgs.res:9:16

	\|> Seq.iteri (fun i json ->
	print_endline (if i == 0 then "[" else ",");
	print_string (Yojson.to_string json));

Uh oh!

Conversation

mununki commented Sep 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mununki Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mununki Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mununki Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pkg-pr-new Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 6, 2026

Uh oh!

mununki commented Sep 7, 2026

Uh oh!

cknitt commented Sep 7, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

mununki commented Sep 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chatgpt-codex-connector Bot commented Sep 6, 2026 •

edited

Loading

codecov Bot commented Sep 6, 2026 •

edited

Loading

pkg-pr-new Bot commented Sep 6, 2026 •

edited

Loading