Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3c64f98
Prepare for regexp_parser update
jaynetics Oct 29, 2026
8f558b9
Fix capture numbering in atomic and possessive groups
jaynetics Oct 29, 2026
c795cd5
Count and map every capture expanded by subexpression calls
jaynetics Oct 29, 2026
b4ef4e4
Preserve required captures when expanding conditionals
jaynetics Oct 29, 2026
2f7e44d
Keep unbounded limits valid in conditional expansions
jaynetics Oct 29, 2026
0afb9be
Require participation for multiplexed named backreferences
jaynetics Oct 29, 2026
13f47a9
Handle repeated keep marks and root alternatives
jaynetics Oct 29, 2026
0dbed14
Emit valid minimum bounds for absence group quantifiers
jaynetics Oct 29, 2026
9fb81c2
Use unicode-compatible ranges for expanded astral properties
jaynetics Oct 29, 2026
109d572
Preserve astral characters in Unicode character types
jaynetics Oct 29, 2026
b846ed2
Separate numeric backreferences from following literal digits
jaynetics Oct 29, 2026
ce5ad32
Match nonword non-boundaries in ES2018 conversions
jaynetics Oct 29, 2026
acd8803
Escape Unicode line separators in JavaScript regex literals
jaynetics Oct 29, 2026
72ffa9e
Preserve earlier call captures when optional copies do not participate
jaynetics Oct 29, 2026
7f1903b
Resolve conditionals against captures made by subexpression calls
jaynetics Oct 29, 2026
81a3de1
Preserve quantifier preference when guarding duplicate-name references
jaynetics Oct 29, 2026
ba5a311
Omit absence-group tails for zero-width forbidden patterns
jaynetics Oct 29, 2026
d17c761
Include Unicode numbers in word-boundary expansions
jaynetics Oct 29, 2026
c1df7be
Rename crawl -> walk
jaynetics Oct 30, 2026
e2e6721
Update prop map
jaynetics Oct 30, 2026
4fdef51
Extract second-pass steps into subclasses
jaynetics Oct 30, 2026
2d48d3c
Preserve Ruby capture participation and loop semantics
jaynetics Oct 30, 2026
87cb8e1
Preserve lazy suffixes when normalizing interval bounds
jaynetics Oct 30, 2026
acf942e
Clean up and clarify
jaynetics Oct 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .rspec
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
--require spec_helper
2 changes: 1 addition & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ AllCops:
- ruby
- rake
SuggestExtensions: false
TargetRubyVersion: 2.1
TargetRubyVersion: 2.6

Lint:
Enabled: true
Expand Down
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,35 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## Unreleased

The minimum ruby version is now 2.6.

### Added

- up-to-date unicode properties map, e.g. for `\p{Dash}`

### Fixed

- capture numbering in atomic and possessive groups
* e.g. `(?>a(b))\1`
- backreferences and conditionals after optional or repeated captures
* e.g. `(a)?\1`
- handling of backreference numbers followed by space and digits in x-mode
* e.g. `/(a)\1 0/x`
- capture choices and repeat limits when expanding conditionals
* e.g. `(a){0}(?(1)b|c)`
- greedy and lazy matching order when expanding repeated groups
* e.g. `(?:(aa)|(a))+?\1\2`
- handling of repeated keep marks (`\K`) and root alternatives
* e.g. `a\Kb\Kc|d`
- repeat limits and zero-width patterns in absence groups
* e.g. `(?~(?=a))`
- handling of astral characters in some Unicode types and properties
* e.g. `(?u:\d)` matching `𝟡`
- handling of Unicode numbers in word boundaries
* e.g. `\b½\b`
- missing escaping of Unicode line separators in regex literal output
* e.g. `Regexp.new("\u2028")`

## [3.14.0] - 2025-08-03

### Added
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,8 @@ Many Regexp tokens work in JavaScript just as they do in Ruby, or allow for a st

The tricky bit here is that these expressions may be nested, and that their expansions may increase the capturing group count. This means that any following backreferences need an update. E.g. <code>(.{3})\g<1>(.)<b>\2</b></code> (which matches strings like "FooBarXX") converts to <code>(.{3})(.{3})(.)<b>\3</b></code>.

**Non-participating capturing groups** are another tricky case and require complex branching in the pattern. See #26 for details.

## Contributions

Feel free to send suggestions, point out issues, or submit pull requests.
Expand Down
2 changes: 1 addition & 1 deletion js_regex.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Gem::Specification.new do |s|

s.files = Dir['lib/**/*.{csv,rb}']

s.required_ruby_version = '>= 2.1.0'
s.required_ruby_version = '>= 2.6.0'

s.add_dependency 'character_set', '~> 1.4'
s.add_dependency 'regexp_parser', '~> 2.11'
Expand Down
4 changes: 2 additions & 2 deletions lib/js_regex/converter/anchor_converter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ def convert_nonboundary

# This is an approximation to the word boundary behavior in Ruby, c.f.
# https://github.com/ruby/ruby/blob/08476c45/tool/enc-unicode.rb#L130
W = '\d\p{L}\p{M}\p{Pc}'
W = '\p{N}\p{L}\p{M}\p{Pc}'
BOUNDARY_EXPANSION = "(?:(?<=[#{W}])(?=[^#{W}]|$)|(?<=[^#{W}]|^)(?=[#{W}]))"
NONBOUNDARY_EXPANSION = "(?<=[#{W}])(?=[#{W}])"
NONBOUNDARY_EXPANSION = "(?:(?<=[#{W}])(?=[#{W}])|(?<=[^#{W}]|^)(?=[^#{W}]|$))"

def pass_boundary_with_warning
warn_of("The anchor '#{data}' at index #{expression.ts} only works "\
Expand Down
67 changes: 16 additions & 51 deletions lib/js_regex/converter/backreference_converter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,28 +32,26 @@ def convert_name_ref
end

def convert_to_plain_num_ref
position = new_position

# Check if this backreference refers to a group that was recursively called
original_group = target_position
if (recursive_position = context.get_recursive_group_position(original_group))
# Use the position of the group created by the recursive call
position = recursive_position
positions = context.get_recursive_group_positions(target_position)
positions = [new_position] if positions.empty?
references = positions.map do |position|
Node.new("\\#{position}", reference: position, type: :backref)
end
if references.length > 1 && Utils::Captures.nonempty?(expression.referenced_expression)
Node.new(*references, type: :capture_backref)
else
references.first
end

text = "\\#{position}#{'(?:)' if expression.x?}"
Node.new(text, reference: position, type: :backref)
end

def convert_multiplexed_name_ref
# Create alternation of all groups with the same name
positions = expression.referenced_expressions.map do |ref_exp|
context.new_capturing_group_position(ref_exp.number)
# Ruby tries groups with the same name from last to first.
references = expression.referenced_expressions.reverse.map do |ref_exp|
position = context.new_capturing_group_position(ref_exp.number)
Node.new("\\#{position}", reference: position, type: :backref)
end

# Build alternation like (?:\1|\2)
alternation = positions.map { |pos| "\\#{pos}" }.join('|')
Node.new("(?:#{alternation})")
Node.new(*references, type: :multiplex_backref)
end

def new_position
Expand All @@ -72,48 +70,15 @@ def convert_call

context.count_recursion(expression)

# Track groups before the wrapper group is added
groups_before_wrapper = context.capturing_group_count

context.increment_local_capturing_group_count
target_copy = expression.referenced_expression.unquantified_clone
# avoid "Duplicate capture group name" error in JS
target_copy.token = :capture if target_copy.is?(:named, :group)
context.start_subexp_recursion
result = convert_expression(target_copy)
ref = expression.referenced_expression
result = Converter.for(ref).convert(ref, context, quantified: false)
context.end_subexp_recursion

# Track all groups created during this recursive call
# This handles both the directly called group and any nested groups within it
# Get all group numbers from the referenced expression
original_groups = collect_group_numbers(expression.referenced_expression)

# The first new group number is groups_before_wrapper + 1
# (the wrapper group from increment_local_capturing_group_count doesn't appear in output)
first_new_group = groups_before_wrapper + 1

# Map each original group to its corresponding new group
# For example, if we recursively called group 1 which contains group 2,
# and this created groups 3 and 4, then:
# - group 1 -> group 3
# - group 2 -> group 4
original_groups.each_with_index do |old_group_num, index|
new_group_num = first_new_group + index
context.track_recursive_group_call(old_group_num, new_group_num)
end

# wrap in passive group if it is a full-pattern recursion
expression.reference == 0 ? Node.new('(?:', result, ')') : result
end

def collect_group_numbers(exp)
return [] if exp.terminal?

numbers = []
numbers << exp.number if exp.capturing?
exp.each_expression { |sub| numbers += collect_group_numbers(sub) }
numbers
end
end
end
end
49 changes: 24 additions & 25 deletions lib/js_regex/converter/base.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,15 @@ module Converter
#
class Base
# returns instance of Node with #quantifier attached.
def convert(expression, context)
def convert(expression, context, quantified: true)
self.context = context
self.expression = expression

node = convert_data
node = Node.new(node) if node.instance_of?(String)
apply_quantifier(node)
if quantified && expression.quantifier&.possessive?
wrap_in_backrefed_lookahead { convert_node(quantified) }
else
convert_node(quantified)
end
end

private
Expand All @@ -33,19 +35,17 @@ def data
end
alias pass_through data

def apply_quantifier(node)
return node if node.dropped? || (qtf = expression.quantifier).nil?
def convert_node(quantified)
node = convert_data
node = Node.new(node) if node.instance_of?(String)
quantified ? apply_quantifier(node) : node
end

if qtf.possessive?
node.update(quantifier: qtf.dup.tap { |q| q.text = q.text[0..-2] })
return wrap_in_backrefed_lookahead(node)
elsif qtf.token == :interval && qtf.text[0..1] == "{,"
node.update(quantifier: qtf.dup.tap { |q| q.text = "{0,#{q.max}}" })
else
node.update(quantifier: qtf)
end
def apply_quantifier(node)
return node if node.dropped? ||
(qtf = Node::Quantifier.from_expression(expression)).nil?

node
node.update(quantifier: qtf)
end

def convert_subexpressions
Expand All @@ -70,25 +70,24 @@ def warn_of_unsupported_feature(description = nil, min_target: nil)
end

def warn_of(text)
if context.fail_fast
raise ConversionError, text.sub(/^Dropped /, '')
else
context.warnings << text
end
context.warn_of(text)
end

def drop
Node.new(type: :dropped)
end
alias drop_without_warning drop

def wrap_in_backrefed_lookahead(content)
number = context.capturing_group_count + 1
def wrap_in_backrefed_lookahead
number = context.capture_group(added: true)
content = yield
if content.is_a?(Node) && content.dropped?
context.cancel_added_capturing_group
return content
end
backref_node = Node.new("\\#{number}", reference: number, type: :backref)
backrefed_group = Node.new('(', *content, ')', reference: number, type: :captured_group)
context.increment_local_capturing_group_count
# The surrounding group is added so that quantifiers apply to the whole.
# Without it, `(?:)` would need to be appended as literal digits may follow.
# The surrounding group makes quantifiers apply to the whole expansion.
Node.new('(?:(?=', backrefed_group, ')', backref_node, ')')
end

Expand Down
14 changes: 10 additions & 4 deletions lib/js_regex/converter/conditional_converter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,17 @@ def convert_data
end

def mark_conditional_for_second_pass
reference = expression.referenced_expression.number
node = Node.new('(?:', reference: reference, type: :conditional)
expression.branches.each do |branch|
node << Node.new('(?:', convert_expression(branch), ')')
target = expression.referenced_expression
positions = context.get_recursive_group_positions(target.number)
branches = expression.branches.map { |branch| Node.new('(?:', convert_expression(branch), ')') }
if positions.length > 1 && Utils::Captures.nonempty?(target)
references = positions.map { |position| Node.new(reference: position) }
return Node.new(Node.new(*references), *branches, type: :capture_conditional)
end

reference = positions.first || context.new_capturing_group_position(target.number)
node = Node.new('(?:', reference: reference, type: :conditional)
branches.each { |branch| node << branch }
node << ')'
end
end
Expand Down
34 changes: 27 additions & 7 deletions lib/js_regex/converter/context.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,20 @@ def initialize(case_insensitive_root: false, fail_fast: false, target: nil)
self.required_options_hash = {}
self.warnings = []
self.recursive_group_map = {}
self.recursive_group_history = Hash.new { |hash, key| hash[key] = [] }

self.case_insensitive_root = case_insensitive_root
self.target = target
end

def warn_of(text)
if fail_fast
raise ConversionError, text.sub(/^Dropped /, '')
else
warnings << text
end
end

# target context

def es_2015_or_higher?
Expand Down Expand Up @@ -57,7 +66,10 @@ def required_options

# group context

def capture_group
def capture_group(added: false)
if added
added_capturing_groups_after_group[original_capturing_group_count] += 1
end
self.capturing_group_count = capturing_group_count + 1
end

Expand All @@ -69,9 +81,9 @@ def end_atomic_group
self.in_atomic_group = false
end

def increment_local_capturing_group_count
added_capturing_groups_after_group[original_capturing_group_count] += 1
capture_group
def cancel_added_capturing_group
added_capturing_groups_after_group[original_capturing_group_count] -= 1
self.capturing_group_count = capturing_group_count - 1
end

def recursions(exp)
Expand All @@ -93,14 +105,14 @@ def start_subexp_recursion
end

def end_subexp_recursion
self.in_subexp_recursion = false
# Pop the last recursion from stack when exiting
recursion_stack.pop if recursion_stack.any?
self.in_subexp_recursion = recursion_stack.any?
end

# Get the number of groups at the start of the current recursion
def recursion_start_group_count
self.recursion_start_group_count || 0
@recursion_start_group_count || 0
end

# takes and returns 1-indexed group positions.
Expand All @@ -120,20 +132,28 @@ def original_capturing_group_count
# Track that a group was created by a recursive call
def track_recursive_group_call(original_group_num, new_group_num)
recursive_group_map[original_group_num] = new_group_num
history = recursive_group_history[original_group_num]
history.delete(new_group_num)
history.unshift(new_group_num)
end

# Get the group number created by a recursive call
def get_recursive_group_position(original_group_num)
recursive_group_map[original_group_num]
end

def get_recursive_group_positions(original_group_num)
recursive_group_history[original_group_num].dup
end

private

attr_accessor :added_capturing_groups_after_group,
:recursions_per_expression,
:recursion_stack,
:required_options_hash,
:recursive_group_map,
:recursive_group_history,
:target

attr_writer :capturing_group_count,
Expand All @@ -145,7 +165,7 @@ def get_recursive_group_position(original_group_num)
:warnings

def total_added_capturing_groups
added_capturing_groups_after_group.values.inject(0, &:+)
added_capturing_groups_after_group.values.sum
end
end
end
Expand Down
Loading
Loading