Optimize XPath step by tompng · Pull Request #315 · ruby/rexml

tompng · 2026-05-21T13:11:49Z

Depends on ~~#317~~ (merged) and #320

Refactor step so that nodeset materialization is deferred. Instead of building the full nodeset up front and filtering through predicates,
each axis returns a scan descriptor ([generator_name, generator_argument]), and step picks a scan strategy based on the predicates' shape.

Predicates are classified into three groups:

kind	examples	strategy passed to the generator
position-independent	`[@a="1"]`, `[name()="foo"]`, `[@a=@b]`	`:uniq` — emit deduplicated matching nodes
simple positional	`[N]`, `[position()=N]`, `[position()>N]`, `[position()<N]`	`[op, value]` — positional scan with one comparison
complex / position-dependent	`[position()*@a]`, `[last()-1]`, ...	`:nodesets` — fall back to per-anchor nodesets + the previous `evaluate_predicate` pipeline

Mixed predicate lists are split: position-independent predicates before the first positional predicate are folded into the node test;
predicates after it are applied per-node on the result.

Each axis can implement zero, one, or all of the three strategies. If a strategy is not implemented, the generator falls back to producing
:nodesets and the common slow path (non_optimized_raw_nodesets_select) handles dedup / positional filtering on flattened nodesets — i.e. the
same behavior as before this PR.

This pull request adds fast paths for:

descendant / descendant-or-self: :uniq (single DFS with a seen-set; this is what speeds up //a//a//a//a)
ancestor / ancestor-or-self: :uniq (parent-chain walk with a seen-set)
preceding-sibling / following-sibling: :uniq and [op, value] (sibling scan with anchor-index tracking)

Other axes (child, parent, self, attribute) keep the previous behavior via the fallback path; they can be optimized in follow-ups without changing call sites.

Detail

For //a//a//a style queries, the previous code built nodesets keyed by each anchor, including the same descendant once per anchor. The new
:uniq path scans every node at most once per step.

For [position() > N] style predicates on wide trees (e.g. //a/preceding-sibling::*[position()>2]), we previously built the full
preceding-sibling nodeset for each anchor and then ran evaluate_predicate. The new [op, value] path scans children once per parent and uses
anchor-index bookkeeping to recover per-anchor positions.

Note: general XPath cannot be linear — e.g. *[position() * number(@a) % number(@b) = 1] is genuinely O(n²) — so the goal is only to add a fast-path for specific case: position-independent predicates and simple-positional predicates.

Remove XPathNode

It turned out that we don't need XPathNode. We're just unnecessarily packing/re-packing to XPathNode.

Benchmark

DEPTH = 500
xml   = '<a>' * DEPTH + '</a>' * DEPTH
doc   = REXML::Document.new(xml)
WIDTH = 1000
xml_wide = '<root>' + '<child/>' * WIDTH + '</root>'
doc_wide = REXML::Document.new(xml_wide)

REXML::XPath.match(doc, "//a//a");
# processing time: 30.756939s → 0.126807s

REXML::XPath.match(doc_wide, "//*/preceding-sibling::*[position()=10]");
# processing time: 2.446333s → 0.083954s

Copilot

Pull request overview

This PR refactors XPath step evaluation in REXML::XPathParser to defer nodeset materialization and introduce axis “scan strategies” that can fast-path common predicate shapes (position-independent and simple positional predicates), improving performance for deep and wide-tree queries.

Changes:

Reworked step to drive axis scans via [scanner_method, scanner_argument] and choose scanning strategies (:uniq, [op, value], :nodesets) based on predicate classification.
Added optimized scanners for descendant(-or-self), ancestor(-or-self), and sibling axes, plus a shared fallback selection path.
Added/updated XPath predicate and sibling-axis tests, including float literal predicates and positional comparisons.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
lib/rexml/xpath_parser.rb	Implements deferred scanning/strategy selection in `step`, predicate classification, and optimized axis scanners.
test/xpath/test_predicate.rb	Adds coverage for float-literal positional predicates and variable-as-position predicates.
test/xpath/test_base.rb	Adds test for `following-sibling::*` with `position() < N` and adjusts nested-predicate test behavior.
test/xpath/test_axis_preceding_sibling.rb	Adds tests for `preceding-sibling::` with `<` and `<=` position predicates.

Comments suppressed due to low confidence (1)

lib/rexml/xpath_parser.rb:834

descendant axis fast path calls raw_node.children when include_self is false without checking the node type or whether children exists. If the context node is not an element/document (e.g., an attribute node), this will raise NoMethodError; previously it would just yield an empty descendant set. Guard this branch so it only iterates children for element/document nodes (or use the same node_type check as in recursive).

        raw_nodes.each do |raw_node|
          if include_self
            recursive.call(raw_node)
          else
            raw_node.children.each(&recursive)
          end

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

+            result = evaluate_predicate(predicate_expr.dclone, [result]).first || []
+          end


+    end
+
+    def preceding_following_sibling(raw_nodes, tester, selector, reverse:)
+      raw_nodes = raw_nodes.select(&:parent)


Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

tompng · 2026-05-26T16:12:37Z

+        raw_nodes.group_by(&:parent).flat_map do |parent, sibling_nodes|
+          anchors = {}.compare_by_identity
+          sibling_nodes.each {|sibling| anchors[sibling] = true }
+          children = parent.children
+          children = children.reverse if reverse
+          followings = children.drop_while {|child| !anchors.key?(child) }.drop(1)
+          anchor_indexes = { 0 => true }
+          last_anchor = 0
+          index = 0
+          matched = []
+          followings.each do |node|
+            if tester.call(node)
+              case operator
+              when :==
+                matched << node if anchor_indexes.include?(index - value + 1)
+              when :<
+                matched << node if last_anchor > index - value + 1
+              when :>
+                matched << node if index >= value
+              end
+              index += 1
+            end
+            if anchors.key?(node)
+              anchor_indexes[index] = true
+              last_anchor = index
+            end
+          end
+          matched


No. In > case, index from the first anchor(always zero) is the possible maximum index among relative index from several anchor nodes. I modified the code a bit, and added source code comment that describe this.

tompng · 2026-05-26T16:22:07Z

+        seen = {}.compare_by_identity
+        nodesets.each do |nodeset|
+          nodeset.each do |node|
+            raw_node = node.respond_to?(:raw_node) ? node.raw_node : node
+            seen[raw_node] = true
+          end
+        end
+        ordered = sort(seen.keys)
+        ordered.map.with_index(1) do |raw_node, position|
+          XPathNode.new(raw_node, position: position)
        end


REXML previously skipped nodesets ordering when nodesets.size == 1, but it turned out that it was a bug. nodesets.first is sometimes sorted in reverse-document-order while the order should be document-order.

We can reduce sort in the future such as:

Add an ordered/reverse-ordered/unordered flag to nodeset

Sort only on the final result and before applying filter predicates

But this kind of separate optimization should be out of scope of this PR.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

+        if pre_predicates.any?
+          original_tester = tester
+          tester = -> (node) do
+            original_tester.call(node) &&
+            pre_predicates.all? do |predicate_expr|
+              evaluate_predicate(predicate_expr.dclone, [[node]]).flatten.size == 1
            end
          end
-          ordered_nodeset = sort(raw_nodes, order)
        end


tompng · 2026-06-04T11:17:37Z

+      when :nodesets
+        nodesets = nodeset.map do |node|
+          parent = node.parent
+          index = parent.children.index(node)
+          reverse ? parent.children[0...index].reverse : parent.children[index + 1..-1]
+        end
+        non_optimized_nodesets_select(nodesets, tester, selector)


No. This is a non-optimizable slow path that needs O(n^2) calculation cost just like before.

+            # Not working at all, so just return an empty nodesets for now.
+            # Needs Namespace-node class
+            [:iterate_nodesets, []]


Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

+        sort(seen.keys)
      ensure
-        leave(:step, path_stack, new_nodeset) if @debug
+        leave(:step, path_stack, ordered) if @debug
      end


+        operator, value = selector
+        nodeset.group_by(&:parent).flat_map do |parent, sibling_nodes|
+          anchors = {}.compare_by_identity
+          sibling_nodes.each {|sibling| anchors[sibling] = true }
+          children = parent.children
+          children = children.reverse if reverse
+          followings = children.drop_while {|child| !anchors.key?(child) }.drop(1)
+          anchor_indexes = { 0 => true }
+          last_anchor = 0
+          index = 0
+          matched = []
+          followings.each do |node|
+            if tester.call(node)
+              case operator
+              when :==
+                matched << node if anchor_indexes.include?(index - value + 1)


+      when :nodesets
+        nodesets = nodeset.map do |node|
+          parent = node.parent
+          index = parent.children.index(node)
+          reverse ? parent.children[0...index].reverse : parent.children[index + 1..-1]
+        end
+        non_optimized_nodesets_select(nodesets, tester, selector)


          nodeset = step(path_stack, any_type: :namespace) do
            nodesets = []
            nodeset.each do |node|
-              raw_node = node.raw_node
-              case raw_node.node_type
+              case node.node_type
              when :element
                if @namespaces
                  nodesets << pre_defined_namespaces.merge(@namespaces)
                else
-                  nodesets << pre_defined_namespaces.merge(raw_node.namespaces)
+                  nodesets << pre_defined_namespaces.merge(node.namespaces)
                end
              when :attribute
                if @namespaces
                  nodesets << pre_defined_namespaces.merge(@namespaces)
                else
-                  nodesets << pre_defined_namespaces.merge(raw_node.element.namespaces)
+                  nodesets << pre_defined_namespaces.merge(node.element.namespaces)
                end
              end
            end
-            nodesets
+            # Not working at all, so just return an empty nodesets for now.
+            # Needs Namespace-node class
+            [:iterate_nodesets, []]
          end


If a predicate of xpath is position-independent, we don't need to create nodesets that has many duplicated nodes with different positions. Implements optimization of preceding-sibling/following-sibling with simple positional predicates as an example. We can add more optimizations for other axis in the future.

Copilot AI review requested due to automatic review settings May 21, 2026 13:11

Copilot started reviewing on behalf of tompng May 21, 2026 13:12 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Comment thread lib/rexml/xpath_parser.rb Outdated

Comment thread lib/rexml/xpath_parser.rb Outdated

Comment thread lib/rexml/xpath_parser.rb Outdated

tompng force-pushed the xpath_step_optimize branch from 6f94f2e to c4a1440 Compare May 21, 2026 15:57

Copilot AI review requested due to automatic review settings May 21, 2026 16:00

tompng force-pushed the xpath_step_optimize branch from c4a1440 to 67e3270 Compare May 21, 2026 16:00

Copilot started reviewing on behalf of tompng May 21, 2026 16:01 View session

tompng force-pushed the xpath_step_optimize branch from 67e3270 to 593620b Compare May 21, 2026 16:06

Copilot AI reviewed May 21, 2026

View reviewed changes

tompng force-pushed the xpath_step_optimize branch from 593620b to 65c5103 Compare May 21, 2026 16:31

Copilot AI review requested due to automatic review settings May 21, 2026 17:11

tompng force-pushed the xpath_step_optimize branch from 65c5103 to 15a8f52 Compare May 21, 2026 17:11

Copilot started reviewing on behalf of tompng May 21, 2026 17:11 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Comment thread lib/rexml/xpath_parser.rb Outdated

Comment thread lib/rexml/parsers/xpathparser.rb Outdated

Comment thread lib/rexml/xpath_parser.rb Outdated

tompng marked this pull request as ready for review May 21, 2026 17:26

tompng force-pushed the xpath_step_optimize branch from 15a8f52 to 020085a Compare May 21, 2026 18:13

Copilot AI review requested due to automatic review settings May 26, 2026 13:20

tompng force-pushed the xpath_step_optimize branch from 020085a to 926e521 Compare May 26, 2026 13:20

Copilot AI reviewed May 26, 2026

View reviewed changes

tompng force-pushed the xpath_step_optimize branch from 926e521 to 73dd670 Compare May 26, 2026 16:07

tompng force-pushed the xpath_step_optimize branch from 73dd670 to a6476a2 Compare June 3, 2026 17:33

pull320

1317467

Copilot AI review requested due to automatic review settings June 4, 2026 10:44

tompng force-pushed the xpath_step_optimize branch from a6476a2 to c2d8012 Compare June 4, 2026 10:44

Copilot AI reviewed Jun 4, 2026

View reviewed changes

tompng force-pushed the xpath_step_optimize branch from c2d8012 to 42c4307 Compare June 4, 2026 10:51

Copilot AI review requested due to automatic review settings June 4, 2026 10:58

tompng force-pushed the xpath_step_optimize branch from 42c4307 to 2f6ea03 Compare June 4, 2026 10:58

Copilot AI reviewed Jun 4, 2026

View reviewed changes

tompng force-pushed the xpath_step_optimize branch from 2f6ea03 to 739cba0 Compare June 4, 2026 11:06

		result = evaluate_predicate(predicate_expr.dclone, [result]).first \|\| []
		end

Conversation

tompng commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Detail

Remove XPathNode

Benchmark

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

tompng May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tompng May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

tompng Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tompng commented May 21, 2026 •

edited

Loading