Repository navigation
NodeNotFoundError for attached elements while page resources are still loading #650
Description
Summary
With Ferrum 0.18.0, I can find an attached element while a page is still loading,
but evaluating an expression on that node raises Ferrum::NodeNotFoundError.
Finding the element again does not help. Explicitly requesting the current
document with page.document_node_id restores access immediately, without waiting
for the outstanding resource to finish.
This surfaced as intermittent Cuprite/Capybara failures in CI. Capybara can catch
the driver exception during visibility/text filtering and report that no matching
element exists, sometimes showing the expected text or <<ERROR>> in the failure
message. The standalone reproduction below does not need Cuprite or Capybara.
See below on how to reproduce.
Environment
- Ferrum 0.18.0
- Ruby 3.4.8 locally
- Local Chrome at the time this draft was prepared: 155.0.8059.80, macOS
- Affected integration suite: Cuprite 0.18, Capybara 3.40.0
- CI: Linux; the exact Chrome version from the failing run was not captured
The local Chrome version above is not a claim about which earlier versions are
affected. I have not established the first affected Ferrum or Chrome release.
Investigation
Captured integration-test backtraces show failures in Ferrum::Node#node_id,
called from Ferrum::Frame::Runtime#call, while Cuprite checks visible? or reads
visible_text. In Ferrum 0.18.0, node_id raises when DOM.requestNode returns
zero. Direct JavaScript snapshots still show the expected attached DOM elements.
Ferrum requests the document in the main frame's Page.frameStoppedLoading
handler. Could node tracking need initialising earlier, when an element is first
used before that event? The reproduction is consistent with that explanation,
but I have not confirmed the appropriate upstream fix.
Temporary workaround
In our test support only, we prepend:
module FerrumDocumentTracking
def node_id
super
rescue Ferrum::NodeNotFoundError
page.document_node_id
super
end
end
Ferrum::Node.prepend(FerrumDocumentTracking)This retries once after refreshing document tracking. We are offering it as a
demonstrated workaround, not proposing that it is necessarily the right library
implementation. Refreshing document tracking may have implications for other
existing node references that deserve review.
Our browser regression passes with this workaround and reproduces the misleading
Capybara failure without it. It also verifies that incorrect text does not match
and Cuprite still rejects detached elements.
To reproduce:
require 'bundler/setup'
require 'ferrum'
require 'socket'
require 'timeout'
server = TCPServer.new('127.0.0.1', 0)
port = server.addr[1]
release_resource = Queue.new
workers = []
thread = Thread.new do
loop do
socket = server.accept
workers << Thread.new(socket) do |s|
request = s.gets
while (line = s.gets) && line != "\r\n"; end
if request.include?('/slow')
release_resource.pop
body = ''
else
body = '<html><body><h1>Expected heading</h1><img src="/slow"></body></html>'
end
s.write "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: #{body.bytesize}\r\nConnection: close\r\n\r\n#{body}"
rescue IOError, Errno::EPIPE
ensure
s.close
end
end
end
begin
options = { timeout: 5 }
options[:browser_path] = ENV['BROWSER_PATH'] if ENV['BROWSER_PATH']
browser = Ferrum::Browser.new(**options)
browser.execute("location.href = 'http://127.0.0.1:#{port}/'")
Timeout.timeout(10) do
sleep 0.01 until browser.evaluate('document.readyState') == 'interactive'
end
puts "State: #{browser.evaluate('document.readyState')}"
2.times do
node = browser.at_css('h1')
begin
puts "Fresh query text: #{node.evaluate('this.innerText')}"
rescue => e
puts "Fresh query failure: #{e.class}: #{e.message}"
end
end
browser.page.document_node_id
puts "After requesting document: #{browser.at_css('h1').evaluate('this.innerText')}"
puts "State after recovery: #{browser.evaluate('document.readyState')}"
ensure
release_resource << true
browser&.quit
thread.kill.join
workers.each { |worker| worker.kill.join }
server.close
end
Activity
route commentedon Oct 16, 2026
Thanks for the detailed report and the repro, it reproduces on the latest version too.
The repro gets into this state by navigating with
execute, which Ferrum doesn't wait for, while an image never finishes loading. In a normal Cuprite flowvisitwaits for the page'sloadevent, andclickwaits for a navigation it triggers, so I'd like to understand how your suite ends up interacting with a page that is still loading. Could you share:visit, a click, a form submit, or a navigation started by JavaScript (e.g.location.href = ...after afetch, a delayed submit, Turbo doing a full reload)?page.driver.browser.network.pending_connectionsandpage.evaluate_script("document.readyState")when the test fails.Capybara.default_max_wait_time, and whether you setFERRUM_CLICK_WAITor a customtimeout.route commentedon Oct 18, 2026
I dug deeper into this. Chrome hands out node ids for a document only after the client requests that document with
DOM.getDocument, and Ferrum does that when the main frame stops loading. So between a navigation committing and its last resource finishing,DOM.requestNodereturns0and you get "node is not trackable".visitand aclickthat navigates wait through that window, so you only end up there after a navigation nothing waited for:execute, a JS redirect, a delayed submit, Turbo. Inside Capybara'ssynchronize(find,have_*matchers) that error is retried until the page finishes loading, so the fix on your side is to wait for the navigation, e.g. withclick(wait:),network.wait_for_idle, or a Capybara matcher for the new page, before touching nodes.Calling
page.document_node_idyourself as a workaround isn't safe either:DOM.getDocumentresets every node id on the page without telling Ferrum, so nodes found before it broke. #656 fixes that, and makes the error say the page is still loading a new document. It doesn't retry for you, because that would only hide the race.If you still see this after waiting for the navigation, please share the answers to the questions above.
fix(page): drop cached node ids after DOM.getDocument (#656)