ref(browser): read browser details over CDP and ask /json/version only for url:

CHANGELOG.md

@@ -3,10 +3,19 @@
 ### Added
 
 ### Changed
+- Browser details (user agent, versions) come from `Browser.getVersion` over CDP for every browser, local or remote.
+  `/json/version` is only asked for the websocket url when `url:` is given, so `ws_url:` works against services
+  without an HTTP endpoint and a local launch makes no HTTP call
+- `NoWebSocketUrlError` names the HTTP status `/json/version` answered with, e.g. `401 Unauthorized` for a missing token or
+  `429 Too Many Requests` for a full remote browser service, and exposes it as `#status`
 
 ### Fixed
+- Headers can be added on a remote browser whose `/json/version` has no `User-Agent`, such as browserless v2, instead of
+  failing with `Invalid parameters` [rubycdp/cuprite#300]
 
 ### Removed
+- `Browser::Process#default_user_agent`, `#browser_version`, `#protocol_version`, `#v8_version` and `#webkit_version`,
+  use `Browser#default_user_agent` and `Browser#version` instead
 
 
 ## [0.19.1](https://github.com/rubycdp/ferrum/compare/v0.19.0...v0.19.1) (October 16, 2026) ##

docker/DESIGN.md

@@ -17,7 +17,9 @@ Most remote-Chrome bugs Ferrum and Cuprite users report come from browserless be
 - v2 kills sessions after 30s by default (ferrum#501).
 - v2 reports `0.0.0.0` as the WebSocket host (ferrum#501) and leaves `User-Agent` out of `/json/version` (cuprite#300).
 - `?token=` is lost on the way (ferrum#216, #439), and v2 needs a `/chrome` or `/chromium` path (ferrum#501).
-- The startup window's context can't be addressed or disposed (ferrum#578, #540).
+- The startup window's context can't be addressed or disposed (ferrum#578, #540). Ferrum adopts it as its default
+  context, so `reset` can't clean it and cookies and storage leak from one test to the next. Ferrum won't fix this:
+  the startup window belongs to whoever launched Chrome, and the driver launches with `--no-startup-window`.
 - `quit` doesn't end the session (ferrum#418), and orphaned Chrome trees pile up (ferrum#505).
 - Zombie processes pile up when the main process is PID 1 (ferrum#364), and there's no native arm64 image (ferrum#270).
 
@@ -88,8 +90,8 @@ The driver serves everything through `net/http`, and WebSocket upgrades go throu
 | Route | Behavior |
 |---|---|
 | `GET /json/version` | Starts the session's Chrome, proxies Chrome's real `/json/version`, and rewrites `webSocketDebuggerUrl` to `ws(s)://<Host>/devtools/browser/<id>?<original query>` |
-| WS `/devtools/browser/<id>` | Attaches to the Chrome started by `/json/version`. If another connection took it, falls back like `/` |
-| WS `/` | Claims a pending Chrome started with the same query, or starts a new one. For `ws_url:` clients |
+| WS `/devtools/browser/<id>` | Attaches to the Chrome started by `/json/version`, or starts a new one if it's no longer pending |
+| WS `/` | Starts a new Chrome. For `ws_url:` clients |
 | `GET /json`, `/json/list` | Targets of every running session, with URLs pointing at the driver |
 | `PUT /json/new?<url>` | Opens a target, only while exactly one session runs (`409` otherwise) |
 | `GET /json/protocol` | Chrome's protocol description, cached. Starts a short-lived Chrome only when none runs |
@@ -105,12 +107,10 @@ Chrome started at boot or just for discovery. browserless v2 instead starts a th
 metadata, and always returns its generic address (`EXTERNAL`, or `HOST:PORT`) with no path and no query.
 Details:
 
-- **Taken ids.** With `ws_url:`, Ferrum calls `/json/version` and then connects to `/`, which claims any pending
-  Chrome with the same query. With parallel clients that can be the Chrome another client's `/json/version`
-  started, so `/devtools/browser/<id>` falls back the same way instead of answering `404`. Chromes started with
-  the same query are interchangeable.
-- **Unclaimed sessions.** A Chrome started by `/json/version` is pending until a WebSocket claims it. If none
-  arrives within 30 seconds, for example from a stray `curl` or a misconfigured healthcheck, it's killed.
+- **Unclaimed sessions.** A Chrome started by `/json/version` is pending until a WebSocket claims it by its id. If
+  none arrives within 30 seconds, for example from a stray `curl` or a misconfigured healthcheck, it's killed, and
+  a late connection to its id gets a new Chrome. Only `url:` clients go through `/json/version`; `ws_url:` connects
+  to `/` directly.
 - **Reachable URL.** The URL is built from `Host` / `X-Forwarded-Host` / `X-Forwarded-Proto`, so it's always
   reachable from the client (no `0.0.0.0`), and the query string is kept. `EXTERNAL` (or `PROXY_URL`)
   overrides the scheme and host when the driver sits behind a proxy that rewrites them.
@@ -229,7 +229,7 @@ Done:
 - Config: `BROWSER_PATH`, `HOST`, `PORT`, `CONCURRENT`, `QUEUED`, `TIMEOUT`, `EXTERNAL` / `PROXY_URL`. Invalid
   values stop the driver at startup.
 - One Chrome per connection, on `--remote-debugging-port=0`, with the endpoint read from stderr.
-- `/json/version` is answered on the fly, with a claim timeout and a fallback for taken ids.
+- `/json/version` is answered on the fly, with a claim timeout.
 - The WebSocket goes through `ReverseProxy`.
 - The process group gets TERM, then KILL, and the profile is removed when the connection ends. Each session logs
   why it ended: `connection closed`, `browser exited`, `timeout`, `never claimed` or `shutdown`.

docker/internal/server/server.go

@@ -93,7 +93,7 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) {
 	}
 
 	info["webSocketDebuggerUrl"] = s.publicURL(r, b.WSPath, r.URL.RawQuery)
-	s.sessions.Reserve(b, r.URL.RawQuery)
+	s.sessions.Reserve(b)
 
 	writeJSON(w, info)
 }
@@ -113,7 +113,7 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	b, err := s.sessions.ClaimOrStart(r.Context(), r.PathValue("id"), r.URL.RawQuery, options)
+	b, err := s.sessions.ClaimOrStart(r.Context(), r.PathValue("id"), options)
 	if err != nil {
 		s.startError(w, err)
 		return

docker/internal/server/server_test.go

@@ -116,33 +116,25 @@ func TestVersionAndClaim(t *testing.T) {
 	eventually(t, func() bool { return len(srv.sessions.Browsers()) == 0 })
 }
 
-func TestConnectClaimsTheBrowserStartedByVersion(t *testing.T) {
+func TestConnectLeavesPendingBrowsersAlone(t *testing.T) {
 	srv := newServer(t, SessionsConfig{}, Config{})
 
-	srv.version(t, "?launch=%7B%7D")
-	conn, _ := dial(t, srv.addr(), "/?launch=%7B%7D")
-	conn.roundTrip(t, "hello")
+	info := srv.version(t, "")
+	anonymous, _ := dial(t, srv.addr(), "/")
+	anonymous.roundTrip(t, "anonymous")
 
-	if n := len(srv.sessions.Browsers()); n != 1 {
-		t.Fatalf("browsers = %d, want 1", n)
+	if n := len(srv.sessions.Browsers()); n != 2 {
+		t.Fatalf("browsers = %d, want 2", n)
 	}
-}
 
-func TestClaimFallsBackWhenTheIdWasTaken(t *testing.T) {
-	srv := newServer(t, SessionsConfig{}, Config{})
-
-	first := srv.version(t, "")
-	second := srv.version(t, "")
-
-	anonymous, _ := dial(t, srv.addr(), "/")
-	anonymous.roundTrip(t, "anonymous")
+	conn, resp := dial(t, srv.addr(), wsPath(t, info["webSocketDebuggerUrl"]))
+	if resp.StatusCode != http.StatusSwitchingProtocols {
+		t.Fatalf("status = %d", resp.StatusCode)
+	}
+	conn.roundTrip(t, "hello")
 
-	for _, info := range []map[string]any{first, second} {
-		conn, resp := dial(t, srv.addr(), wsPath(t, info["webSocketDebuggerUrl"]))
-		if resp.StatusCode != http.StatusSwitchingProtocols {
-			t.Fatalf("status = %d", resp.StatusCode)
-		}
-		conn.roundTrip(t, "hello")
+	if n := len(srv.sessions.Browsers()); n != 2 {
+		t.Fatalf("browsers = %d, want 2", n)
 	}
 }
 

docker/internal/server/sessions.go

@@ -38,12 +38,7 @@ type Sessions struct {
 	mu       sync.Mutex
 	waiting  int
 	browsers map[string]*browser.Browser
-	pending  map[string]reservation
-}
-
-type reservation struct {
-	browser *browser.Browser
-	query   string
+	pending  map[string]*browser.Browser
 }
 
 func NewSessions(config SessionsConfig) *Sessions {
@@ -58,7 +53,7 @@ func NewSessions(config SessionsConfig) *Sessions {
 		slots:        make(chan struct{}, max(config.Concurrent, 1)),
 		closed:       make(chan struct{}),
 		browsers:     make(map[string]*browser.Browser),
-		pending:      make(map[string]reservation),
+		pending:      make(map[string]*browser.Browser),
 	}
 }
 
@@ -90,9 +85,9 @@ func (s *Sessions) Start(ctx context.Context, options browser.Options) (*browser
 	return b, nil
 }
 
-func (s *Sessions) Reserve(b *browser.Browser, query string) {
+func (s *Sessions) Reserve(b *browser.Browser) {
 	s.mu.Lock()
-	s.pending[b.ID] = reservation{browser: b, query: query}
+	s.pending[b.ID] = b
 	s.mu.Unlock()
 
 	time.AfterFunc(s.claimTimeout, func() {
@@ -106,27 +101,17 @@ func (s *Sessions) Claim(id string) *browser.Browser {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 
-	r := s.pending[id]
+	b := s.pending[id]
 	delete(s.pending, id)
 
-	return r.browser
+	return b
 }
 
-func (s *Sessions) ClaimOrStart(ctx context.Context, id, query string, options browser.Options) (*browser.Browser, error) {
+func (s *Sessions) ClaimOrStart(ctx context.Context, id string, options browser.Options) (*browser.Browser, error) {
 	if b := s.Claim(id); b != nil {
 		return b, nil
 	}
 
-	s.mu.Lock()
-	for id, r := range s.pending {
-		if r.query == query {
-			delete(s.pending, id)
-			s.mu.Unlock()
-			return r.browser, nil
-		}
-	}
-	s.mu.Unlock()
-
 	return s.Start(ctx, options)
 }
 

docker/internal/server/sessions_test.go

@@ -66,7 +66,7 @@ func TestSessionsClaim(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	s.Reserve(b, "q=1")
+	s.Reserve(b)
 
 	if got := s.Claim(b.ID); got != b {
 		t.Fatalf("claim = %v, want %v", got, b)
@@ -84,41 +84,35 @@ func TestSessionsClaimOrStart(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	s.Reserve(pending, "q=1")
+	s.Reserve(pending)
 
-	t.Run("claims a pending browser with the same query", func(t *testing.T) {
-		b, err := s.ClaimOrStart(ctx, "", "q=1", browser.Options{})
+	t.Run("starts a browser without an id", func(t *testing.T) {
+		b, err := s.ClaimOrStart(ctx, "", browser.Options{})
 		if err != nil {
 			t.Fatal(err)
 		}
-		if b != pending {
-			t.Fatal("started a new browser instead of claiming the pending one")
+		if b == pending {
+			t.Fatal("claimed a pending browser without its id")
 		}
 	})
 
-	t.Run("falls back when the id was already claimed", func(t *testing.T) {
-		other, err := s.Start(ctx, browser.Options{})
+	t.Run("claims the pending browser by id", func(t *testing.T) {
+		b, err := s.ClaimOrStart(ctx, pending.ID, browser.Options{})
 		if err != nil {
 			t.Fatal(err)
 		}
-		s.Reserve(other, "q=1")
-
-		b, err := s.ClaimOrStart(ctx, pending.ID, "q=1", browser.Options{})
-		if err != nil {
-			t.Fatal(err)
-		}
-		if b != other {
-			t.Fatal("expected the other pending browser")
+		if b != pending {
+			t.Fatal("started a new browser instead of claiming the pending one")
 		}
 	})
 
-	t.Run("starts a browser for another query", func(t *testing.T) {
-		b, err := s.ClaimOrStart(ctx, "", "q=2", browser.Options{})
+	t.Run("starts a browser when the id is no longer pending", func(t *testing.T) {
+		b, err := s.ClaimOrStart(ctx, pending.ID, browser.Options{})
 		if err != nil {
 			t.Fatal(err)
 		}
 		if b == pending {
-			t.Fatal("claimed a browser with other options")
+			t.Fatal("claimed the same browser twice")
 		}
 	})
 }
@@ -130,7 +124,7 @@ func TestSessionsUnclaimedBrowserIsStopped(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	s.Reserve(b, "")
+	s.Reserve(b)
 
 	eventually(t, func() bool { return len(s.Browsers()) == 0 })
 	<-b.Done()

lib/ferrum/browser.rb

@@ -309,9 +309,9 @@ def start
 
       begin
         @process.start
-        @options.default_user_agent = @process.default_user_agent
 
         @client = Client.new(@process.ws_url, options)
+        @options.default_user_agent = version.user_agent
         @contexts = Contexts.new(@client)
       rescue StandardError
         @process.stop

lib/ferrum/browser/process.rb

@@ -42,23 +42,14 @@ def self.start(*args)
         new(*args).tap(&:start)
       end
 
-      attr_reader :host, :port, :ws_url, :pid, :command,
-                  :default_user_agent, :browser_version, :protocol_version,
-                  :v8_version, :webkit_version, :xvfb
+      attr_reader :host, :port, :ws_url, :pid, :command, :xvfb
 
       def initialize(options)
         @pid = @xvfb = @user_data_dir = nil
         @requested_host = options.host
 
         if options.ws_url || options.url
-          # `:ws_url` option is higher priority than `:url`, parse versions
-          # and use it as a ws_url, otherwise use what has been parsed.
-          endpoint = options.ws_url || options.url
-          response = parse_json_version(endpoint)
-          ws_url = options.ws_url || reachable_ws_url(response&.[]("webSocketDebuggerUrl"), endpoint)
-          raise NoWebSocketUrlError, endpoint unless ws_url
-
-          self.ws_url = ws_url
+          self.ws_url = options.ws_url || fetch_ws_url(options.url)
           return
         end
 
@@ -92,7 +83,6 @@ def start
           ObjectSpace.define_finalizer(self, Killer.finalizer([@pid, @xvfb&.pid], @user_data_dir))
 
           parse_ws_url(read_io, @process_timeout)
-          parse_json_version(ws_url)
         ensure
           close_io(read_io, write_io)
         end
@@ -143,11 +133,7 @@ def inspect
         "#<#{self.class} " \
           "@user_data_dir=#{@user_data_dir.inspect} " \
           "@command=#<#{@command.class}:#{@command.object_id}> " \
-          "@default_user_agent=#{@default_user_agent.inspect} " \
-          "@ws_url=#{@ws_url.inspect} " \
-          "@v8_version=#{@v8_version.inspect} " \
-          "@browser_version=#{@browser_version.inspect} " \
-          "@webkit_version=#{@webkit_version.inspect}>"
+          "@ws_url=#{@ws_url.inspect}>"
       end
 
       private
@@ -233,18 +219,16 @@ def close_io(*ios)
         end
       end
 
-      def parse_json_version(url)
-        response = JSON.parse(::Net::HTTP.get(URI(json_version_url(url).to_s)))
-
-        @v8_version = response["V8-Version"]
-        @browser_version = response["Browser"]
-        @webkit_version = response["WebKit-Version"]
-        @default_user_agent = response["User-Agent"]
-        @protocol_version = response["Protocol-Version"]
+      def fetch_ws_url(url)
+        response = ::Net::HTTP.get_response(URI(json_version_url(url).to_s))
+        status = "#{response.code} #{response.message}" unless response.is_a?(::Net::HTTPSuccess)
+        ws_url = begin
+          JSON.parse(response.body.to_s)["webSocketDebuggerUrl"]
+        rescue JSON::ParserError
+          nil
+        end
 
-        response
-      rescue JSON::ParserError
-        # nop
+        reachable_ws_url(ws_url, url) || raise(NoWebSocketUrlError.new(url, status))
       end
 
       def json_version_url(url)

lib/ferrum/errors.rb

@@ -112,11 +112,13 @@ def initialize(timeout, output)
   # Raised when connecting to an already running browser, but its
   # `/json/version` endpoint doesn't answer with the websocket url.
   class NoWebSocketUrlError < Error
-    attr_reader :url
+    attr_reader :url, :status
 
-    def initialize(url)
+    def initialize(url, status = nil)
       @url = url
-      super("Couldn't get a websocket url out of #{url}, `/json/version` there didn't answer with JSON. " \
+      @status = status
+      reason = status ? "answered #{status}" : "didn't answer with JSON"
+      super("Couldn't get a websocket url out of #{url}, `/json/version` there #{reason}. " \
             "Check the browser is reachable at that url and that any credentials it needs are part of it")
     end
   end