From 4c7478e103b08dba67d2eafff26ec43e279a2c9a Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Mon, 12 Oct 2026 21:12:03 +0300 Subject: [PATCH 01/23] feat(docker): add a driver that serves one Chrome per CDP connection A small Go server that looks like browserless v2 from the outside but runs no Puppeteer: each WebSocket connection gets its own Chrome, traffic is proxied unchanged, and Chrome's process group is killed when the connection ends. /json/version starts the session's Chrome and returns its real metadata with a reachable ws URL; unclaimed browsers are stopped after 30s. DESIGN.md lists the browserless pitfalls it avoids. --- docker/.gitignore | 1 + docker/DESIGN.md | 191 +++++++++++++++++++++++++++++ docker/go.mod | 3 + docker/go.sum | 0 docker/internal/browser/browser.go | 150 ++++++++++++++++++++++ docker/internal/browser/flags.go | 63 ++++++++++ docker/internal/config/config.go | 21 ++++ docker/internal/server/server.go | 158 ++++++++++++++++++++++++ docker/internal/server/sessions.go | 114 +++++++++++++++++ docker/main.go | 51 ++++++++ 10 files changed, 752 insertions(+) create mode 100644 docker/.gitignore create mode 100644 docker/DESIGN.md create mode 100644 docker/go.mod create mode 100644 docker/go.sum create mode 100644 docker/internal/browser/browser.go create mode 100644 docker/internal/browser/flags.go create mode 100644 docker/internal/config/config.go create mode 100644 docker/internal/server/server.go create mode 100644 docker/internal/server/sessions.go create mode 100644 docker/main.go diff --git a/docker/.gitignore b/docker/.gitignore new file mode 100644 index 00000000..e660fd93 --- /dev/null +++ b/docker/.gitignore @@ -0,0 +1 @@ +bin/ diff --git a/docker/DESIGN.md b/docker/DESIGN.md new file mode 100644 index 00000000..21741c4d --- /dev/null +++ b/docker/DESIGN.md @@ -0,0 +1,191 @@ +# Design + +A small Go binary that runs as the entrypoint of a Docker container and gives each client its own Chrome over CDP. +It looks like browserless v2 (checked against v2.57.0) from the outside, but works differently inside. There's no Puppeteer, and nothing +touches the client's targets: bytes go through unchanged. It's self-hosted only (local machines, CI, your own +servers) and Docker only. + +## Why + +Most remote-Chrome bugs Ferrum and Cuprite users report come from browserless being too clever, not from CDP: + +- Its own Puppeteer resets the viewport to 800x600 (ferrum#592, #492) and sets its own download folder (cuprite#279). +- It writes a raw `HTTP 408` onto a socket that is already a WebSocket when a limit is hit (ferrum#538). +- v2 kills sessions after 30s by default (ferrum#501). +- v2 reports `0.0.0.0` as the WebSocket host (ferrum#501) and leaves `User-Agent` out of `/json/version` (cuprite#300). +- `?token=` is lost on the way (ferrum#216, #439), and v2 needs a `/chrome` or `/chromium` path (ferrum#501). +- The startup window's context can't be addressed or disposed (ferrum#578, #540). +- `quit` doesn't end the session (ferrum#418), and orphaned Chrome trees pile up (ferrum#505). +- Zombie processes pile up when the main process is PID 1 (ferrum#364), and there's no native arm64 image (ferrum#270). + +## Process model + +- The container runs dumb-init as PID 1, then `driver`, then one Chrome per session. dumb-init reaps orphans, + including crashpad's double-forked handler. That's why the driver isn't PID 1 itself: reaping everything there would race + `exec.Cmd.Wait`. +- Each session gets its own Chrome with a fresh profile directory and `--remote-debugging-port=0`. The driver reads + `DevTools listening on ws://…` from Chrome's stderr, so there are no port collisions and no startup polling. +- A session ends when the client's socket closes, which also covers `Browser.close`: Chrome exits, its socket + closes, and the proxy sees EOF. + - Both proxy directions are then torn down. + - The whole process group gets TERM, then KILL after a grace period, and the profile directory is deleted. + - Go's default TCP keepalive on accepted connections catches a client that vanished without closing. +- `CONCURRENT` limits running sessions and defaults to the number of CPUs. Up to `QUEUED` extra connections wait + for a free slot. Beyond that they get `429` with `Retry-After`, as a plain HTTP response sent before the + WebSocket upgrade, never bytes on a live socket. The names match browserless v2, so its compose files work + unchanged. +- There's no session timeout by default; browserless v2 defaults to 30s. `TIMEOUT` (env) or `?timeout=` (query), + both in milliseconds, set one. The driver logs why each session ended, so `docker logs` shows it. + +## HTTP surface + +The driver serves everything through `net/http`, and WebSocket upgrades go through `httputil.ReverseProxy`. + +| Route | Behavior | +|---|---| +| `GET /json/version` | Starts the session's Chrome, proxies Chrome's real `/json/version`, and rewrites `webSocketDebuggerUrl` to `ws(s):///devtools/browser/?` | +| WS `/devtools/browser/` | Attaches to the Chrome started by `/json/version`, `404` if there's none (as in v2) | +| WS `/`, `/chrome`, `/chromium` | Claims a pending Chrome started with the same query, or starts a new one. For `ws_url:` clients and browserless URLs | +| `GET /json/list`, `/json/new`, `/json/protocol`, `/devtools/*`, WS `/devtools/page/` | Forwarded to the Chrome that owns the target, for the DevTools viewer (v2 has the same routes) | +| `GET /health` | Driver liveness. Never starts a Chrome | + +Like v2, every route also accepts a trailing slash: `/chrome/` works like `/chrome`. + +`/json/version` is answered on the fly, so the client gets real `Browser`, `User-Agent` and protocol data, with no +Chrome started at boot or just for discovery. This differs from v2 on purpose. v2 starts a throwaway Chrome once, +caches its metadata, and always returns its generic address (`EXTERNAL`, or `HOST:PORT`) with no path and no +query, so the client connects to `/`. Clients work against either. Details: + +- **Unclaimed sessions.** A Chrome started by `/json/version` is pending until a WebSocket claims it. If none + arrives within 30 seconds, for example from a stray `curl` or a misconfigured healthcheck, it's killed. +- **Reachable URL.** The URL is built from `Host` / `X-Forwarded-Host` / `X-Forwarded-Proto`, so it's always + reachable from the client (no `0.0.0.0`), and the query string is kept. `EXTERNAL` (or `PROXY_URL`, as in v2) + overrides the scheme and host when the driver sits behind a proxy that rewrites them. +- **Host header.** Requests forwarded to Chrome carry `Host: 127.0.0.1:`, because Chrome rejects any other + host on its HTTP endpoint. +- **DevTools URLs.** In the `devtoolsFrontendUrl` returned by `/json`, the `ws=` parameter is rewritten to the + driver's host. Ferrum's `debug_url` and the browser's DevTools then give a live, clickable view of the session. + That's our answer to browserless's debug viewer. + +### Zero-copy tunnel (potential) + +After the upgrade, `ReverseProxy` copies raw bytes between the two connections in userspace and never parses +frames. A possible optimization: `Hijack()` the client connection, dial Chrome, write the rewritten handshake, +then `io.Copy` between the two `*net.TCPConn`s. On Linux that uses `splice(2)`, so the bytes never enter +userspace. Bytes already buffered by `net/http` have to be flushed first, and both connections closed when either +copy ends. It's worth doing only if profiling shows the copy matters, since Chrome's JSON serialization dominates +CDP traffic. + +## Launch options + +The query string from `/json/version` or the WebSocket URL accepts the same parameters as browserless v2: + +- `launch`: JSON or base64-encoded JSON, in v2's (Puppeteer launch) format. Supported keys: + - `args`: extra Chrome flags, e.g. `--proxy-server=…` or `--lang=…`; + - `headless`: `false` starts Chrome under Xvfb; + - `defaultViewport` `{width, height}`: becomes `--window-size`; + - `ignoreDefaultArgs`: `true`, or a list of default flags to drop. +- `timeout`: the session timeout in milliseconds. + +Other parameters (`token`, `blockAds`, `blockAdsInclude`) and other `launch` keys are ignored, so existing +browserless URLs keep working. + +Anyone who can connect can pass flags like `--renderer-cmd-prefix` and run commands in the container. There's no +authentication, so never publish the port beyond localhost or a private network. + +Default flags: + +- headless; +- `--no-startup-window`, so no implicit context; +- `--no-sandbox`, since the container is the boundary; +- `--disable-dev-shm-usage`; +- `--disable-features=site-per-process`, which makes ferrum#640 less likely. + +Never added: `--disable-web-security`, `--disable-crashpad-for-testing` (the network service crash-loops), or +`NetworkServiceInProcess`. + +## Files + +There's no file API. Mount directories at the same absolute path on the host or app container and in the driver +container. Ferrum passes its own paths to `Browser.setDownloadBehavior` and `DOM.setFileInputFiles`, and Chrome +resolves them inside the container: + +- uploads: the app source, e.g. `-v $PWD:$PWD`; +- downloads: e.g. `./tmp/downloads` mounted at the same path. + +Screenshots and PDFs need no mount: they come back over CDP and Ruby writes the file. + +The image works under `--user $(id -u):$(id -g)`: any uid, a world-writable `HOME` with the XDG directories +under it, which crashpad needs (ferrum#541, #544). Downloads then belong to the user, and Chrome can write into the mounted folder. + +## Image + +- `debian:trixie-slim`, never Alpine (Chromium on musl aborts at startup, ferrum#571). Not Ubuntu either, because + Ubuntu's `chromium` package is a snap, which doesn't work in Docker. +- Two images, `ghcr.io/rubycdp/chrome` (Google's `.deb`) and `ghcr.io/rubycdp/chromium` (Debian's package), each + for amd64 and arm64. Tags: `latest`, the browser's major version (`155`) and its full version (`155.0.8059.39`). +- The Go binary is cross-compiled in a build stage, so the image has no Node, Playwright or npm. +- `.github/workflows/docker.yml` builds each browser and architecture pair on a native runner, never through + QEMU, which crashes Chrome (ferrum#270). It smoke-tests every image and merges the architectures into one + multi-arch tag by digest. +- Fonts: Debian's font packages plus open substitutes for Microsoft fonts. Selawik replaces Segoe UI, Carlito + replaces Calibri, Caladea replaces Cambria, and DejaVu replaces Tahoma. +- Later: the Ferrum suite run against each image before it's published. +- Env, with names matching browserless v2: + - `PORT` (default 3000) and `HOST` (default `0.0.0.0`); + - `CONCURRENT`, `QUEUED` and `TIMEOUT`; + - `EXTERNAL` / `PROXY_URL`; + - plus `BROWSER_PATH`, which the image sets. +- A healthcheck on `/health`, so compose can use `condition: service_healthy`. + +## Usage with Cuprite in compose + +This follows the Evil Martians "System of a test" setup: + +- **The `chrome` service.** Use this image instead of `browserless/chrome`. `CONNECTION_TIMEOUT` and the + M1-specific tag aren't needed, and `depends_on` can use `service_healthy`. +- **Connecting.** `url: ENV["CHROME_URL"]` (`http://chrome:3333`) works as is: one `/json/version`, then the + WebSocket. The TCP availability check with fallback to a local Chrome also works as is. +- **Sessions.** Cuprite opens one connection per test process and calls `reset` between tests, so it's one Chrome + per process. Parallel workers each get their own, so set `CONCURRENT` to at least the worker count. +- **The app's address.** `Capybara.server_host = "0.0.0.0"`, with `app_host` set to the app's hostname on the + compose network. When Rails runs on the host instead, use `host.docker.internal`, which needs + `extra_hosts: ["host.docker.internal:host-gateway"]` on Linux. +- **WebMock and VCR.** Allow the `chrome` host, because `/json/version` goes through `Net::HTTP`. +- **Launch flags.** `browser_options: {"no-sandbox" => nil}` isn't needed. Ferrum doesn't send launch flags to a + remote browser anyway. +- **GitHub Actions.** Service containers start before checkout, so start the driver with `docker run` in a step + after checkout when uploads need the workspace mounted. + +## Ferrum changes + +1. Apply `window_size` to each page in remote mode. Today it only feeds the local `--window-size` flag, so a + remote browser silently ignores it. +2. `Browser#debug` in remote mode prints the DevTools URL instead of `Process.spawn(process.path, …)`, which + fails because the path is nil. +3. Never look for a local binary when `url` or `ws_url` is set (ferrum#554). +4. Optionally, send `window_size` and `browser_options` in v2's `launch` query parameter (`defaultViewport` and + `args`) when connecting remotely. + +## Status + +Done: +- Config is `BROWSER_PATH`, `HOST` and `PORT`. +- One Chrome per connection, on `--remote-debugging-port=0`, with the endpoint read from stderr. +- `/json/version` is answered on the fly, with a claim timeout. +- The WebSocket goes through `ReverseProxy`. +- The process group gets TERM, then KILL, and the profile is removed when the connection ends. +- `/health`. +- A graceful shutdown stops every Chrome. +- Trailing-slash routes (`/chrome/`, `/chromium/`, `/json/version/`), as in v2. + +With `ws_url:`, Ferrum still calls `/json/version` before connecting to `/`. Pending Chromes are therefore matched +by query string, so a connection to `/` claims the one that call started, instead of starting a second. + +Not done yet: +- `CONCURRENT`, `QUEUED` and `TIMEOUT`, plus `EXTERNAL`; +- the `launch` and `timeout` query parameters; +- the DevTools proxy (`/json/list`, `/json/new`, `/json/protocol`, `/devtools/*`); +- cleaned-up default flags; +- a `HEALTHCHECK` in the image; +- Go tests. diff --git a/docker/go.mod b/docker/go.mod new file mode 100644 index 00000000..7b0c0f1d --- /dev/null +++ b/docker/go.mod @@ -0,0 +1,3 @@ +module github.com/rubycdp/ferrum/docker + +go 1.27 diff --git a/docker/go.sum b/docker/go.sum new file mode 100644 index 00000000..e69de29b diff --git a/docker/internal/browser/browser.go b/docker/internal/browser/browser.go new file mode 100644 index 00000000..bca8e0d4 --- /dev/null +++ b/docker/internal/browser/browser.go @@ -0,0 +1,150 @@ +package browser + +import ( + "bufio" + "errors" + "fmt" + "log/slog" + "net/url" + "os" + "os/exec" + "path" + "regexp" + "strings" + "sync" + "syscall" + "time" +) + +const killTimeout = 5 * time.Second + +var listeningPattern = regexp.MustCompile(`DevTools listening on (ws://\S+)`) + +type Browser struct { + ID string + Addr string + WSPath string + + cmd *exec.Cmd + dir string + done chan struct{} + stopOnce sync.Once +} + +func Start(binary string, timeout time.Duration) (*Browser, error) { + dir, err := os.MkdirTemp("", "chrome-") + if err != nil { + return nil, fmt.Errorf("create user data dir: %w", err) + } + + stderr, writer, err := os.Pipe() + if err != nil { + os.RemoveAll(dir) + return nil, fmt.Errorf("create stderr pipe: %w", err) + } + + cmd := exec.Command(binary, Flags(dir)...) + cmd.Stderr = writer + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + + err = cmd.Start() + writer.Close() + if err != nil { + stderr.Close() + os.RemoveAll(dir) + return nil, fmt.Errorf("start browser: %w", err) + } + + b := &Browser{cmd: cmd, dir: dir, done: make(chan struct{})} + go b.wait() + + wsURL := make(chan string, 1) + go b.readStderr(stderr, wsURL) + + select { + case raw := <-wsURL: + if err := b.setEndpoint(raw); err != nil { + b.Stop() + return nil, err + } + slog.Info("browser started", "pid", b.Pid(), "id", b.ID, "addr", b.Addr) + return b, nil + case <-b.done: + b.Stop() + return nil, errors.New("browser exited before it was ready") + case <-time.After(timeout): + b.Stop() + return nil, fmt.Errorf("browser did not start within %s", timeout) + } +} + +func (b *Browser) Pid() int { + return b.cmd.Process.Pid +} + +func (b *Browser) Done() <-chan struct{} { + return b.done +} + +func (b *Browser) Stop() { + b.stopOnce.Do(func() { + pgid := b.Pid() + syscall.Kill(-pgid, syscall.SIGTERM) + + select { + case <-b.done: + case <-time.After(killTimeout): + slog.Warn("browser ignored SIGTERM, killing", "pid", pgid) + } + + syscall.Kill(-pgid, syscall.SIGKILL) + <-b.done + + if err := os.RemoveAll(b.dir); err != nil { + slog.Warn("cannot remove user data dir", "dir", b.dir, "error", err) + } + + slog.Info("browser stopped", "pid", pgid, "id", b.ID) + }) +} + +func (b *Browser) wait() { + err := b.cmd.Wait() + slog.Debug("browser exited", "pid", b.Pid(), "error", err) + close(b.done) +} + +func (b *Browser) readStderr(stderr *os.File, wsURL chan<- string) { + defer stderr.Close() + + reader := bufio.NewReader(stderr) + for { + line, err := reader.ReadString('\n') + if line = strings.TrimSpace(line); line != "" { + if match := listeningPattern.FindStringSubmatch(line); match != nil { + select { + case wsURL <- match[1]: + default: + } + } else { + slog.Debug("browser stderr", "pid", b.Pid(), "line", line) + } + } + if err != nil { + return + } + } +} + +func (b *Browser) setEndpoint(raw string) error { + endpoint, err := url.Parse(raw) + if err != nil { + return fmt.Errorf("parse browser websocket url %q: %w", raw, err) + } + + b.Addr = endpoint.Host + b.WSPath = endpoint.Path + b.ID = path.Base(endpoint.Path) + + return nil +} diff --git a/docker/internal/browser/flags.go b/docker/internal/browser/flags.go new file mode 100644 index 00000000..758115f5 --- /dev/null +++ b/docker/internal/browser/flags.go @@ -0,0 +1,63 @@ +package browser + +import ( + "fmt" +) + +const ( + chromeDebuggingPortFlag = "--remote-debugging-port" + chromeWindowSizeFlag = "--window-size" + chromeUserDataDirFlag = "--user-data-dir" + chromeResolution = "1920,1080" +) + +var ( + flags = []string{ + "--no-sandbox", + "--headless", + "--disable-gpu", + "--hide-scrollbars", + "--mute-audio", + "--enable-automation", + "--disable-web-security", + "--disable-session-crashed-bubble", + "--disable-breakpad", + "--disable-sync", + "--no-first-run", + "--use-mock-keychain", + "--keep-alive-for-test", + "--disable-popup-blocking", + "--disable-extensions", + "--disable-component-extensions-with-background-pages", + "--disable-hang-monitor", + "--disable-features=site-per-process,TranslateUI", + "--disable-translate", + "--disable-background-networking", + "--enable-features=NetworkService,NetworkServiceInProcess", + "--disable-background-timer-throttling", + "--disable-backgrounding-occluded-windows", + "--disable-client-side-phishing-detection", + "--disable-default-apps", + "--disable-dev-shm-usage", + "--disable-ipc-flooding-protection", + "--disable-prompt-on-repost", + "--disable-renderer-backgrounding", + "--force-color-profile=srgb", + "--metrics-recording-only", + "--safebrowsing-disable-auto-update", + "--password-store=basic", + "--no-startup-window", + } +) + +func Flags(dir string) []string { + f := make([]string, len(flags)) + copy(f, flags) + f = append(f, []string{ + fmt.Sprintf("%s=%s", chromeDebuggingPortFlag, "0"), + fmt.Sprintf("%s=%s", chromeWindowSizeFlag, chromeResolution), + fmt.Sprintf("%s=%s", chromeUserDataDirFlag, dir), + }...) + + return f +} diff --git a/docker/internal/config/config.go b/docker/internal/config/config.go new file mode 100644 index 00000000..2882df3c --- /dev/null +++ b/docker/internal/config/config.go @@ -0,0 +1,21 @@ +package config + +import ( + "net" + "os" +) + +var ( + Version string + + BrowserPath = os.Getenv("BROWSER_PATH") + Address = net.JoinHostPort(env("HOST", "0.0.0.0"), env("PORT", "3000")) +) + +func env(key, fallback string) string { + if value, ok := os.LookupEnv(key); ok && value != "" { + return value + } + + return fallback +} diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go new file mode 100644 index 00000000..1871c453 --- /dev/null +++ b/docker/internal/server/server.go @@ -0,0 +1,158 @@ +package server + +import ( + "encoding/json" + "fmt" + "log/slog" + "net/http" + "net/http/httputil" + "net/url" + "strings" + + "github.com/rubycdp/ferrum/docker/internal/browser" +) + +type Server struct { + sessions *Sessions +} + +func New(sessions *Sessions) http.Handler { + s := &Server{sessions: sessions} + + mux := http.NewServeMux() + handle(mux, "/health", s.health) + handle(mux, "/json/version", s.version) + handle(mux, "/devtools/browser/{id}", s.claim) + handle(mux, "/chrome", s.connect) + handle(mux, "/chromium", s.connect) + mux.HandleFunc("GET /{$}", s.connect) + + return mux +} + +func handle(mux *http.ServeMux, path string, handler http.HandlerFunc) { + mux.HandleFunc("GET "+path, handler) + mux.HandleFunc("GET "+path+"/{$}", handler) +} + +func (s *Server) health(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok")) +} + +func (s *Server) version(w http.ResponseWriter, r *http.Request) { + b, err := s.sessions.Start() + if err != nil { + slog.Error("cannot start browser", "error", err) + http.Error(w, "cannot start browser", http.StatusServiceUnavailable) + return + } + + info, err := fetchVersion(b) + if err != nil { + slog.Error("cannot fetch browser version", "id", b.ID, "error", err) + s.sessions.Stop(b) + http.Error(w, "cannot fetch browser version", http.StatusBadGateway) + return + } + + info["webSocketDebuggerUrl"] = publicURL(r, b.WSPath) + s.sessions.Reserve(b, r.URL.RawQuery) + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(info) +} + +func (s *Server) claim(w http.ResponseWriter, r *http.Request) { + if !isUpgrade(r) { + http.Error(w, "websocket upgrade required", http.StatusUpgradeRequired) + return + } + + b := s.sessions.Claim(r.PathValue("id")) + if b == nil { + http.Error(w, "browser not found", http.StatusNotFound) + return + } + + s.serve(w, r, b) +} + +func (s *Server) connect(w http.ResponseWriter, r *http.Request) { + if !isUpgrade(r) { + http.Error(w, "websocket upgrade required", http.StatusUpgradeRequired) + return + } + + b, err := s.sessions.ClaimOrStart(r.URL.RawQuery) + if err != nil { + slog.Error("cannot start browser", "error", err) + http.Error(w, "cannot start browser", http.StatusServiceUnavailable) + return + } + + s.serve(w, r, b) +} + +func (s *Server) serve(w http.ResponseWriter, r *http.Request, b *browser.Browser) { + defer s.sessions.Stop(b) + + slog.Info("session opened", "id", b.ID, "client", r.RemoteAddr) + proxyTo(b).ServeHTTP(w, r) + slog.Info("session closed", "id", b.ID, "client", r.RemoteAddr) +} + +func proxyTo(b *browser.Browser) *httputil.ReverseProxy { + return &httputil.ReverseProxy{ + Rewrite: func(pr *httputil.ProxyRequest) { + pr.Out.URL.Scheme = "http" + pr.Out.URL.Host = b.Addr + pr.Out.URL.Path = b.WSPath + pr.Out.URL.RawPath = "" + pr.Out.URL.RawQuery = "" + pr.Out.Host = b.Addr + pr.Out.Header.Del("Origin") + }, + ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) { + slog.Error("cannot proxy to browser", "id", b.ID, "error", err) + http.Error(w, "browser unavailable", http.StatusBadGateway) + }, + } +} + +func fetchVersion(b *browser.Browser) (map[string]any, error) { + resp, err := http.Get("http://" + b.Addr + "/json/version") + if err != nil { + return nil, err + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("unexpected status %s", resp.Status) + } + + var info map[string]any + if err := json.NewDecoder(resp.Body).Decode(&info); err != nil { + return nil, fmt.Errorf("decode: %w", err) + } + + return info, nil +} + +func publicURL(r *http.Request, wsPath string) string { + scheme := "ws" + if r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") { + scheme = "wss" + } + + host := r.Header.Get("X-Forwarded-Host") + if host == "" { + host = r.Host + } + + u := url.URL{Scheme: scheme, Host: host, Path: wsPath, RawQuery: r.URL.RawQuery} + return u.String() +} + +func isUpgrade(r *http.Request) bool { + return strings.EqualFold(r.Header.Get("Upgrade"), "websocket") +} diff --git a/docker/internal/server/sessions.go b/docker/internal/server/sessions.go new file mode 100644 index 00000000..2fab7da6 --- /dev/null +++ b/docker/internal/server/sessions.go @@ -0,0 +1,114 @@ +package server + +import ( + "log/slog" + "sync" + "time" + + "github.com/rubycdp/ferrum/docker/internal/browser" +) + +const ( + startTimeout = 30 * time.Second + claimTimeout = 30 * time.Second +) + +type Sessions struct { + binary string + + mu sync.Mutex + browsers map[string]*browser.Browser + pending map[string]reservation +} + +type reservation struct { + browser *browser.Browser + options string +} + +func NewSessions(binary string) *Sessions { + return &Sessions{ + binary: binary, + browsers: make(map[string]*browser.Browser), + pending: make(map[string]reservation), + } +} + +func (s *Sessions) Start() (*browser.Browser, error) { + b, err := browser.Start(s.binary, startTimeout) + if err != nil { + return nil, err + } + + s.mu.Lock() + s.browsers[b.ID] = b + s.mu.Unlock() + + go func() { + <-b.Done() + s.Stop(b) + }() + + return b, nil +} + +func (s *Sessions) Reserve(b *browser.Browser, options string) { + s.mu.Lock() + s.pending[b.ID] = reservation{browser: b, options: options} + s.mu.Unlock() + + time.AfterFunc(claimTimeout, func() { + if s.Claim(b.ID) != nil { + slog.Warn("browser was never claimed, stopping", "id", b.ID, "timeout", claimTimeout) + s.Stop(b) + } + }) +} + +func (s *Sessions) Claim(id string) *browser.Browser { + s.mu.Lock() + defer s.mu.Unlock() + + r := s.pending[id] + delete(s.pending, id) + + return r.browser +} + +func (s *Sessions) ClaimOrStart(options string) (*browser.Browser, error) { + s.mu.Lock() + for id, r := range s.pending { + if r.options == options { + delete(s.pending, id) + s.mu.Unlock() + return r.browser, nil + } + } + s.mu.Unlock() + + return s.Start() +} + +func (s *Sessions) Stop(b *browser.Browser) { + s.mu.Lock() + delete(s.browsers, b.ID) + delete(s.pending, b.ID) + s.mu.Unlock() + + b.Stop() +} + +func (s *Sessions) StopAll() { + s.mu.Lock() + browsers := make([]*browser.Browser, 0, len(s.browsers)) + for _, b := range s.browsers { + browsers = append(browsers, b) + } + s.mu.Unlock() + + var wg sync.WaitGroup + for _, b := range browsers { + wg.Go(func() { s.Stop(b) }) + } + wg.Wait() +} diff --git a/docker/main.go b/docker/main.go new file mode 100644 index 00000000..04e61f57 --- /dev/null +++ b/docker/main.go @@ -0,0 +1,51 @@ +package main + +import ( + "context" + "errors" + "log/slog" + "net/http" + "os" + "os/signal" + "syscall" + "time" + + "github.com/rubycdp/ferrum/docker/internal/config" + "github.com/rubycdp/ferrum/docker/internal/server" +) + +const shutdownTimeout = 10 * time.Second + +func main() { + if config.BrowserPath == "" { + slog.Error("BROWSER_PATH is not set") + os.Exit(1) + } + + sessions := server.NewSessions(config.BrowserPath) + srv := &http.Server{ + Addr: config.Address, + Handler: server.New(sessions), + ReadHeaderTimeout: 10 * time.Second, + } + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + go func() { + <-ctx.Done() + slog.Info("shutting down") + + shutdownCtx, cancel := context.WithTimeout(context.Background(), shutdownTimeout) + defer cancel() + srv.Shutdown(shutdownCtx) + }() + + slog.Info("listening", "address", config.Address, "browser", config.BrowserPath, "version", config.Version) + if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) { + slog.Error("server failed", "error", err) + os.Exit(1) + } + + sessions.StopAll() +} From 5f4b310d5e0c0cc4ef3ab710ec680cae7bd862a5 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Mon, 12 Oct 2026 21:12:03 +0300 Subject: [PATCH 02/23] feat(docker): add chrome and chromium images for amd64 and arm64 --- docker/.dockerignore | 3 ++ docker/Dockerfile | 88 ++++++++++++++++++++++++++++++++++++++++ docker/Makefile | 53 ++++++++++++++++++++++++ docker/deploy/fonts.conf | 12 ++++++ 4 files changed, 156 insertions(+) create mode 100644 docker/.dockerignore create mode 100644 docker/Dockerfile create mode 100644 docker/Makefile create mode 100644 docker/deploy/fonts.conf diff --git a/docker/.dockerignore b/docker/.dockerignore new file mode 100644 index 00000000..4a91c419 --- /dev/null +++ b/docker/.dockerignore @@ -0,0 +1,3 @@ +bin/ +DESIGN.md +Makefile diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 00000000..e4e67741 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,88 @@ +ARG BROWSER=chromium +ARG GO_VERSION=1.27 +ARG DEBIAN_VERSION=trixie + +FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-${DEBIAN_VERSION} AS build +ARG TARGETOS +ARG TARGETARCH +ARG VERSION=dev +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +COPY . . +RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath \ + -ldflags="-s -w -X github.com/rubycdp/ferrum/docker/internal/config.Version=${VERSION}" \ + -o /out/driver . + +FROM --platform=$BUILDPLATFORM debian:${DEBIAN_VERSION}-slim AS fonts +ARG SELAWIK_SHA256=3f62c51e05e3b5a1e6241cf92a371f0be2ea1183aa87b30718bbd40832a8d423 +ADD --checksum=sha256:${SELAWIK_SHA256} \ + https://github.com/microsoft/Selawik/releases/download/1.01/Selawik_Release.zip /tmp/selawik.zip +RUN apt-get update && apt-get install -y --no-install-recommends unzip \ + && mkdir -p /out/selawik && unzip -j /tmp/selawik.zip '*.ttf' -d /out/selawik + +FROM debian:${DEBIAN_VERSION}-slim AS base +LABEL org.opencontainers.image.source=https://github.com/rubycdp/ferrum +LABEL org.opencontainers.image.licenses=MIT + +ENV DEBIAN_FRONTEND=noninteractive \ + LANG=C.UTF-8 \ + TZ=UTC + +RUN printf 'APT::Install-Recommends "false";\n' > /etc/apt/apt.conf.d/99no-recommends \ + && apt-get update && apt-get install -y \ + ca-certificates \ + curl \ + dumb-init \ + tzdata \ + xvfb \ + fontconfig \ + fonts-crosextra-caladea \ + fonts-crosextra-carlito \ + fonts-dejavu-core \ + fonts-freefont-ttf \ + fonts-gfs-neohellenic \ + fonts-indic \ + fonts-ipafont-gothic \ + fonts-liberation \ + fonts-noto-cjk \ + fonts-noto-color-emoji \ + fonts-open-sans \ + fonts-roboto \ + fonts-thai-tlwg \ + fonts-wqy-zenhei \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=fonts /out/selawik /usr/share/fonts/truetype/selawik +COPY deploy/fonts.conf /etc/fonts/local.conf +RUN fc-cache -f + +FROM base AS chromium +RUN apt-get update && apt-get install -y chromium \ + && rm -rf /var/lib/apt/lists/* +ENV BROWSER_PATH=/usr/bin/chromium + +FROM base AS chrome +ARG TARGETARCH +RUN curl -fsSL --retry 5 --retry-all-errors -o /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_${TARGETARCH}.deb \ + && apt-get update && apt-get install -y /tmp/chrome.deb \ + && rm -rf /var/lib/apt/lists/* /tmp/chrome.deb +ENV BROWSER_PATH=/usr/bin/google-chrome + +FROM ${BROWSER} + +RUN mkdir -p /home/driver && chmod 1777 /home/driver \ + && useradd --uid 999 --home-dir /home/driver --no-create-home --shell /usr/sbin/nologin driver + +ENV HOME=/home/driver \ + XDG_CONFIG_HOME=/home/driver/.config \ + XDG_CACHE_HOME=/home/driver/.cache + +COPY --from=build /out/driver /usr/local/bin/driver + +USER driver +WORKDIR /home/driver +EXPOSE 3000 +STOPSIGNAL SIGTERM + +ENTRYPOINT ["dumb-init", "--", "driver"] diff --git a/docker/Makefile b/docker/Makefile new file mode 100644 index 00000000..0b262955 --- /dev/null +++ b/docker/Makefile @@ -0,0 +1,53 @@ +VERSION := $(shell git rev-parse --short HEAD) +LDFLAGS := "-s -w -X 'github.com/rubycdp/ferrum/docker/internal/config.Version=$(VERSION)'" +NAME := "driver" +BIN := "bin/$(NAME)" + +.PHONY: build run fmt formatting-check lint vet test clean docker* version + +CGO_ENABLED ?= 0 +build: fmt + CGO_ENABLED=$(CGO_ENABLED) go build -o $(BIN) -ldflags=$(LDFLAGS) . + +run: build + $(BIN) + +fmt: + go fmt ./... + +formatting-check: + @gofmt -l -d $$(find . -not -path '*/\.*' -type f -name '*.go') | grep ^ && exit 1 || true > /dev/null 2>&1 + +lint: + staticcheck ./... + +vet: + go vet ./... + +test: + go test -race ./... + +clean: + rm -f bin/* + +BROWSER ?= chromium +IMAGE ?= ghcr.io/rubycdp/$(BROWSER) +TAG ?= latest + +docker.build: + docker buildx build --load --build-arg BROWSER=$(BROWSER) --build-arg VERSION=$(VERSION) -t $(IMAGE):$(TAG) . + +docker.run: + docker run --rm --name $(BROWSER) -p 3000:3000 $(IMAGE):$(TAG) + +docker.stop: + docker stop $(BROWSER) + +docker.sh: + docker run --rm -it --entrypoint /bin/bash $(IMAGE):$(TAG) + +docker.ssh: + docker exec -it $(BROWSER) /bin/bash + +version: + @echo $(VERSION) diff --git a/docker/deploy/fonts.conf b/docker/deploy/fonts.conf new file mode 100644 index 00000000..266288e9 --- /dev/null +++ b/docker/deploy/fonts.conf @@ -0,0 +1,12 @@ + + + + + Segoe UI + Selawik + + + Tahoma + DejaVu Sans + + From 21cddbdb2599bd0cc824f659e979474f156d2a07 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Mon, 12 Oct 2026 21:12:03 +0300 Subject: [PATCH 03/23] chore(ci): build, smoke-test and publish docker images on native runners --- .github/workflows/docker.yml | 151 +++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 .github/workflows/docker.yml diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml new file mode 100644 index 00000000..b8653b8f --- /dev/null +++ b/.github/workflows/docker.yml @@ -0,0 +1,151 @@ +name: Docker +on: + push: + branches: + - main + paths: + - "docker/**" + - ".github/workflows/docker.yml" + pull_request: + paths: + - "docker/**" + - ".github/workflows/docker.yml" + workflow_dispatch: + +jobs: + build: + name: Build ${{ matrix.browser }} (${{ matrix.arch }}) + strategy: + fail-fast: false + matrix: + browser: [chrome, chromium] + arch: [amd64, arm64] + include: + - arch: amd64 + runner: ubuntu-latest + - arch: arm64 + runner: ubuntu-24.04-arm + runs-on: ${{ matrix.runner }} + timeout-minutes: 30 + permissions: + contents: read + packages: write + env: + IMAGE: ghcr.io/${{ github.repository_owner }}/${{ matrix.browser }} + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to registry + if: github.event_name != 'pull_request' + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Build image + uses: docker/build-push-action@v6 + with: + context: docker + platforms: linux/${{ matrix.arch }} + build-args: | + BROWSER=${{ matrix.browser }} + VERSION=${{ github.sha }} + load: true + tags: ${{ matrix.browser }}:test + cache-from: type=gha,scope=${{ matrix.browser }}-${{ matrix.arch }} + cache-to: type=gha,mode=max,scope=${{ matrix.browser }}-${{ matrix.arch }} + + - name: Smoke test + run: | + docker run -d --name smoke -p 3000:3000 ${{ matrix.browser }}:test + for i in $(seq 1 30); do + curl -sf http://localhost:3000/json/version | grep -q webSocketDebuggerUrl && break + sleep 1 + done + docker logs smoke + curl -sf http://localhost:3000/json/version | tee /dev/stderr | grep -q webSocketDebuggerUrl + + - name: Detect browser version + run: | + mkdir -p /tmp/meta + docker run --rm --entrypoint sh ${{ matrix.browser }}:test -c '"$BROWSER_PATH" --version' \ + | grep -oE '[0-9]+(\.[0-9]+){3}' | head -1 > "/tmp/meta/version-${{ matrix.arch }}" + cat "/tmp/meta/version-${{ matrix.arch }}" + + - name: Push by digest + id: push + if: github.event_name != 'pull_request' + uses: docker/build-push-action@v6 + with: + context: docker + platforms: linux/${{ matrix.arch }} + build-args: | + BROWSER=${{ matrix.browser }} + VERSION=${{ github.sha }} + cache-from: type=gha,scope=${{ matrix.browser }}-${{ matrix.arch }} + outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true + + - name: Export digest + if: github.event_name != 'pull_request' + run: | + mkdir -p /tmp/meta/digests + touch "/tmp/meta/digests/${DIGEST#sha256:}" + env: + DIGEST: ${{ steps.push.outputs.digest }} + + - name: Upload metadata + if: github.event_name != 'pull_request' + uses: actions/upload-artifact@v4 + with: + name: meta-${{ matrix.browser }}-${{ matrix.arch }} + path: /tmp/meta + retention-days: 1 + + publish: + name: Publish ${{ matrix.browser }} + if: github.event_name != 'pull_request' + needs: build + strategy: + matrix: + browser: [chrome, chromium] + runs-on: ubuntu-latest + permissions: + packages: write + env: + IMAGE: ghcr.io/${{ github.repository_owner }}/${{ matrix.browser }} + steps: + - name: Download metadata + uses: actions/download-artifact@v4 + with: + pattern: meta-${{ matrix.browser }}-* + path: /tmp/meta + merge-multiple: true + + - name: Setup Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to registry + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Create multi-arch manifest + working-directory: /tmp/meta + run: | + tags=(-t "$IMAGE:latest") + amd64=$(cat version-amd64) + arm64=$(cat version-arm64) + if [ -n "$amd64" ] && [ "$amd64" = "$arm64" ]; then + tags+=(-t "$IMAGE:$amd64" -t "$IMAGE:${amd64%%.*}") + else + echo "::warning::Browser versions differ (amd64: $amd64, arm64: $arm64), publishing only latest" + fi + docker buildx imagetools create "${tags[@]}" $(printf "$IMAGE@sha256:%s " $(ls digests)) + docker buildx imagetools inspect "$IMAGE:latest" From ccaeee99db21f8a33878ee645f51b35e0ee9d220 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 08:20:13 +0300 Subject: [PATCH 04/23] feat(docker): add --help and --version to the driver --- docker/main.go | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docker/main.go b/docker/main.go index 04e61f57..1e618dbc 100644 --- a/docker/main.go +++ b/docker/main.go @@ -3,6 +3,8 @@ package main import ( "context" "errors" + "flag" + "fmt" "log/slog" "net/http" "os" @@ -16,7 +18,46 @@ import ( const shutdownTimeout = 10 * time.Second +const usage = `Usage: driver [options] + +Serves one Chrome per CDP connection, compatible with browserless v2 clients. +Each WebSocket gets a fresh Chrome that is killed when the client disconnects. + +Options: + -h, --help show this help + -v, --version print the version + +Environment: + BROWSER_PATH Chrome or Chromium binary (set by the image) + HOST address to listen on (default 0.0.0.0) + PORT port to listen on (default 3000) + +Endpoints: + GET /json/version start a Chrome, return its webSocketDebuggerUrl + WS /devtools/browser/ connect to the Chrome started by /json/version + WS /, /chrome, /chromium connect to a new Chrome + GET /health liveness, never starts a Chrome + +Examples: + docker run --rm -p 3000:3000 ghcr.io/rubycdp/chrome + docker run --rm -p 3000:3000 -v "$PWD:$PWD" ghcr.io/rubycdp/chromium + BROWSER_PATH=/usr/bin/chromium PORT=9222 driver + + Ferrum::Browser.new(url: "http://localhost:3000") + Ferrum::Browser.new(ws_url: "ws://localhost:3000/chrome") +` + func main() { + showVersion := flag.Bool("version", false, "") + flag.BoolVar(showVersion, "v", false, "") + flag.Usage = func() { fmt.Fprint(flag.CommandLine.Output(), usage) } + flag.Parse() + + if *showVersion { + fmt.Println(config.Version) + return + } + if config.BrowserPath == "" { slog.Error("BROWSER_PATH is not set") os.Exit(1) From 3f1a68db40438211f12070ca061444b3fcf3fd72 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:10:30 +0300 Subject: [PATCH 05/23] feat(docker): accept browserless launch options in the connection URL and document them in --help --- docker/DESIGN.md | 20 ++-- docker/internal/browser/browser.go | 4 +- docker/internal/browser/flags.go | 43 +++++++-- docker/internal/browser/options.go | 82 +++++++++++++++++ docker/internal/browser/options_test.go | 116 ++++++++++++++++++++++++ docker/internal/server/server.go | 24 ++++- docker/internal/server/sessions.go | 10 +- docker/main.go | 26 +++++- 8 files changed, 300 insertions(+), 25 deletions(-) create mode 100644 docker/internal/browser/options.go create mode 100644 docker/internal/browser/options_test.go diff --git a/docker/DESIGN.md b/docker/DESIGN.md index 21741c4d..3712a51f 100644 --- a/docker/DESIGN.md +++ b/docker/DESIGN.md @@ -81,14 +81,16 @@ CDP traffic. The query string from `/json/version` or the WebSocket URL accepts the same parameters as browserless v2: - `launch`: JSON or base64-encoded JSON, in v2's (Puppeteer launch) format. Supported keys: - - `args`: extra Chrome flags, e.g. `--proxy-server=…` or `--lang=…`; - - `headless`: `false` starts Chrome under Xvfb; + - `args`: extra Chrome flags, e.g. `--proxy-server=…` or `--accept-lang=…`. A flag with the same name as a + default replaces it; - `defaultViewport` `{width, height}`: becomes `--window-size`; - - `ignoreDefaultArgs`: `true`, or a list of default flags to drop. -- `timeout`: the session timeout in milliseconds. + - `ignoreDefaultArgs`: `true`, or a list of default flags to drop, by full flag or by name. + - later: `headless: false` starts Chrome under Xvfb. +- `timeout` (later): the session timeout in milliseconds. -Other parameters (`token`, `blockAds`, `blockAdsInclude`) and other `launch` keys are ignored, so existing -browserless URLs keep working. +`--remote-debugging-port` and `--user-data-dir` are always set by the driver. An invalid `launch` gets `400` +before any Chrome starts. Other parameters (`token`, `blockAds`, `blockAdsInclude`) and other `launch` keys are +ignored, so existing browserless URLs keep working. `driver --help` lists the launch options and default flags. Anyone who can connect can pass flags like `--renderer-cmd-prefix` and run commands in the container. There's no authentication, so never publish the port beyond localhost or a private network. @@ -178,14 +180,16 @@ Done: - `/health`. - A graceful shutdown stops every Chrome. - Trailing-slash routes (`/chrome/`, `/chromium/`, `/json/version/`), as in v2. +- The `launch` query parameter (`args`, `defaultViewport`, `ignoreDefaultArgs`). +- `driver --help` and `--version`. With `ws_url:`, Ferrum still calls `/json/version` before connecting to `/`. Pending Chromes are therefore matched by query string, so a connection to `/` claims the one that call started, instead of starting a second. Not done yet: - `CONCURRENT`, `QUEUED` and `TIMEOUT`, plus `EXTERNAL`; -- the `launch` and `timeout` query parameters; +- the `timeout` query parameter and `launch.headless`; - the DevTools proxy (`/json/list`, `/json/new`, `/json/protocol`, `/devtools/*`); - cleaned-up default flags; - a `HEALTHCHECK` in the image; -- Go tests. +- Go tests beyond the launch options, and a Go job in CI. diff --git a/docker/internal/browser/browser.go b/docker/internal/browser/browser.go index bca8e0d4..6214d05c 100644 --- a/docker/internal/browser/browser.go +++ b/docker/internal/browser/browser.go @@ -31,7 +31,7 @@ type Browser struct { stopOnce sync.Once } -func Start(binary string, timeout time.Duration) (*Browser, error) { +func Start(binary string, options Options, timeout time.Duration) (*Browser, error) { dir, err := os.MkdirTemp("", "chrome-") if err != nil { return nil, fmt.Errorf("create user data dir: %w", err) @@ -43,7 +43,7 @@ func Start(binary string, timeout time.Duration) (*Browser, error) { return nil, fmt.Errorf("create stderr pipe: %w", err) } - cmd := exec.Command(binary, Flags(dir)...) + cmd := exec.Command(binary, Flags(dir, options)...) cmd.Stderr = writer cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} diff --git a/docker/internal/browser/flags.go b/docker/internal/browser/flags.go index 758115f5..88e9771b 100644 --- a/docker/internal/browser/flags.go +++ b/docker/internal/browser/flags.go @@ -2,6 +2,8 @@ package browser import ( "fmt" + "slices" + "strings" ) const ( @@ -47,17 +49,44 @@ var ( "--safebrowsing-disable-auto-update", "--password-store=basic", "--no-startup-window", + fmt.Sprintf("%s=%s", chromeWindowSizeFlag, chromeResolution), } + + reserved = []string{chromeDebuggingPortFlag, chromeUserDataDirFlag} ) -func Flags(dir string) []string { - f := make([]string, len(flags)) - copy(f, flags) - f = append(f, []string{ +func Defaults() []string { + return slices.Clone(flags) +} + +func Flags(dir string, options Options) []string { + f := slices.DeleteFunc(slices.Clone(flags), options.IgnoreDefaultArgs.ignores) + + extra := slices.Clone(options.Args) + if v := options.DefaultViewport; v != nil { + extra = append(extra, fmt.Sprintf("%s=%d,%d", chromeWindowSizeFlag, v.Width, v.Height)) + } + + for _, flag := range extra { + name := flagName(flag) + if slices.Contains(reserved, name) { + continue + } + + if i := slices.IndexFunc(f, func(d string) bool { return flagName(d) == name }); i >= 0 { + f[i] = flag + } else { + f = append(f, flag) + } + } + + return append(f, fmt.Sprintf("%s=%s", chromeDebuggingPortFlag, "0"), - fmt.Sprintf("%s=%s", chromeWindowSizeFlag, chromeResolution), fmt.Sprintf("%s=%s", chromeUserDataDirFlag, dir), - }...) + ) +} - return f +func flagName(flag string) string { + name, _, _ := strings.Cut(flag, "=") + return name } diff --git a/docker/internal/browser/options.go b/docker/internal/browser/options.go new file mode 100644 index 00000000..1d53677a --- /dev/null +++ b/docker/internal/browser/options.go @@ -0,0 +1,82 @@ +package browser + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "strings" +) + +type Options struct { + Args []string `json:"args"` + DefaultViewport *Viewport `json:"defaultViewport"` + IgnoreDefaultArgs IgnoredArgs `json:"ignoreDefaultArgs"` +} + +type Viewport struct { + Width int `json:"width"` + Height int `json:"height"` +} + +type IgnoredArgs struct { + All bool + Args []string +} + +func (i *IgnoredArgs) UnmarshalJSON(data []byte) error { + if err := json.Unmarshal(data, &i.All); err == nil { + return nil + } + + return json.Unmarshal(data, &i.Args) +} + +func (i IgnoredArgs) ignores(flag string) bool { + if i.All { + return true + } + + for _, arg := range i.Args { + if arg == flag || arg == flagName(flag) { + return true + } + } + + return false +} + +func ParseOptions(launch string) (Options, error) { + var options Options + if launch == "" { + return options, nil + } + + data := []byte(launch) + if !strings.HasPrefix(strings.TrimSpace(launch), "{") { + decoded, err := decodeBase64(launch) + if err != nil { + return options, fmt.Errorf("launch is neither JSON nor base64: %w", err) + } + data = decoded + } + + if err := json.Unmarshal(data, &options); err != nil { + return options, fmt.Errorf("invalid launch options: %w", err) + } + + if v := options.DefaultViewport; v != nil && (v.Width <= 0 || v.Height <= 0) { + return options, fmt.Errorf("invalid defaultViewport %dx%d", v.Width, v.Height) + } + + return options, nil +} + +func decodeBase64(s string) ([]byte, error) { + for _, encoding := range []*base64.Encoding{base64.StdEncoding, base64.URLEncoding, base64.RawStdEncoding, base64.RawURLEncoding} { + if data, err := encoding.DecodeString(s); err == nil { + return data, nil + } + } + + return nil, base64.CorruptInputError(0) +} diff --git a/docker/internal/browser/options_test.go b/docker/internal/browser/options_test.go new file mode 100644 index 00000000..afd65c1b --- /dev/null +++ b/docker/internal/browser/options_test.go @@ -0,0 +1,116 @@ +package browser + +import ( + "encoding/base64" + "slices" + "testing" +) + +func TestParseOptions(t *testing.T) { + launch := `{"args":["--accept-lang=de-DE"],"defaultViewport":{"width":1280,"height":720},"ignoreDefaultArgs":["--headless"],"headless":false}` + + for name, input := range map[string]string{ + "json": launch, + "base64": base64.StdEncoding.EncodeToString([]byte(launch)), + "base64 url": base64.RawURLEncoding.EncodeToString([]byte(launch)), + } { + t.Run(name, func(t *testing.T) { + options, err := ParseOptions(input) + if err != nil { + t.Fatal(err) + } + + if !slices.Equal(options.Args, []string{"--accept-lang=de-DE"}) { + t.Errorf("args = %v", options.Args) + } + if v := options.DefaultViewport; v == nil || v.Width != 1280 || v.Height != 720 { + t.Errorf("defaultViewport = %+v", v) + } + if !slices.Equal(options.IgnoreDefaultArgs.Args, []string{"--headless"}) { + t.Errorf("ignoreDefaultArgs = %+v", options.IgnoreDefaultArgs) + } + }) + } +} + +func TestParseOptionsIgnoreAllDefaultArgs(t *testing.T) { + options, err := ParseOptions(`{"ignoreDefaultArgs":true}`) + if err != nil { + t.Fatal(err) + } + + if !options.IgnoreDefaultArgs.All { + t.Error("expected all default args to be ignored") + } +} + +func TestParseOptionsErrors(t *testing.T) { + for name, input := range map[string]string{ + "not json or base64": "%%%", + "broken json": `{"args":`, + "wrong type": `{"args":"--foo"}`, + "empty viewport": `{"defaultViewport":{"width":0,"height":720}}`, + } { + t.Run(name, func(t *testing.T) { + if _, err := ParseOptions(input); err == nil { + t.Error("expected an error") + } + }) + } +} + +func TestFlags(t *testing.T) { + t.Run("defaults", func(t *testing.T) { + f := Flags("/tmp/profile", Options{}) + + if !slices.Equal(f[:len(flags)], flags) { + t.Errorf("defaults changed: %v", f) + } + if !slices.Equal(f[len(flags):], []string{"--remote-debugging-port=0", "--user-data-dir=/tmp/profile"}) { + t.Errorf("managed flags = %v", f[len(flags):]) + } + }) + + t.Run("args replace defaults by name and add the rest", func(t *testing.T) { + f := Flags("/tmp/profile", Options{Args: []string{"--window-size=800,600", "--accept-lang=de-DE"}}) + + if !slices.Contains(f, "--window-size=800,600") || slices.Contains(f, "--window-size=1920,1080") { + t.Errorf("window size not replaced: %v", f) + } + if !slices.Contains(f, "--accept-lang=de-DE") { + t.Errorf("flag not added: %v", f) + } + }) + + t.Run("defaultViewport becomes the window size", func(t *testing.T) { + f := Flags("/tmp/profile", Options{DefaultViewport: &Viewport{Width: 1280, Height: 720}}) + + if !slices.Contains(f, "--window-size=1280,720") || slices.Contains(f, "--window-size=1920,1080") { + t.Errorf("window size not replaced: %v", f) + } + }) + + t.Run("managed flags can't be overridden", func(t *testing.T) { + f := Flags("/tmp/profile", Options{ + Args: []string{"--remote-debugging-port=9222", "--user-data-dir=/elsewhere"}, + IgnoreDefaultArgs: IgnoredArgs{All: true}, + }) + + if !slices.Equal(f, []string{"--remote-debugging-port=0", "--user-data-dir=/tmp/profile"}) { + t.Errorf("flags = %v", f) + } + }) + + t.Run("ignoreDefaultArgs drops by full flag or name", func(t *testing.T) { + f := Flags("/tmp/profile", Options{IgnoreDefaultArgs: IgnoredArgs{Args: []string{"--headless", "--disable-features"}}}) + + for _, flag := range f { + if flagName(flag) == "--headless" || flagName(flag) == "--disable-features" { + t.Errorf("%s not dropped", flag) + } + } + if len(f) != len(flags) { + t.Errorf("expected %d flags, got %d", len(flags), len(f)) + } + }) +} diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index 1871c453..9f18a239 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -40,7 +40,12 @@ func (s *Server) health(w http.ResponseWriter, _ *http.Request) { } func (s *Server) version(w http.ResponseWriter, r *http.Request) { - b, err := s.sessions.Start() + options, ok := launchOptions(w, r) + if !ok { + return + } + + b, err := s.sessions.Start(options) if err != nil { slog.Error("cannot start browser", "error", err) http.Error(w, "cannot start browser", http.StatusServiceUnavailable) @@ -83,7 +88,12 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { return } - b, err := s.sessions.ClaimOrStart(r.URL.RawQuery) + options, ok := launchOptions(w, r) + if !ok { + return + } + + b, err := s.sessions.ClaimOrStart(r.URL.RawQuery, options) if err != nil { slog.Error("cannot start browser", "error", err) http.Error(w, "cannot start browser", http.StatusServiceUnavailable) @@ -119,6 +129,16 @@ func proxyTo(b *browser.Browser) *httputil.ReverseProxy { } } +func launchOptions(w http.ResponseWriter, r *http.Request) (browser.Options, bool) { + options, err := browser.ParseOptions(r.URL.Query().Get("launch")) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return options, false + } + + return options, true +} + func fetchVersion(b *browser.Browser) (map[string]any, error) { resp, err := http.Get("http://" + b.Addr + "/json/version") if err != nil { diff --git a/docker/internal/server/sessions.go b/docker/internal/server/sessions.go index 2fab7da6..e462da24 100644 --- a/docker/internal/server/sessions.go +++ b/docker/internal/server/sessions.go @@ -34,8 +34,8 @@ func NewSessions(binary string) *Sessions { } } -func (s *Sessions) Start() (*browser.Browser, error) { - b, err := browser.Start(s.binary, startTimeout) +func (s *Sessions) Start(options browser.Options) (*browser.Browser, error) { + b, err := browser.Start(s.binary, options, startTimeout) if err != nil { return nil, err } @@ -75,10 +75,10 @@ func (s *Sessions) Claim(id string) *browser.Browser { return r.browser } -func (s *Sessions) ClaimOrStart(options string) (*browser.Browser, error) { +func (s *Sessions) ClaimOrStart(query string, options browser.Options) (*browser.Browser, error) { s.mu.Lock() for id, r := range s.pending { - if r.options == options { + if r.options == query { delete(s.pending, id) s.mu.Unlock() return r.browser, nil @@ -86,7 +86,7 @@ func (s *Sessions) ClaimOrStart(options string) (*browser.Browser, error) { } s.mu.Unlock() - return s.Start() + return s.Start(options) } func (s *Sessions) Stop(b *browser.Browser) { diff --git a/docker/main.go b/docker/main.go index 1e618dbc..cbd416da 100644 --- a/docker/main.go +++ b/docker/main.go @@ -9,9 +9,11 @@ import ( "net/http" "os" "os/signal" + "strings" "syscall" "time" + "github.com/rubycdp/ferrum/docker/internal/browser" "github.com/rubycdp/ferrum/docker/internal/config" "github.com/rubycdp/ferrum/docker/internal/server" ) @@ -38,6 +40,19 @@ Endpoints: WS /, /chrome, /chromium connect to a new Chrome GET /health liveness, never starts a Chrome +Launch options (per connection, in the URL of /json/version or the WebSocket): + launch= + args extra Chrome flags, a flag with a default's name replaces it + defaultViewport {"width":1280,"height":720}, becomes --window-size + ignoreDefaultArgs true, or a list of default flags to drop + Other launch keys and query params (token, blockAds) are ignored. + --remote-debugging-port and --user-data-dir are managed by the driver. + + http://localhost:3000?launch={"args":["--accept-lang=de-DE"]} + ws://localhost:3000/chrome?launch={"defaultViewport":{"width":1280,"height":720}} + +Default Chrome flags: +%s Examples: docker run --rm -p 3000:3000 ghcr.io/rubycdp/chrome docker run --rm -p 3000:3000 -v "$PWD:$PWD" ghcr.io/rubycdp/chromium @@ -50,7 +65,7 @@ Examples: func main() { showVersion := flag.Bool("version", false, "") flag.BoolVar(showVersion, "v", false, "") - flag.Usage = func() { fmt.Fprint(flag.CommandLine.Output(), usage) } + flag.Usage = func() { fmt.Fprintf(flag.CommandLine.Output(), usage, defaultFlags()) } flag.Parse() if *showVersion { @@ -90,3 +105,12 @@ func main() { sessions.StopAll() } + +func defaultFlags() string { + var b strings.Builder + for _, flag := range browser.Defaults() { + b.WriteString(" " + flag + "\n") + } + + return b.String() +} From 0dadf3b8b888469aad9b06bc290c3c159ffe6e84 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:36:32 +0300 Subject: [PATCH 06/23] ref(docker): use Ferrum's default Chrome flags and window size, except --remote-allow-origins --- docker/internal/browser/flags.go | 59 +++++++++++++++---------- docker/internal/browser/options_test.go | 4 +- 2 files changed, 37 insertions(+), 26 deletions(-) diff --git a/docker/internal/browser/flags.go b/docker/internal/browser/flags.go index 88e9771b..112119f9 100644 --- a/docker/internal/browser/flags.go +++ b/docker/internal/browser/flags.go @@ -10,46 +10,57 @@ const ( chromeDebuggingPortFlag = "--remote-debugging-port" chromeWindowSizeFlag = "--window-size" chromeUserDataDirFlag = "--user-data-dir" - chromeResolution = "1920,1080" + defaultWidth = 1024 + defaultHeight = 768 ) var ( flags = []string{ - "--no-sandbox", - "--headless", - "--disable-gpu", - "--hide-scrollbars", - "--mute-audio", - "--enable-automation", - "--disable-web-security", - "--disable-session-crashed-bubble", - "--disable-breakpad", - "--disable-sync", - "--no-first-run", - "--use-mock-keychain", - "--keep-alive-for-test", - "--disable-popup-blocking", - "--disable-extensions", - "--disable-component-extensions-with-background-pages", - "--disable-hang-monitor", - "--disable-features=site-per-process,TranslateUI", - "--disable-translate", + "--allow-pre-commit-input", "--disable-background-networking", - "--enable-features=NetworkService,NetworkServiceInProcess", "--disable-background-timer-throttling", "--disable-backgrounding-occluded-windows", + "--disable-blink-features=AutomationControlled", + "--disable-breakpad", "--disable-client-side-phishing-detection", + "--disable-component-extensions-with-background-pages", + "--disable-component-update", + "--disable-crash-reporter", "--disable-default-apps", "--disable-dev-shm-usage", + "--disable-extensions", + "--disable-features=site-per-process,IsolateOrigins,TranslateUI,Translate,MacAppCodeSignClone,InterestFeedContentSuggestion,OptimizationHints,AcceptCHFrame,MediaRouter", + "--disable-field-trial-config", + "--disable-hang-monitor", + "--disable-infobars", "--disable-ipc-flooding-protection", + "--disable-popup-blocking", "--disable-prompt-on-repost", "--disable-renderer-backgrounding", + "--disable-search-engine-choice-screen", + "--disable-session-crashed-bubble", + "--disable-site-isolation-trials", + "--disable-smooth-scrolling", + "--disable-sync", + "--disable-translate", + "--enable-automation", + "--enable-features=NetworkService,NetworkServiceInProcess", "--force-color-profile=srgb", + "--headless", + "--hide-scrollbars", + "--keep-alive-for-test", "--metrics-recording-only", - "--safebrowsing-disable-auto-update", - "--password-store=basic", + "--mute-audio", + "--no-crash-upload", + "--no-default-browser-check", + "--no-first-run", "--no-startup-window", - fmt.Sprintf("%s=%s", chromeWindowSizeFlag, chromeResolution), + "--password-store=basic", + "--safebrowsing-disable-auto-update", + "--use-mock-keychain", + "--no-sandbox", + "--disable-setuid-sandbox", + fmt.Sprintf("%s=%d,%d", chromeWindowSizeFlag, defaultWidth, defaultHeight), } reserved = []string{chromeDebuggingPortFlag, chromeUserDataDirFlag} diff --git a/docker/internal/browser/options_test.go b/docker/internal/browser/options_test.go index afd65c1b..5be6914d 100644 --- a/docker/internal/browser/options_test.go +++ b/docker/internal/browser/options_test.go @@ -74,7 +74,7 @@ func TestFlags(t *testing.T) { t.Run("args replace defaults by name and add the rest", func(t *testing.T) { f := Flags("/tmp/profile", Options{Args: []string{"--window-size=800,600", "--accept-lang=de-DE"}}) - if !slices.Contains(f, "--window-size=800,600") || slices.Contains(f, "--window-size=1920,1080") { + if !slices.Contains(f, "--window-size=800,600") || slices.Contains(f, "--window-size=1024,768") { t.Errorf("window size not replaced: %v", f) } if !slices.Contains(f, "--accept-lang=de-DE") { @@ -85,7 +85,7 @@ func TestFlags(t *testing.T) { t.Run("defaultViewport becomes the window size", func(t *testing.T) { f := Flags("/tmp/profile", Options{DefaultViewport: &Viewport{Width: 1280, Height: 720}}) - if !slices.Contains(f, "--window-size=1280,720") || slices.Contains(f, "--window-size=1920,1080") { + if !slices.Contains(f, "--window-size=1280,720") || slices.Contains(f, "--window-size=1024,768") { t.Errorf("window size not replaced: %v", f) } }) From 4a96dbf0f2b39861063b5bd31914987f4429e470 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:37:28 +0300 Subject: [PATCH 07/23] feat(docker): limit sessions with CONCURRENT and QUEUED, answering 429 with Retry-After --- docker/internal/config/config.go | 49 +++++++++- docker/internal/server/server.go | 26 ++++-- docker/internal/server/sessions.go | 143 ++++++++++++++++++++++++----- docker/main.go | 15 ++- 4 files changed, 200 insertions(+), 33 deletions(-) diff --git a/docker/internal/config/config.go b/docker/internal/config/config.go index 2882df3c..cc083ecd 100644 --- a/docker/internal/config/config.go +++ b/docker/internal/config/config.go @@ -1,17 +1,48 @@ package config import ( + "fmt" "net" "os" + "runtime" + "strconv" ) var ( Version string - BrowserPath = os.Getenv("BROWSER_PATH") - Address = net.JoinHostPort(env("HOST", "0.0.0.0"), env("PORT", "3000")) + BrowserPath string + Address string + Concurrent int + Queued int ) +func Load() error { + BrowserPath = os.Getenv("BROWSER_PATH") + if BrowserPath == "" { + return fmt.Errorf("BROWSER_PATH is not set") + } + + Address = net.JoinHostPort(env("HOST", "0.0.0.0"), env("PORT", "3000")) + + var err error + if Concurrent, err = intEnv("CONCURRENT", runtime.NumCPU()); err != nil { + return err + } + if Concurrent < 1 { + return fmt.Errorf("CONCURRENT must be at least 1, got %d", Concurrent) + } + + if Queued, err = intEnv("QUEUED", 10); err != nil { + return err + } + if Queued < 0 { + return fmt.Errorf("QUEUED must not be negative, got %d", Queued) + } + + return nil +} + func env(key, fallback string) string { if value, ok := os.LookupEnv(key); ok && value != "" { return value @@ -19,3 +50,17 @@ func env(key, fallback string) string { return fallback } + +func intEnv(key string, fallback int) (int, error) { + value := env(key, "") + if value == "" { + return fallback, nil + } + + n, err := strconv.Atoi(value) + if err != nil { + return 0, fmt.Errorf("%s must be a number, got %q", key, value) + } + + return n, nil +} diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index 9f18a239..81e7f215 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -1,7 +1,9 @@ package server import ( + "context" "encoding/json" + "errors" "fmt" "log/slog" "net/http" @@ -12,6 +14,8 @@ import ( "github.com/rubycdp/ferrum/docker/internal/browser" ) +const retryAfter = "5" + type Server struct { sessions *Sessions } @@ -45,10 +49,9 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { return } - b, err := s.sessions.Start(options) + b, err := s.sessions.Start(r.Context(), options) if err != nil { - slog.Error("cannot start browser", "error", err) - http.Error(w, "cannot start browser", http.StatusServiceUnavailable) + startError(w, err) return } @@ -93,10 +96,9 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { return } - b, err := s.sessions.ClaimOrStart(r.URL.RawQuery, options) + b, err := s.sessions.ClaimOrStart(r.Context(), r.URL.RawQuery, options) if err != nil { - slog.Error("cannot start browser", "error", err) - http.Error(w, "cannot start browser", http.StatusServiceUnavailable) + startError(w, err) return } @@ -111,6 +113,18 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, b *browser.Browse slog.Info("session closed", "id", b.ID, "client", r.RemoteAddr) } +func startError(w http.ResponseWriter, err error) { + switch { + case errors.Is(err, ErrQueueFull): + w.Header().Set("Retry-After", retryAfter) + http.Error(w, err.Error(), http.StatusTooManyRequests) + case errors.Is(err, context.Canceled): + default: + slog.Error("cannot start browser", "error", err) + http.Error(w, "cannot start browser", http.StatusServiceUnavailable) + } +} + func proxyTo(b *browser.Browser) *httputil.ReverseProxy { return &httputil.ReverseProxy{ Rewrite: func(pr *httputil.ProxyRequest) { diff --git a/docker/internal/server/sessions.go b/docker/internal/server/sessions.go index e462da24..013d2b8c 100644 --- a/docker/internal/server/sessions.go +++ b/docker/internal/server/sessions.go @@ -1,6 +1,8 @@ package server import ( + "context" + "errors" "log/slog" "sync" "time" @@ -13,30 +15,61 @@ const ( claimTimeout = 30 * time.Second ) +var ( + ErrQueueFull = errors.New("too many sessions") + ErrClosed = errors.New("shutting down") +) + +type SessionsConfig struct { + Binary string + Concurrent int + Queued int + ClaimTimeout time.Duration +} + type Sessions struct { - binary string + binary string + queued int + claimTimeout time.Duration + slots chan struct{} + closed chan struct{} + closeOnce sync.Once mu sync.Mutex + waiting int browsers map[string]*browser.Browser pending map[string]reservation } type reservation struct { browser *browser.Browser - options string + query string } -func NewSessions(binary string) *Sessions { +func NewSessions(config SessionsConfig) *Sessions { + if config.ClaimTimeout == 0 { + config.ClaimTimeout = claimTimeout + } + return &Sessions{ - binary: binary, - browsers: make(map[string]*browser.Browser), - pending: make(map[string]reservation), + binary: config.Binary, + queued: config.Queued, + claimTimeout: config.ClaimTimeout, + slots: make(chan struct{}, max(config.Concurrent, 1)), + closed: make(chan struct{}), + browsers: make(map[string]*browser.Browser), + pending: make(map[string]reservation), } } -func (s *Sessions) Start(options browser.Options) (*browser.Browser, error) { +func (s *Sessions) Start(ctx context.Context, options browser.Options) (*browser.Browser, error) { + if err := s.acquire(ctx); err != nil { + return nil, err + } + b, err := browser.Start(s.binary, options, startTimeout) if err != nil { + s.release() return nil, err } @@ -44,6 +77,11 @@ func (s *Sessions) Start(options browser.Options) (*browser.Browser, error) { s.browsers[b.ID] = b s.mu.Unlock() + if s.isClosed() { + s.Stop(b) + return nil, ErrClosed + } + go func() { <-b.Done() s.Stop(b) @@ -52,14 +90,14 @@ func (s *Sessions) Start(options browser.Options) (*browser.Browser, error) { return b, nil } -func (s *Sessions) Reserve(b *browser.Browser, options string) { +func (s *Sessions) Reserve(b *browser.Browser, query string) { s.mu.Lock() - s.pending[b.ID] = reservation{browser: b, options: options} + s.pending[b.ID] = reservation{browser: b, query: query} s.mu.Unlock() - time.AfterFunc(claimTimeout, func() { + time.AfterFunc(s.claimTimeout, func() { if s.Claim(b.ID) != nil { - slog.Warn("browser was never claimed, stopping", "id", b.ID, "timeout", claimTimeout) + slog.Warn("browser was never claimed, stopping", "id", b.ID, "timeout", s.claimTimeout) s.Stop(b) } }) @@ -75,10 +113,10 @@ func (s *Sessions) Claim(id string) *browser.Browser { return r.browser } -func (s *Sessions) ClaimOrStart(query string, options browser.Options) (*browser.Browser, error) { +func (s *Sessions) ClaimOrStart(ctx context.Context, query string, options browser.Options) (*browser.Browser, error) { s.mu.Lock() for id, r := range s.pending { - if r.options == query { + if r.query == query { delete(s.pending, id) s.mu.Unlock() return r.browser, nil @@ -86,29 +124,92 @@ func (s *Sessions) ClaimOrStart(query string, options browser.Options) (*browser } s.mu.Unlock() - return s.Start(options) + return s.Start(ctx, options) +} + +func (s *Sessions) Browsers() []*browser.Browser { + s.mu.Lock() + defer s.mu.Unlock() + + browsers := make([]*browser.Browser, 0, len(s.browsers)) + for _, b := range s.browsers { + browsers = append(browsers, b) + } + + return browsers } func (s *Sessions) Stop(b *browser.Browser) { s.mu.Lock() + _, running := s.browsers[b.ID] delete(s.browsers, b.ID) delete(s.pending, b.ID) s.mu.Unlock() + if !running { + return + } + b.Stop() + s.release() } func (s *Sessions) StopAll() { - s.mu.Lock() - browsers := make([]*browser.Browser, 0, len(s.browsers)) - for _, b := range s.browsers { - browsers = append(browsers, b) - } - s.mu.Unlock() + s.closeOnce.Do(func() { close(s.closed) }) var wg sync.WaitGroup - for _, b := range browsers { + for _, b := range s.Browsers() { wg.Go(func() { s.Stop(b) }) } wg.Wait() } + +func (s *Sessions) acquire(ctx context.Context) error { + select { + case <-s.closed: + return ErrClosed + case s.slots <- struct{}{}: + return nil + default: + } + + s.mu.Lock() + if s.waiting >= s.queued { + s.mu.Unlock() + return ErrQueueFull + } + s.waiting++ + s.mu.Unlock() + + defer func() { + s.mu.Lock() + s.waiting-- + s.mu.Unlock() + }() + + select { + case s.slots <- struct{}{}: + if s.isClosed() { + s.release() + return ErrClosed + } + return nil + case <-ctx.Done(): + return ctx.Err() + case <-s.closed: + return ErrClosed + } +} + +func (s *Sessions) isClosed() bool { + select { + case <-s.closed: + return true + default: + return false + } +} + +func (s *Sessions) release() { + <-s.slots +} diff --git a/docker/main.go b/docker/main.go index cbd416da..39cebb08 100644 --- a/docker/main.go +++ b/docker/main.go @@ -33,6 +33,8 @@ Environment: BROWSER_PATH Chrome or Chromium binary (set by the image) HOST address to listen on (default 0.0.0.0) PORT port to listen on (default 3000) + CONCURRENT running sessions at most (default: number of CPUs) + QUEUED connections waiting for a free session, the rest get 429 (default 10) Endpoints: GET /json/version start a Chrome, return its webSocketDebuggerUrl @@ -73,12 +75,16 @@ func main() { return } - if config.BrowserPath == "" { - slog.Error("BROWSER_PATH is not set") + if err := config.Load(); err != nil { + slog.Error("invalid configuration", "error", err) os.Exit(1) } - sessions := server.NewSessions(config.BrowserPath) + sessions := server.NewSessions(server.SessionsConfig{ + Binary: config.BrowserPath, + Concurrent: config.Concurrent, + Queued: config.Queued, + }) srv := &http.Server{ Addr: config.Address, Handler: server.New(sessions), @@ -97,7 +103,8 @@ func main() { srv.Shutdown(shutdownCtx) }() - slog.Info("listening", "address", config.Address, "browser", config.BrowserPath, "version", config.Version) + slog.Info("listening", "address", config.Address, "browser", config.BrowserPath, "version", config.Version, + "concurrent", config.Concurrent, "queued", config.Queued) if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) { slog.Error("server failed", "error", err) os.Exit(1) From 6472179e826240fa9010f1e17865163889693ac5 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:37:57 +0300 Subject: [PATCH 08/23] feat(docker): end sessions after TIMEOUT or ?timeout= and log why each session ended --- docker/internal/config/config.go | 8 +++++ docker/internal/server/server.go | 50 ++++++++++++++++++++++++------ docker/internal/server/sessions.go | 12 +++---- docker/main.go | 6 ++-- 4 files changed, 59 insertions(+), 17 deletions(-) diff --git a/docker/internal/config/config.go b/docker/internal/config/config.go index cc083ecd..f9f90090 100644 --- a/docker/internal/config/config.go +++ b/docker/internal/config/config.go @@ -6,6 +6,7 @@ import ( "os" "runtime" "strconv" + "time" ) var ( @@ -15,6 +16,7 @@ var ( Address string Concurrent int Queued int + Timeout time.Duration ) func Load() error { @@ -40,6 +42,12 @@ func Load() error { return fmt.Errorf("QUEUED must not be negative, got %d", Queued) } + timeout, err := intEnv("TIMEOUT", 0) + if err != nil { + return err + } + Timeout = time.Duration(max(timeout, 0)) * time.Millisecond + return nil } diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index 81e7f215..a8dbce58 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -9,19 +9,26 @@ import ( "net/http" "net/http/httputil" "net/url" + "strconv" "strings" + "time" "github.com/rubycdp/ferrum/docker/internal/browser" ) const retryAfter = "5" +type Config struct { + Timeout time.Duration +} + type Server struct { sessions *Sessions + config Config } -func New(sessions *Sessions) http.Handler { - s := &Server{sessions: sessions} +func New(sessions *Sessions, config Config) http.Handler { + s := &Server{sessions: sessions, config: config} mux := http.NewServeMux() handle(mux, "/health", s.health) @@ -48,6 +55,9 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { if !ok { return } + if _, ok := s.sessionTimeout(w, r); !ok { + return + } b, err := s.sessions.Start(r.Context(), options) if err != nil { @@ -58,7 +68,7 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { info, err := fetchVersion(b) if err != nil { slog.Error("cannot fetch browser version", "id", b.ID, "error", err) - s.sessions.Stop(b) + s.sessions.Stop(b, "version failed") http.Error(w, "cannot fetch browser version", http.StatusBadGateway) return } @@ -82,7 +92,7 @@ func (s *Server) claim(w http.ResponseWriter, r *http.Request) { return } - s.serve(w, r, b) + s.serve(w, r, b, s.config.Timeout) } func (s *Server) connect(w http.ResponseWriter, r *http.Request) { @@ -95,6 +105,10 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { if !ok { return } + timeout, ok := s.sessionTimeout(w, r) + if !ok { + return + } b, err := s.sessions.ClaimOrStart(r.Context(), r.URL.RawQuery, options) if err != nil { @@ -102,15 +116,33 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { return } - s.serve(w, r, b) + s.serve(w, r, b, timeout) } -func (s *Server) serve(w http.ResponseWriter, r *http.Request, b *browser.Browser) { - defer s.sessions.Stop(b) +func (s *Server) serve(w http.ResponseWriter, r *http.Request, b *browser.Browser, timeout time.Duration) { + if timeout > 0 { + timer := time.AfterFunc(timeout, func() { s.sessions.Stop(b, "timeout") }) + defer timer.Stop() + } - slog.Info("session opened", "id", b.ID, "client", r.RemoteAddr) + slog.Info("session opened", "id", b.ID, "client", r.RemoteAddr, "timeout", timeout) proxyTo(b).ServeHTTP(w, r) - slog.Info("session closed", "id", b.ID, "client", r.RemoteAddr) + s.sessions.Stop(b, "connection closed") +} + +func (s *Server) sessionTimeout(w http.ResponseWriter, r *http.Request) (time.Duration, bool) { + value := r.URL.Query().Get("timeout") + if value == "" { + return s.config.Timeout, true + } + + ms, err := strconv.Atoi(value) + if err != nil { + http.Error(w, fmt.Sprintf("invalid timeout %q", value), http.StatusBadRequest) + return 0, false + } + + return time.Duration(max(ms, 0)) * time.Millisecond, true } func startError(w http.ResponseWriter, err error) { diff --git a/docker/internal/server/sessions.go b/docker/internal/server/sessions.go index 013d2b8c..91f631fb 100644 --- a/docker/internal/server/sessions.go +++ b/docker/internal/server/sessions.go @@ -78,13 +78,13 @@ func (s *Sessions) Start(ctx context.Context, options browser.Options) (*browser s.mu.Unlock() if s.isClosed() { - s.Stop(b) + s.Stop(b, "shutdown") return nil, ErrClosed } go func() { <-b.Done() - s.Stop(b) + s.Stop(b, "browser exited") }() return b, nil @@ -97,8 +97,7 @@ func (s *Sessions) Reserve(b *browser.Browser, query string) { time.AfterFunc(s.claimTimeout, func() { if s.Claim(b.ID) != nil { - slog.Warn("browser was never claimed, stopping", "id", b.ID, "timeout", s.claimTimeout) - s.Stop(b) + s.Stop(b, "never claimed") } }) } @@ -139,7 +138,7 @@ func (s *Sessions) Browsers() []*browser.Browser { return browsers } -func (s *Sessions) Stop(b *browser.Browser) { +func (s *Sessions) Stop(b *browser.Browser, reason string) { s.mu.Lock() _, running := s.browsers[b.ID] delete(s.browsers, b.ID) @@ -152,6 +151,7 @@ func (s *Sessions) Stop(b *browser.Browser) { b.Stop() s.release() + slog.Info("session ended", "id", b.ID, "reason", reason) } func (s *Sessions) StopAll() { @@ -159,7 +159,7 @@ func (s *Sessions) StopAll() { var wg sync.WaitGroup for _, b := range s.Browsers() { - wg.Go(func() { s.Stop(b) }) + wg.Go(func() { s.Stop(b, "shutdown") }) } wg.Wait() } diff --git a/docker/main.go b/docker/main.go index 39cebb08..4d1ddc9a 100644 --- a/docker/main.go +++ b/docker/main.go @@ -35,6 +35,7 @@ Environment: PORT port to listen on (default 3000) CONCURRENT running sessions at most (default: number of CPUs) QUEUED connections waiting for a free session, the rest get 429 (default 10) + TIMEOUT session timeout in milliseconds (default 0, no timeout) Endpoints: GET /json/version start a Chrome, return its webSocketDebuggerUrl @@ -47,6 +48,7 @@ Launch options (per connection, in the URL of /json/version or the WebSocket): args extra Chrome flags, a flag with a default's name replaces it defaultViewport {"width":1280,"height":720}, becomes --window-size ignoreDefaultArgs true, or a list of default flags to drop + timeout= session timeout for this connection, overrides TIMEOUT Other launch keys and query params (token, blockAds) are ignored. --remote-debugging-port and --user-data-dir are managed by the driver. @@ -87,7 +89,7 @@ func main() { }) srv := &http.Server{ Addr: config.Address, - Handler: server.New(sessions), + Handler: server.New(sessions, server.Config{Timeout: config.Timeout}), ReadHeaderTimeout: 10 * time.Second, } @@ -104,7 +106,7 @@ func main() { }() slog.Info("listening", "address", config.Address, "browser", config.BrowserPath, "version", config.Version, - "concurrent", config.Concurrent, "queued", config.Queued) + "concurrent", config.Concurrent, "queued", config.Queued, "timeout", config.Timeout) if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) { slog.Error("server failed", "error", err) os.Exit(1) From 55d0557f98633fcc9108f464f53649caa4fd36ca Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:38:27 +0300 Subject: [PATCH 09/23] feat(docker): advertise EXTERNAL or PROXY_URL as the WebSocket address behind a proxy --- docker/internal/config/config.go | 20 ++++++++++++++++++++ docker/internal/server/server.go | 20 ++++++++++++++++---- docker/main.go | 4 +++- 3 files changed, 39 insertions(+), 5 deletions(-) diff --git a/docker/internal/config/config.go b/docker/internal/config/config.go index f9f90090..1deeffb0 100644 --- a/docker/internal/config/config.go +++ b/docker/internal/config/config.go @@ -3,6 +3,7 @@ package config import ( "fmt" "net" + "net/url" "os" "runtime" "strconv" @@ -17,6 +18,7 @@ var ( Concurrent int Queued int Timeout time.Duration + External *url.URL ) func Load() error { @@ -48,6 +50,10 @@ func Load() error { } Timeout = time.Duration(max(timeout, 0)) * time.Millisecond + if External, err = externalURL(); err != nil { + return err + } + return nil } @@ -72,3 +78,17 @@ func intEnv(key string, fallback int) (int, error) { return n, nil } + +func externalURL() (*url.URL, error) { + value := env("EXTERNAL", env("PROXY_URL", "")) + if value == "" { + return nil, nil + } + + u, err := url.Parse(value) + if err != nil || u.Host == "" { + return nil, fmt.Errorf("EXTERNAL must be an absolute URL, got %q", value) + } + + return u, nil +} diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index a8dbce58..6167a70f 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -9,6 +9,7 @@ import ( "net/http" "net/http/httputil" "net/url" + "path" "strconv" "strings" "time" @@ -19,7 +20,8 @@ import ( const retryAfter = "5" type Config struct { - Timeout time.Duration + Timeout time.Duration + External *url.URL } type Server struct { @@ -73,7 +75,7 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { return } - info["webSocketDebuggerUrl"] = publicURL(r, b.WSPath) + info["webSocketDebuggerUrl"] = s.publicURL(r, b.WSPath, r.URL.RawQuery) s.sessions.Reserve(b, r.URL.RawQuery) w.Header().Set("Content-Type", "application/json") @@ -204,7 +206,17 @@ func fetchVersion(b *browser.Browser) (map[string]any, error) { return info, nil } -func publicURL(r *http.Request, wsPath string) string { +func (s *Server) publicURL(r *http.Request, wsPath, query string) string { + if external := s.config.External; external != nil { + scheme := "ws" + if external.Scheme == "https" || external.Scheme == "wss" { + scheme = "wss" + } + + u := url.URL{Scheme: scheme, Host: external.Host, Path: path.Join("/", external.Path, wsPath), RawQuery: query} + return u.String() + } + scheme := "ws" if r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") { scheme = "wss" @@ -215,7 +227,7 @@ func publicURL(r *http.Request, wsPath string) string { host = r.Host } - u := url.URL{Scheme: scheme, Host: host, Path: wsPath, RawQuery: r.URL.RawQuery} + u := url.URL{Scheme: scheme, Host: host, Path: wsPath, RawQuery: query} return u.String() } diff --git a/docker/main.go b/docker/main.go index 4d1ddc9a..6bd2fbc3 100644 --- a/docker/main.go +++ b/docker/main.go @@ -36,6 +36,8 @@ Environment: CONCURRENT running sessions at most (default: number of CPUs) QUEUED connections waiting for a free session, the rest get 429 (default 10) TIMEOUT session timeout in milliseconds (default 0, no timeout) + EXTERNAL public URL of the driver behind a proxy, e.g. https://chrome.example.com + (PROXY_URL works too) Endpoints: GET /json/version start a Chrome, return its webSocketDebuggerUrl @@ -89,7 +91,7 @@ func main() { }) srv := &http.Server{ Addr: config.Address, - Handler: server.New(sessions, server.Config{Timeout: config.Timeout}), + Handler: server.New(sessions, server.Config{Timeout: config.Timeout, External: config.External}), ReadHeaderTimeout: 10 * time.Second, } From 19253a18e7e2bba552ca92f98afa5277d9937fd6 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:38:45 +0300 Subject: [PATCH 10/23] fix(docker): fall back to a pending browser when the claimed id was taken by another connection With ws_url:, Ferrum calls /json/version and then connects to /, which claims any pending Chrome with the same query. With parallel clients that can be the Chrome another client's /json/version started, and that client then got 404 on /devtools/browser/. Chromes started with the same query are interchangeable, so the id route now falls back the same way. --- docker/internal/server/server.go | 19 ++----------------- docker/internal/server/sessions.go | 6 +++++- 2 files changed, 7 insertions(+), 18 deletions(-) diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index 6167a70f..188e642a 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -35,7 +35,7 @@ func New(sessions *Sessions, config Config) http.Handler { mux := http.NewServeMux() handle(mux, "/health", s.health) handle(mux, "/json/version", s.version) - handle(mux, "/devtools/browser/{id}", s.claim) + handle(mux, "/devtools/browser/{id}", s.connect) handle(mux, "/chrome", s.connect) handle(mux, "/chromium", s.connect) mux.HandleFunc("GET /{$}", s.connect) @@ -82,21 +82,6 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { json.NewEncoder(w).Encode(info) } -func (s *Server) claim(w http.ResponseWriter, r *http.Request) { - if !isUpgrade(r) { - http.Error(w, "websocket upgrade required", http.StatusUpgradeRequired) - return - } - - b := s.sessions.Claim(r.PathValue("id")) - if b == nil { - http.Error(w, "browser not found", http.StatusNotFound) - return - } - - s.serve(w, r, b, s.config.Timeout) -} - func (s *Server) connect(w http.ResponseWriter, r *http.Request) { if !isUpgrade(r) { http.Error(w, "websocket upgrade required", http.StatusUpgradeRequired) @@ -112,7 +97,7 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { return } - b, err := s.sessions.ClaimOrStart(r.Context(), r.URL.RawQuery, options) + b, err := s.sessions.ClaimOrStart(r.Context(), r.PathValue("id"), r.URL.RawQuery, options) if err != nil { startError(w, err) return diff --git a/docker/internal/server/sessions.go b/docker/internal/server/sessions.go index 91f631fb..af85aa44 100644 --- a/docker/internal/server/sessions.go +++ b/docker/internal/server/sessions.go @@ -112,7 +112,11 @@ func (s *Sessions) Claim(id string) *browser.Browser { return r.browser } -func (s *Sessions) ClaimOrStart(ctx context.Context, query string, options browser.Options) (*browser.Browser, error) { +func (s *Sessions) ClaimOrStart(ctx context.Context, id, query string, options browser.Options) (*browser.Browser, error) { + if b := s.Claim(id); b != nil { + return b, nil + } + s.mu.Lock() for id, r := range s.pending { if r.query == query { From f3ac28b095d6318046fc99109e1949044113e342 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:38:59 +0300 Subject: [PATCH 11/23] fix(docker): time out requests to Chrome after 10 seconds instead of hanging on a stuck browser --- docker/internal/server/server.go | 37 ++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index 188e642a..a03cfea8 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -19,6 +19,8 @@ import ( const retryAfter = "5" +var requestTimeout = 10 * time.Second + type Config struct { Timeout time.Duration External *url.URL @@ -27,10 +29,15 @@ type Config struct { type Server struct { sessions *Sessions config Config + client *http.Client } func New(sessions *Sessions, config Config) http.Handler { - s := &Server{sessions: sessions, config: config} + s := &Server{ + sessions: sessions, + config: config, + client: &http.Client{Timeout: requestTimeout}, + } mux := http.NewServeMux() handle(mux, "/health", s.health) @@ -67,8 +74,8 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { return } - info, err := fetchVersion(b) - if err != nil { + var info map[string]any + if err := s.getJSON(b, "/json/version", &info); err != nil { slog.Error("cannot fetch browser version", "id", b.ID, "error", err) s.sessions.Stop(b, "version failed") http.Error(w, "cannot fetch browser version", http.StatusBadGateway) @@ -172,23 +179,31 @@ func launchOptions(w http.ResponseWriter, r *http.Request) (browser.Options, boo return options, true } -func fetchVersion(b *browser.Browser) (map[string]any, error) { - resp, err := http.Get("http://" + b.Addr + "/json/version") +func (s *Server) getJSON(b *browser.Browser, path string, v any) error { + req, err := http.NewRequest(http.MethodGet, "http://"+b.Addr+path, nil) + if err != nil { + return err + } + + return s.doJSON(req, v) +} + +func (s *Server) doJSON(req *http.Request, v any) error { + resp, err := s.client.Do(req) if err != nil { - return nil, err + return err } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("unexpected status %s", resp.Status) + return fmt.Errorf("unexpected status %s", resp.Status) } - var info map[string]any - if err := json.NewDecoder(resp.Body).Decode(&info); err != nil { - return nil, fmt.Errorf("decode: %w", err) + if err := json.NewDecoder(resp.Body).Decode(v); err != nil { + return fmt.Errorf("decode: %w", err) } - return info, nil + return nil } func (s *Server) publicURL(r *http.Request, wsPath, query string) string { From c0731394f47078cb99087597de1e43c36a3c181a Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:39:18 +0300 Subject: [PATCH 12/23] feat(docker): serve the DevTools routes with target URLs pointing at the driver --- docker/internal/server/server.go | 282 ++++++++++++++++++++++++------- docker/main.go | 5 + 2 files changed, 223 insertions(+), 64 deletions(-) diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index a03cfea8..3fa54704 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "fmt" + "io" "log/slog" "net/http" "net/http/httputil" @@ -12,6 +13,7 @@ import ( "path" "strconv" "strings" + "sync" "time" "github.com/rubycdp/ferrum/docker/internal/browser" @@ -30,6 +32,9 @@ type Server struct { sessions *Sessions config Config client *http.Client + + protocolMu sync.Mutex + protocol []byte } func New(sessions *Sessions, config Config) http.Handler { @@ -40,19 +45,26 @@ func New(sessions *Sessions, config Config) http.Handler { } mux := http.NewServeMux() - handle(mux, "/health", s.health) - handle(mux, "/json/version", s.version) - handle(mux, "/devtools/browser/{id}", s.connect) - handle(mux, "/chrome", s.connect) - handle(mux, "/chromium", s.connect) + handle(mux, "GET", "/health", s.health) + handle(mux, "GET", "/json/version", s.version) + handle(mux, "GET", "/json", s.list) + handle(mux, "GET", "/json/list", s.list) + handle(mux, "GET", "/json/protocol", s.jsonProtocol) + handle(mux, "PUT", "/json/new", s.newTarget) + handle(mux, "GET", "/json/new", s.newTarget) + handle(mux, "GET", "/devtools/browser/{id}", s.connect) + handle(mux, "GET", "/devtools/page/{id}", s.page) + handle(mux, "GET", "/chrome", s.connect) + handle(mux, "GET", "/chromium", s.connect) + mux.HandleFunc("GET /devtools/{path...}", s.frontend) mux.HandleFunc("GET /{$}", s.connect) return mux } -func handle(mux *http.ServeMux, path string, handler http.HandlerFunc) { - mux.HandleFunc("GET "+path, handler) - mux.HandleFunc("GET "+path+"/{$}", handler) +func handle(mux *http.ServeMux, method, path string, handler http.HandlerFunc) { + mux.HandleFunc(method+" "+path, handler) + mux.HandleFunc(method+" "+path+"/{$}", handler) } func (s *Server) health(w http.ResponseWriter, _ *http.Request) { @@ -70,7 +82,7 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { b, err := s.sessions.Start(r.Context(), options) if err != nil { - startError(w, err) + s.startError(w, err) return } @@ -85,8 +97,7 @@ func (s *Server) version(w http.ResponseWriter, r *http.Request) { info["webSocketDebuggerUrl"] = s.publicURL(r, b.WSPath, r.URL.RawQuery) s.sessions.Reserve(b, r.URL.RawQuery) - w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode(info) + writeJSON(w, info) } func (s *Server) connect(w http.ResponseWriter, r *http.Request) { @@ -106,40 +117,126 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { b, err := s.sessions.ClaimOrStart(r.Context(), r.PathValue("id"), r.URL.RawQuery, options) if err != nil { - startError(w, err) + s.startError(w, err) return } - s.serve(w, r, b, timeout) -} - -func (s *Server) serve(w http.ResponseWriter, r *http.Request, b *browser.Browser, timeout time.Duration) { if timeout > 0 { timer := time.AfterFunc(timeout, func() { s.sessions.Stop(b, "timeout") }) defer timer.Stop() } slog.Info("session opened", "id", b.ID, "client", r.RemoteAddr, "timeout", timeout) - proxyTo(b).ServeHTTP(w, r) + proxy(b, b.WSPath, "").ServeHTTP(w, r) s.sessions.Stop(b, "connection closed") } -func (s *Server) sessionTimeout(w http.ResponseWriter, r *http.Request) (time.Duration, bool) { - value := r.URL.Query().Get("timeout") - if value == "" { - return s.config.Timeout, true +func (s *Server) page(w http.ResponseWriter, r *http.Request) { + if !isUpgrade(r) { + http.Error(w, "websocket upgrade required", http.StatusUpgradeRequired) + return } - ms, err := strconv.Atoi(value) + id := r.PathValue("id") + for _, b := range s.sessions.Browsers() { + targets, err := s.targets(b) + if err != nil { + continue + } + + for _, target := range targets { + if target["id"] == id { + proxy(b, "/devtools/page/"+id, "").ServeHTTP(w, r) + return + } + } + } + + http.Error(w, "target not found", http.StatusNotFound) +} + +func (s *Server) list(w http.ResponseWriter, r *http.Request) { + all := []map[string]any{} + for _, b := range s.sessions.Browsers() { + targets, err := s.targets(b) + if err != nil { + slog.Warn("cannot list browser targets", "id", b.ID, "error", err) + continue + } + + for _, target := range targets { + all = append(all, s.publicTarget(r, b, target)) + } + } + + writeJSON(w, all) +} + +func (s *Server) newTarget(w http.ResponseWriter, r *http.Request) { + browsers := s.sessions.Browsers() + if len(browsers) != 1 { + http.Error(w, fmt.Sprintf("/json/new needs exactly one running session, there are %d", len(browsers)), http.StatusConflict) + return + } + + b := browsers[0] + req, err := http.NewRequestWithContext(r.Context(), http.MethodPut, "http://"+b.Addr+"/json/new?"+r.URL.RawQuery, nil) if err != nil { - http.Error(w, fmt.Sprintf("invalid timeout %q", value), http.StatusBadRequest) - return 0, false + http.Error(w, err.Error(), http.StatusBadRequest) + return } - return time.Duration(max(ms, 0)) * time.Millisecond, true + var target map[string]any + if err := s.doJSON(req, &target); err != nil { + http.Error(w, "cannot create target", http.StatusBadGateway) + return + } + + writeJSON(w, s.publicTarget(r, b, target)) } -func startError(w http.ResponseWriter, err error) { +func (s *Server) jsonProtocol(w http.ResponseWriter, r *http.Request) { + s.protocolMu.Lock() + defer s.protocolMu.Unlock() + + if s.protocol == nil { + protocol, err := s.fetchProtocol(r.Context()) + if err != nil { + s.startError(w, err) + return + } + s.protocol = protocol + } + + w.Header().Set("Content-Type", "application/json") + w.Write(s.protocol) +} + +func (s *Server) frontend(w http.ResponseWriter, r *http.Request) { + browsers := s.sessions.Browsers() + if len(browsers) == 0 { + http.Error(w, "no running session", http.StatusServiceUnavailable) + return + } + + proxy(browsers[0], r.URL.Path, r.URL.RawQuery).ServeHTTP(w, r) +} + +func (s *Server) fetchProtocol(ctx context.Context) ([]byte, error) { + if browsers := s.sessions.Browsers(); len(browsers) > 0 { + return s.get(browsers[0], "/json/protocol") + } + + b, err := s.sessions.Start(ctx, browser.Options{}) + if err != nil { + return nil, err + } + defer s.sessions.Stop(b, "protocol fetched") + + return s.get(b, "/json/protocol") +} + +func (s *Server) startError(w http.ResponseWriter, err error) { switch { case errors.Is(err, ErrQueueFull): w.Header().Set("Retry-After", retryAfter) @@ -151,32 +248,81 @@ func startError(w http.ResponseWriter, err error) { } } -func proxyTo(b *browser.Browser) *httputil.ReverseProxy { - return &httputil.ReverseProxy{ - Rewrite: func(pr *httputil.ProxyRequest) { - pr.Out.URL.Scheme = "http" - pr.Out.URL.Host = b.Addr - pr.Out.URL.Path = b.WSPath - pr.Out.URL.RawPath = "" - pr.Out.URL.RawQuery = "" - pr.Out.Host = b.Addr - pr.Out.Header.Del("Origin") - }, - ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) { - slog.Error("cannot proxy to browser", "id", b.ID, "error", err) - http.Error(w, "browser unavailable", http.StatusBadGateway) - }, +func (s *Server) sessionTimeout(w http.ResponseWriter, r *http.Request) (time.Duration, bool) { + value := r.URL.Query().Get("timeout") + if value == "" { + return s.config.Timeout, true + } + + ms, err := strconv.Atoi(value) + if err != nil { + http.Error(w, fmt.Sprintf("invalid timeout %q", value), http.StatusBadRequest) + return 0, false } + + return time.Duration(max(ms, 0)) * time.Millisecond, true } -func launchOptions(w http.ResponseWriter, r *http.Request) (browser.Options, bool) { - options, err := browser.ParseOptions(r.URL.Query().Get("launch")) +func (s *Server) targets(b *browser.Browser) ([]map[string]any, error) { + var targets []map[string]any + return targets, s.getJSON(b, "/json/list", &targets) +} + +func (s *Server) publicTarget(r *http.Request, b *browser.Browser, target map[string]any) map[string]any { + id, _ := target["id"].(string) + wsURL := s.publicURL(r, "/devtools/page/"+id, "") + target["webSocketDebuggerUrl"] = wsURL + + if frontend, ok := target["devtoolsFrontendUrl"].(string); ok { + u, _ := url.Parse(wsURL) + param := "ws=" + if u.Scheme == "wss" { + param = "wss=" + } + frontend = strings.Replace(frontend, "ws="+b.Addr, param+u.Host+strings.TrimSuffix(u.Path, "/devtools/page/"+id), 1) + target["devtoolsFrontendUrl"] = frontend + } + + return target +} + +func (s *Server) publicURL(r *http.Request, wsPath, query string) string { + if external := s.config.External; external != nil { + scheme := "ws" + if external.Scheme == "https" || external.Scheme == "wss" { + scheme = "wss" + } + + u := url.URL{Scheme: scheme, Host: external.Host, Path: path.Join("/", external.Path, wsPath), RawQuery: query} + return u.String() + } + + scheme := "ws" + if r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") { + scheme = "wss" + } + + host := r.Header.Get("X-Forwarded-Host") + if host == "" { + host = r.Host + } + + u := url.URL{Scheme: scheme, Host: host, Path: wsPath, RawQuery: query} + return u.String() +} + +func (s *Server) get(b *browser.Browser, path string) ([]byte, error) { + resp, err := s.client.Get("http://" + b.Addr + path) if err != nil { - http.Error(w, err.Error(), http.StatusBadRequest) - return options, false + return nil, err } + defer resp.Body.Close() - return options, true + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("unexpected status %s", resp.Status) + } + + return io.ReadAll(resp.Body) } func (s *Server) getJSON(b *browser.Browser, path string, v any) error { @@ -206,29 +352,37 @@ func (s *Server) doJSON(req *http.Request, v any) error { return nil } -func (s *Server) publicURL(r *http.Request, wsPath, query string) string { - if external := s.config.External; external != nil { - scheme := "ws" - if external.Scheme == "https" || external.Scheme == "wss" { - scheme = "wss" - } - - u := url.URL{Scheme: scheme, Host: external.Host, Path: path.Join("/", external.Path, wsPath), RawQuery: query} - return u.String() +func proxy(b *browser.Browser, path, query string) *httputil.ReverseProxy { + return &httputil.ReverseProxy{ + Rewrite: func(pr *httputil.ProxyRequest) { + pr.Out.URL.Scheme = "http" + pr.Out.URL.Host = b.Addr + pr.Out.URL.Path = path + pr.Out.URL.RawPath = "" + pr.Out.URL.RawQuery = query + pr.Out.Host = b.Addr + pr.Out.Header.Del("Origin") + }, + ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) { + slog.Error("cannot proxy to browser", "id", b.ID, "error", err) + http.Error(w, "browser unavailable", http.StatusBadGateway) + }, } +} - scheme := "ws" - if r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") { - scheme = "wss" +func launchOptions(w http.ResponseWriter, r *http.Request) (browser.Options, bool) { + options, err := browser.ParseOptions(r.URL.Query().Get("launch")) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return options, false } - host := r.Header.Get("X-Forwarded-Host") - if host == "" { - host = r.Host - } + return options, true +} - u := url.URL{Scheme: scheme, Host: host, Path: wsPath, RawQuery: query} - return u.String() +func writeJSON(w http.ResponseWriter, v any) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(v) } func isUpgrade(r *http.Request) bool { diff --git a/docker/main.go b/docker/main.go index 6bd2fbc3..b307a0ba 100644 --- a/docker/main.go +++ b/docker/main.go @@ -43,6 +43,11 @@ Endpoints: GET /json/version start a Chrome, return its webSocketDebuggerUrl WS /devtools/browser/ connect to the Chrome started by /json/version WS /, /chrome, /chromium connect to a new Chrome + GET /json, /json/list targets of every running session + PUT /json/new? open a target, when exactly one session runs + GET /json/protocol the CDP protocol description + WS /devtools/page/ connect to one target, e.g. from DevTools + GET /devtools/* the DevTools frontend of a running session GET /health liveness, never starts a Chrome Launch options (per connection, in the URL of /json/version or the WebSocket): From 4900f64c8322c4de43d44a69e5aedeb1af3f92ab Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:39:33 +0300 Subject: [PATCH 13/23] feat(docker): run Chrome with a window on its own Xvfb display when launch.headless is false --- docker/Dockerfile | 2 +- docker/internal/browser/browser.go | 34 ++++++++-- docker/internal/browser/flags.go | 26 +++++++- docker/internal/browser/options.go | 6 ++ docker/internal/browser/options_test.go | 35 ++++++++++ docker/internal/browser/xvfb.go | 87 +++++++++++++++++++++++++ docker/main.go | 1 + 7 files changed, 182 insertions(+), 9 deletions(-) create mode 100644 docker/internal/browser/xvfb.go diff --git a/docker/Dockerfile b/docker/Dockerfile index e4e67741..0bbe76aa 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -71,7 +71,7 @@ ENV BROWSER_PATH=/usr/bin/google-chrome FROM ${BROWSER} -RUN mkdir -p /home/driver && chmod 1777 /home/driver \ +RUN mkdir -p /home/driver /tmp/.X11-unix && chmod 1777 /home/driver /tmp/.X11-unix \ && useradd --uid 999 --home-dir /home/driver --no-create-home --shell /usr/sbin/nologin driver ENV HOME=/home/driver \ diff --git a/docker/internal/browser/browser.go b/docker/internal/browser/browser.go index 6214d05c..8e1435a3 100644 --- a/docker/internal/browser/browser.go +++ b/docker/internal/browser/browser.go @@ -26,6 +26,7 @@ type Browser struct { WSPath string cmd *exec.Cmd + xvfb *xvfb dir string done chan struct{} stopOnce sync.Once @@ -37,25 +38,35 @@ func Start(binary string, options Options, timeout time.Duration) (*Browser, err return nil, fmt.Errorf("create user data dir: %w", err) } + flags := Flags(dir, options) + cmd := exec.Command(binary, flags...) + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + + var x *xvfb + if needsXvfb(options) { + if x, err = startXvfb(windowSize(flags)); err != nil { + os.RemoveAll(dir) + return nil, err + } + cmd.Env = append(os.Environ(), "DISPLAY="+x.display) + } + stderr, writer, err := os.Pipe() if err != nil { - os.RemoveAll(dir) + cleanup(x, dir) return nil, fmt.Errorf("create stderr pipe: %w", err) } - - cmd := exec.Command(binary, Flags(dir, options)...) cmd.Stderr = writer - cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} err = cmd.Start() writer.Close() if err != nil { stderr.Close() - os.RemoveAll(dir) + cleanup(x, dir) return nil, fmt.Errorf("start browser: %w", err) } - b := &Browser{cmd: cmd, dir: dir, done: make(chan struct{})} + b := &Browser{cmd: cmd, xvfb: x, dir: dir, done: make(chan struct{})} go b.wait() wsURL := make(chan string, 1) @@ -100,6 +111,10 @@ func (b *Browser) Stop() { syscall.Kill(-pgid, syscall.SIGKILL) <-b.done + if b.xvfb != nil { + b.xvfb.stop() + } + if err := os.RemoveAll(b.dir); err != nil { slog.Warn("cannot remove user data dir", "dir", b.dir, "error", err) } @@ -108,6 +123,13 @@ func (b *Browser) Stop() { }) } +func cleanup(x *xvfb, dir string) { + if x != nil { + x.stop() + } + os.RemoveAll(dir) +} + func (b *Browser) wait() { err := b.cmd.Wait() slog.Debug("browser exited", "pid", b.Pid(), "error", err) diff --git a/docker/internal/browser/flags.go b/docker/internal/browser/flags.go index 112119f9..0a11d2a2 100644 --- a/docker/internal/browser/flags.go +++ b/docker/internal/browser/flags.go @@ -3,11 +3,13 @@ package browser import ( "fmt" "slices" + "strconv" "strings" ) const ( chromeDebuggingPortFlag = "--remote-debugging-port" + chromeHeadlessFlag = "--headless" chromeWindowSizeFlag = "--window-size" chromeUserDataDirFlag = "--user-data-dir" defaultWidth = 1024 @@ -46,7 +48,7 @@ var ( "--enable-automation", "--enable-features=NetworkService,NetworkServiceInProcess", "--force-color-profile=srgb", - "--headless", + chromeHeadlessFlag, "--hide-scrollbars", "--keep-alive-for-test", "--metrics-recording-only", @@ -71,7 +73,9 @@ func Defaults() []string { } func Flags(dir string, options Options) []string { - f := slices.DeleteFunc(slices.Clone(flags), options.IgnoreDefaultArgs.ignores) + f := slices.DeleteFunc(slices.Clone(flags), func(flag string) bool { + return options.IgnoreDefaultArgs.ignores(flag) || (options.Headful() && flagName(flag) == chromeHeadlessFlag) + }) extra := slices.Clone(options.Args) if v := options.DefaultViewport; v != nil { @@ -97,6 +101,24 @@ func Flags(dir string, options Options) []string { ) } +func windowSize(flags []string) (int, int) { + for _, flag := range slices.Backward(flags) { + name, value, _ := strings.Cut(flag, "=") + if name != chromeWindowSizeFlag { + continue + } + + w, h, _ := strings.Cut(value, ",") + width, errW := strconv.Atoi(strings.TrimSpace(w)) + height, errH := strconv.Atoi(strings.TrimSpace(h)) + if errW == nil && errH == nil && width > 0 && height > 0 { + return width, height + } + } + + return defaultWidth, defaultHeight +} + func flagName(flag string) string { name, _, _ := strings.Cut(flag, "=") return name diff --git a/docker/internal/browser/options.go b/docker/internal/browser/options.go index 1d53677a..a69bb46b 100644 --- a/docker/internal/browser/options.go +++ b/docker/internal/browser/options.go @@ -11,6 +11,12 @@ type Options struct { Args []string `json:"args"` DefaultViewport *Viewport `json:"defaultViewport"` IgnoreDefaultArgs IgnoredArgs `json:"ignoreDefaultArgs"` + Headless any `json:"headless"` +} + +func (o Options) Headful() bool { + headless, ok := o.Headless.(bool) + return ok && !headless } type Viewport struct { diff --git a/docker/internal/browser/options_test.go b/docker/internal/browser/options_test.go index 5be6914d..cbb27557 100644 --- a/docker/internal/browser/options_test.go +++ b/docker/internal/browser/options_test.go @@ -114,3 +114,38 @@ func TestFlags(t *testing.T) { } }) } + +func TestHeadful(t *testing.T) { + for launch, headful := range map[string]bool{ + `{}`: false, + `{"headless":true}`: false, + `{"headless":"new"}`: false, + `{"headless":"shell"}`: false, + `{"headless":false}`: true, + } { + options, err := ParseOptions(launch) + if err != nil { + t.Fatal(err) + } + + if options.Headful() != headful { + t.Errorf("%s: headful = %v, want %v", launch, options.Headful(), headful) + } + + if f := Flags("/tmp/profile", options); slices.Contains(f, "--headless") == headful { + t.Errorf("%s: flags = %v", launch, f) + } + } +} + +func TestWindowSize(t *testing.T) { + if w, h := windowSize(Flags("/tmp/profile", Options{})); w != 1024 || h != 768 { + t.Errorf("default = %dx%d", w, h) + } + if w, h := windowSize(Flags("/tmp/profile", Options{DefaultViewport: &Viewport{Width: 800, Height: 600}})); w != 800 || h != 600 { + t.Errorf("viewport = %dx%d", w, h) + } + if w, h := windowSize([]string{"--window-size=bogus"}); w != 1024 || h != 768 { + t.Errorf("invalid = %dx%d", w, h) + } +} diff --git a/docker/internal/browser/xvfb.go b/docker/internal/browser/xvfb.go new file mode 100644 index 00000000..7e381a69 --- /dev/null +++ b/docker/internal/browser/xvfb.go @@ -0,0 +1,87 @@ +package browser + +import ( + "bufio" + "errors" + "fmt" + "os" + "os/exec" + "strings" + "syscall" + "time" +) + +const ( + xvfbBinary = "Xvfb" + xvfbTimeout = 10 * time.Second +) + +type xvfb struct { + cmd *exec.Cmd + display string + done chan struct{} +} + +func needsXvfb(options Options) bool { + if !options.Headful() || os.Getenv("DISPLAY") != "" { + return false + } + + _, err := exec.LookPath(xvfbBinary) + return err == nil +} + +func startXvfb(width, height int) (*xvfb, error) { + reader, writer, err := os.Pipe() + if err != nil { + return nil, fmt.Errorf("create xvfb display pipe: %w", err) + } + defer reader.Close() + + cmd := exec.Command(xvfbBinary, "-displayfd", "3", "-screen", "0", fmt.Sprintf("%dx%dx24", width, height), "-nolisten", "tcp") + cmd.ExtraFiles = []*os.File{writer} + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + + err = cmd.Start() + writer.Close() + if err != nil { + return nil, fmt.Errorf("start xvfb: %w", err) + } + + x := &xvfb{cmd: cmd, done: make(chan struct{})} + go func() { + cmd.Wait() + close(x.done) + }() + + display := make(chan string, 1) + go func() { + line, _ := bufio.NewReader(reader).ReadString('\n') + display <- strings.TrimSpace(line) + }() + + select { + case d := <-display: + if d == "" { + x.stop() + return nil, errors.New("xvfb exited before it was ready") + } + x.display = ":" + d + return x, nil + case <-time.After(xvfbTimeout): + x.stop() + return nil, fmt.Errorf("xvfb did not start within %s", xvfbTimeout) + } +} + +func (x *xvfb) stop() { + pgid := x.cmd.Process.Pid + syscall.Kill(-pgid, syscall.SIGTERM) + + select { + case <-x.done: + case <-time.After(killTimeout): + syscall.Kill(-pgid, syscall.SIGKILL) + <-x.done + } +} diff --git a/docker/main.go b/docker/main.go index b307a0ba..c071d762 100644 --- a/docker/main.go +++ b/docker/main.go @@ -55,6 +55,7 @@ Launch options (per connection, in the URL of /json/version or the WebSocket): args extra Chrome flags, a flag with a default's name replaces it defaultViewport {"width":1280,"height":720}, becomes --window-size ignoreDefaultArgs true, or a list of default flags to drop + headless false runs Chrome with a window, under its own Xvfb timeout= session timeout for this connection, overrides TIMEOUT Other launch keys and query params (token, blockAds) are ignored. --remote-debugging-port and --user-data-dir are managed by the driver. From 9a60a3a7734a8a28aa8faaae1b3c2b9b595c714b Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 09:39:45 +0300 Subject: [PATCH 14/23] test(docker): cover config, sessions and routes, with the test binary acting as a fake Chrome --- docker/internal/config/config_test.go | 63 ++++ docker/internal/server/main_test.go | 170 ++++++++++ docker/internal/server/server_test.go | 393 ++++++++++++++++++++++++ docker/internal/server/sessions_test.go | 227 ++++++++++++++ 4 files changed, 853 insertions(+) create mode 100644 docker/internal/config/config_test.go create mode 100644 docker/internal/server/main_test.go create mode 100644 docker/internal/server/server_test.go create mode 100644 docker/internal/server/sessions_test.go diff --git a/docker/internal/config/config_test.go b/docker/internal/config/config_test.go new file mode 100644 index 00000000..8ca45bde --- /dev/null +++ b/docker/internal/config/config_test.go @@ -0,0 +1,63 @@ +package config + +import ( + "runtime" + "testing" + "time" +) + +func TestLoadDefaults(t *testing.T) { + t.Setenv("BROWSER_PATH", "/usr/bin/chromium") + for _, key := range []string{"HOST", "PORT", "CONCURRENT", "QUEUED", "TIMEOUT", "EXTERNAL", "PROXY_URL"} { + t.Setenv(key, "") + } + + if err := Load(); err != nil { + t.Fatal(err) + } + + if Address != "0.0.0.0:3000" || Concurrent != runtime.NumCPU() || Queued != 10 || Timeout != 0 || External != nil { + t.Fatalf("address=%s concurrent=%d queued=%d timeout=%s external=%v", Address, Concurrent, Queued, Timeout, External) + } +} + +func TestLoad(t *testing.T) { + t.Setenv("BROWSER_PATH", "/usr/bin/chromium") + t.Setenv("HOST", "127.0.0.1") + t.Setenv("PORT", "9222") + t.Setenv("CONCURRENT", "2") + t.Setenv("QUEUED", "0") + t.Setenv("TIMEOUT", "60000") + t.Setenv("EXTERNAL", "") + t.Setenv("PROXY_URL", "https://chrome.example.com") + + if err := Load(); err != nil { + t.Fatal(err) + } + + if Address != "127.0.0.1:9222" || Concurrent != 2 || Queued != 0 || Timeout != time.Minute || External.Host != "chrome.example.com" { + t.Fatalf("address=%s concurrent=%d queued=%d timeout=%s external=%v", Address, Concurrent, Queued, Timeout, External) + } +} + +func TestLoadErrors(t *testing.T) { + for name, env := range map[string]map[string]string{ + "no browser": {"BROWSER_PATH": ""}, + "concurrent text": {"CONCURRENT": "many"}, + "concurrent zero": {"CONCURRENT": "0"}, + "queued negative": {"QUEUED": "-1"}, + "timeout text": {"TIMEOUT": "30s"}, + "external relative": {"EXTERNAL": "chrome.example.com"}, + } { + t.Run(name, func(t *testing.T) { + t.Setenv("BROWSER_PATH", "/usr/bin/chromium") + for key, value := range env { + t.Setenv(key, value) + } + + if err := Load(); err == nil { + t.Error("expected an error") + } + }) + } +} diff --git a/docker/internal/server/main_test.go b/docker/internal/server/main_test.go new file mode 100644 index 00000000..7581e2b9 --- /dev/null +++ b/docker/internal/server/main_test.go @@ -0,0 +1,170 @@ +package server + +import ( + "bufio" + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "os" + "slices" + "strings" + "testing" + "time" +) + +const fakeBrowserEnv = "FAKE_BROWSER" + +func TestMain(m *testing.M) { + if os.Getenv(fakeBrowserEnv) == "1" { + fakeBrowser(os.Args[1:]) + return + } + + os.Setenv(fakeBrowserEnv, "1") + os.Exit(m.Run()) +} + +func fakeBrowser(args []string) { + if slices.Contains(args, "--fake-exit") { + os.Exit(1) + } + + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + os.Exit(1) + } + + addr := listener.Addr().String() + id := randomID() + pageID := "page-" + id + + mux := http.NewServeMux() + mux.HandleFunc("GET /json/version", func(w http.ResponseWriter, _ *http.Request) { + if slices.Contains(args, "--fake-slow-version") { + time.Sleep(time.Minute) + } + json.NewEncoder(w).Encode(map[string]any{ + "Browser": "Fake/1.0", + "Args": args, + "webSocketDebuggerUrl": "ws://" + addr + "/devtools/browser/" + id, + }) + }) + mux.HandleFunc("GET /json/list", func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode([]map[string]any{{ + "id": pageID, + "type": "page", + "webSocketDebuggerUrl": "ws://" + addr + "/devtools/page/" + pageID, + "devtoolsFrontendUrl": "/devtools/inspector.html?ws=" + addr + "/devtools/page/" + pageID, + }}) + }) + mux.HandleFunc("GET /json/protocol", func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte(`{"version":{"major":"1","minor":"3"}}`)) + }) + mux.HandleFunc("PUT /json/new", func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(map[string]any{ + "id": "new-" + id, + "url": r.URL.RawQuery, + "webSocketDebuggerUrl": "ws://" + addr + "/devtools/page/new-" + id, + "devtoolsFrontendUrl": "/devtools/inspector.html?ws=" + addr + "/devtools/page/new-" + id, + }) + }) + mux.HandleFunc("GET /devtools/inspector.html", func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("inspector " + id)) + }) + mux.HandleFunc("GET /devtools/browser/"+id, echo) + mux.HandleFunc("GET /devtools/page/"+pageID, echo) + + fmt.Fprintf(os.Stderr, "starting\nDevTools listening on ws://%s/devtools/browser/%s\n", addr, id) + http.Serve(listener, mux) +} + +func echo(w http.ResponseWriter, r *http.Request) { + if !isUpgrade(r) { + http.Error(w, "upgrade required", http.StatusBadRequest) + return + } + + conn, buf, err := w.(http.Hijacker).Hijack() + if err != nil { + return + } + defer conn.Close() + + buf.WriteString("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\r\n") + buf.Flush() + io.Copy(conn, buf) +} + +func randomID() string { + b := make([]byte, 8) + rand.Read(b) + return hex.EncodeToString(b) +} + +type wsConn struct { + net.Conn + reader *bufio.Reader +} + +func dial(t *testing.T, addr, path string) (*wsConn, *http.Response) { + t.Helper() + + conn, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { conn.Close() }) + + fmt.Fprintf(conn, "GET %s HTTP/1.1\r\nHost: %s\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n"+ + "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\n\r\n", path, addr) + + reader := bufio.NewReader(conn) + resp, err := http.ReadResponse(reader, nil) + if err != nil { + t.Fatal(err) + } + + return &wsConn{Conn: conn, reader: reader}, resp +} + +func (c *wsConn) roundTrip(t *testing.T, message string) { + t.Helper() + + c.SetDeadline(time.Now().Add(5 * time.Second)) + if _, err := io.WriteString(c, message); err != nil { + t.Fatal(err) + } + + got := make([]byte, len(message)) + if _, err := io.ReadFull(c.reader, got); err != nil { + t.Fatal(err) + } + if string(got) != message { + t.Fatalf("echo = %q, want %q", got, message) + } +} + +func (c *wsConn) waitClosed(t *testing.T) { + t.Helper() + + c.SetDeadline(time.Now().Add(5 * time.Second)) + if _, err := c.reader.ReadByte(); err == nil || strings.Contains(err.Error(), "timeout") { + t.Fatalf("connection still open: %v", err) + } +} + +func eventually(t *testing.T, condition func() bool) { + t.Helper() + + deadline := time.Now().Add(5 * time.Second) + for !condition() { + if time.Now().After(deadline) { + t.Fatal("condition not met in time") + } + time.Sleep(10 * time.Millisecond) + } +} diff --git a/docker/internal/server/server_test.go b/docker/internal/server/server_test.go new file mode 100644 index 00000000..1c3db89e --- /dev/null +++ b/docker/internal/server/server_test.go @@ -0,0 +1,393 @@ +package server + +import ( + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "strings" + "testing" + "time" +) + +type testServer struct { + *httptest.Server + sessions *Sessions +} + +func newServer(t *testing.T, sessions SessionsConfig, config Config) *testServer { + t.Helper() + + s := newSessions(t, sessions) + srv := httptest.NewServer(New(s, config)) + t.Cleanup(srv.Close) + + return &testServer{Server: srv, sessions: s} +} + +func (s *testServer) addr() string { + return strings.TrimPrefix(s.URL, "http://") +} + +func (s *testServer) get(t *testing.T, path string) *http.Response { + t.Helper() + + resp, err := http.Get(s.URL + path) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { resp.Body.Close() }) + + return resp +} + +func (s *testServer) version(t *testing.T, query string) map[string]any { + t.Helper() + + resp := s.get(t, "/json/version"+query) + if resp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(resp.Body) + t.Fatalf("status = %d: %s", resp.StatusCode, body) + } + + var info map[string]any + if err := json.NewDecoder(resp.Body).Decode(&info); err != nil { + t.Fatal(err) + } + + return info +} + +func wsPath(t *testing.T, raw any) string { + t.Helper() + + u, err := url.Parse(raw.(string)) + if err != nil { + t.Fatal(err) + } + + return u.RequestURI() +} + +func TestHealth(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + if resp := srv.get(t, "/health"); resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d", resp.StatusCode) + } + if len(srv.sessions.Browsers()) != 0 { + t.Fatal("health started a browser") + } +} + +func TestVersionAndClaim(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + info := srv.version(t, "?token=abc") + if info["Browser"] != "Fake/1.0" { + t.Errorf("Browser = %v", info["Browser"]) + } + + ws, _ := url.Parse(info["webSocketDebuggerUrl"].(string)) + if ws.Host != srv.addr() || ws.RawQuery != "token=abc" || !strings.HasPrefix(ws.Path, "/devtools/browser/") { + t.Fatalf("webSocketDebuggerUrl = %s", ws) + } + + conn, resp := dial(t, srv.addr(), ws.RequestURI()) + if resp.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("status = %d", resp.StatusCode) + } + conn.roundTrip(t, "hello") + + conn.Close() + eventually(t, func() bool { return len(srv.sessions.Browsers()) == 0 }) +} + +func TestConnectRoutes(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + for _, path := range []string{"/", "/chrome", "/chrome/", "/chromium", "/chromium/"} { + t.Run(path, func(t *testing.T) { + conn, resp := dial(t, srv.addr(), path) + if resp.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("status = %d", resp.StatusCode) + } + conn.roundTrip(t, "hello") + }) + } +} + +func TestConnectClaimsTheBrowserStartedByVersion(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + srv.version(t, "?launch=%7B%7D") + conn, _ := dial(t, srv.addr(), "/chrome?launch=%7B%7D") + conn.roundTrip(t, "hello") + + if n := len(srv.sessions.Browsers()); n != 1 { + t.Fatalf("browsers = %d, want 1", n) + } +} + +func TestClaimFallsBackWhenTheIdWasTaken(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + first := srv.version(t, "") + second := srv.version(t, "") + + anonymous, _ := dial(t, srv.addr(), "/") + anonymous.roundTrip(t, "anonymous") + + for _, info := range []map[string]any{first, second} { + conn, resp := dial(t, srv.addr(), wsPath(t, info["webSocketDebuggerUrl"])) + if resp.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("status = %d", resp.StatusCode) + } + conn.roundTrip(t, "hello") + } +} + +func TestBadRequests(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + for path, status := range map[string]int{ + "/json/version?launch=%7Bbroken": http.StatusBadRequest, + "/json/version?timeout=soon": http.StatusBadRequest, + "/chrome": http.StatusUpgradeRequired, + "/devtools/browser/missing": http.StatusUpgradeRequired, + "/devtools/page/missing": http.StatusUpgradeRequired, + "/unknown": http.StatusNotFound, + "/devtools/inspector.html": http.StatusServiceUnavailable, + "/json/new?https://example.com/": http.StatusConflict, + "/json/version/?launch=%7Bbroken": http.StatusBadRequest, + } { + if resp := srv.get(t, path); resp.StatusCode != status { + t.Errorf("%s: status = %d, want %d", path, resp.StatusCode, status) + } + } + + if n := len(srv.sessions.Browsers()); n != 0 { + t.Fatalf("bad requests started %d browsers", n) + } +} + +func TestLaunchOptionsReachTheBrowser(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + launch := url.QueryEscape(`{"args":["--accept-lang=de-DE","--user-data-dir=/elsewhere"],"defaultViewport":{"width":1280,"height":720},"ignoreDefaultArgs":["--mute-audio"]}`) + info := srv.version(t, "?launch="+launch) + + var args []string + for _, arg := range info["Args"].([]any) { + args = append(args, arg.(string)) + } + + for _, want := range []string{"--accept-lang=de-DE", "--window-size=1280,720", "--remote-debugging-port=0"} { + if !slices.Contains(args, want) { + t.Errorf("missing %s in %v", want, args) + } + } + for _, unwanted := range []string{"--mute-audio", "--window-size=1024,768", "--user-data-dir=/elsewhere"} { + if slices.Contains(args, unwanted) { + t.Errorf("unexpected %s in %v", unwanted, args) + } + } +} + +func TestQueueLimit(t *testing.T) { + srv := newServer(t, SessionsConfig{Concurrent: 1, Queued: 0}, Config{}) + + conn, _ := dial(t, srv.addr(), "/") + conn.roundTrip(t, "hello") + + resp := srv.get(t, "/json/version") + if resp.StatusCode != http.StatusTooManyRequests { + t.Fatalf("status = %d, want 429", resp.StatusCode) + } + if resp.Header.Get("Retry-After") == "" { + t.Error("missing Retry-After") + } + + _, resp = dial(t, srv.addr(), "/chrome") + if resp.StatusCode != http.StatusTooManyRequests { + t.Fatalf("websocket status = %d, want 429", resp.StatusCode) + } +} + +func TestQueuedConnectionWaitsForAFreeSession(t *testing.T) { + srv := newServer(t, SessionsConfig{Concurrent: 1, Queued: 1}, Config{}) + + first, _ := dial(t, srv.addr(), "/") + first.roundTrip(t, "first") + + time.AfterFunc(200*time.Millisecond, func() { first.Close() }) + + second, resp := dial(t, srv.addr(), "/") + if resp.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("status = %d", resp.StatusCode) + } + second.roundTrip(t, "second") +} + +func TestSessionTimeout(t *testing.T) { + t.Run("from the query", func(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + conn, _ := dial(t, srv.addr(), "/?timeout=100") + conn.roundTrip(t, "hello") + conn.waitClosed(t) + eventually(t, func() bool { return len(srv.sessions.Browsers()) == 0 }) + }) + + t.Run("from the config", func(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{Timeout: 100 * time.Millisecond}) + + conn, _ := dial(t, srv.addr(), "/") + conn.roundTrip(t, "hello") + conn.waitClosed(t) + }) + + t.Run("query 0 turns it off", func(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{Timeout: 100 * time.Millisecond}) + + conn, _ := dial(t, srv.addr(), "/?timeout=0") + time.Sleep(300 * time.Millisecond) + conn.roundTrip(t, "still here") + }) +} + +func TestPublicURL(t *testing.T) { + t.Run("forwarded headers", func(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + req, _ := http.NewRequest(http.MethodGet, srv.URL+"/json/version", nil) + req.Header.Set("X-Forwarded-Host", "chrome.example.com") + req.Header.Set("X-Forwarded-Proto", "https") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + + var info map[string]any + json.NewDecoder(resp.Body).Decode(&info) + if ws := info["webSocketDebuggerUrl"].(string); !strings.HasPrefix(ws, "wss://chrome.example.com/devtools/browser/") { + t.Fatalf("webSocketDebuggerUrl = %s", ws) + } + }) + + t.Run("external", func(t *testing.T) { + external, _ := url.Parse("https://proxy.example.com/chrome") + srv := newServer(t, SessionsConfig{}, Config{External: external}) + + info := srv.version(t, "?a=1") + ws := info["webSocketDebuggerUrl"].(string) + if !strings.HasPrefix(ws, "wss://proxy.example.com/chrome/devtools/browser/") || !strings.HasSuffix(ws, "?a=1") { + t.Fatalf("webSocketDebuggerUrl = %s", ws) + } + }) +} + +func TestVersionTimesOutOnAStuckBrowser(t *testing.T) { + defer func(timeout time.Duration) { requestTimeout = timeout }(requestTimeout) + requestTimeout = 100 * time.Millisecond + + srv := newServer(t, SessionsConfig{}, Config{}) + + launch := url.QueryEscape(`{"args":["--fake-slow-version"]}`) + if resp := srv.get(t, "/json/version?launch="+launch); resp.StatusCode != http.StatusBadGateway { + t.Fatalf("status = %d, want 502", resp.StatusCode) + } + eventually(t, func() bool { return len(srv.sessions.Browsers()) == 0 }) +} + +func TestDevTools(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + conn, _ := dial(t, srv.addr(), "/") + conn.roundTrip(t, "hello") + + var targets []map[string]any + json.NewDecoder(srv.get(t, "/json/list").Body).Decode(&targets) + if len(targets) != 1 { + t.Fatalf("targets = %v", targets) + } + + target := targets[0] + id := target["id"].(string) + if ws := target["webSocketDebuggerUrl"]; ws != "ws://"+srv.addr()+"/devtools/page/"+id { + t.Errorf("webSocketDebuggerUrl = %v", ws) + } + if frontend := target["devtoolsFrontendUrl"]; frontend != "/devtools/inspector.html?ws="+srv.addr()+"/devtools/page/"+id { + t.Errorf("devtoolsFrontendUrl = %v", frontend) + } + + t.Run("/json is an alias", func(t *testing.T) { + var alias []map[string]any + json.NewDecoder(srv.get(t, "/json").Body).Decode(&alias) + if len(alias) != 1 || alias[0]["id"] != id { + t.Fatalf("targets = %v", alias) + } + }) + + t.Run("page websocket", func(t *testing.T) { + page, resp := dial(t, srv.addr(), "/devtools/page/"+id) + if resp.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("status = %d", resp.StatusCode) + } + page.roundTrip(t, "page") + page.Close() + + conn.roundTrip(t, "session survives the page connection") + }) + + t.Run("unknown page", func(t *testing.T) { + _, resp := dial(t, srv.addr(), "/devtools/page/missing") + if resp.StatusCode != http.StatusNotFound { + t.Fatalf("status = %d, want 404", resp.StatusCode) + } + }) + + t.Run("frontend", func(t *testing.T) { + body, _ := io.ReadAll(srv.get(t, "/devtools/inspector.html").Body) + if !strings.HasPrefix(string(body), "inspector ") { + t.Fatalf("body = %s", body) + } + }) + + t.Run("protocol", func(t *testing.T) { + body, _ := io.ReadAll(srv.get(t, "/json/protocol").Body) + if !strings.Contains(string(body), `"major":"1"`) { + t.Fatalf("body = %s", body) + } + }) + + t.Run("new target", func(t *testing.T) { + req, _ := http.NewRequest(http.MethodPut, srv.URL+"/json/new?https://example.com/", nil) + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + + var created map[string]any + json.NewDecoder(resp.Body).Decode(&created) + if created["url"] != "https://example.com/" || !strings.HasPrefix(created["webSocketDebuggerUrl"].(string), "ws://"+srv.addr()+"/devtools/page/") { + t.Fatalf("target = %v", created) + } + }) +} + +func TestProtocolWithoutASession(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + body, _ := io.ReadAll(srv.get(t, "/json/protocol").Body) + if !strings.Contains(string(body), `"major":"1"`) { + t.Fatalf("body = %s", body) + } + if n := len(srv.sessions.Browsers()); n != 0 { + t.Fatalf("browsers = %d, want 0", n) + } +} diff --git a/docker/internal/server/sessions_test.go b/docker/internal/server/sessions_test.go new file mode 100644 index 00000000..5fb33e18 --- /dev/null +++ b/docker/internal/server/sessions_test.go @@ -0,0 +1,227 @@ +package server + +import ( + "context" + "errors" + "os" + "testing" + "time" + + "github.com/rubycdp/ferrum/docker/internal/browser" +) + +func newSessions(t *testing.T, config SessionsConfig) *Sessions { + t.Helper() + + config.Binary = os.Args[0] + if config.Concurrent == 0 { + config.Concurrent = 4 + } + + s := NewSessions(config) + t.Cleanup(s.StopAll) + + return s +} + +func TestSessionsStartAndStop(t *testing.T) { + s := newSessions(t, SessionsConfig{}) + + b, err := s.Start(context.Background(), browser.Options{}) + if err != nil { + t.Fatal(err) + } + if len(s.Browsers()) != 1 { + t.Fatalf("browsers = %d, want 1", len(s.Browsers())) + } + + s.Stop(b, "test") + + if len(s.Browsers()) != 0 { + t.Fatalf("browsers = %d, want 0", len(s.Browsers())) + } + select { + case <-b.Done(): + default: + t.Fatal("browser process still running") + } +} + +func TestSessionsStartFailureFreesTheSlot(t *testing.T) { + s := newSessions(t, SessionsConfig{Concurrent: 1}) + + if _, err := s.Start(context.Background(), browser.Options{Args: []string{"--fake-exit"}}); err == nil { + t.Fatal("expected an error") + } + + if _, err := s.Start(context.Background(), browser.Options{}); err != nil { + t.Fatalf("slot not freed: %v", err) + } +} + +func TestSessionsClaim(t *testing.T) { + s := newSessions(t, SessionsConfig{}) + + b, err := s.Start(context.Background(), browser.Options{}) + if err != nil { + t.Fatal(err) + } + s.Reserve(b, "q=1") + + if got := s.Claim(b.ID); got != b { + t.Fatalf("claim = %v, want %v", got, b) + } + if got := s.Claim(b.ID); got != nil { + t.Fatal("claimed twice") + } +} + +func TestSessionsClaimOrStart(t *testing.T) { + s := newSessions(t, SessionsConfig{}) + ctx := context.Background() + + pending, err := s.Start(ctx, browser.Options{}) + if err != nil { + t.Fatal(err) + } + s.Reserve(pending, "q=1") + + t.Run("claims a pending browser with the same query", func(t *testing.T) { + b, err := s.ClaimOrStart(ctx, "", "q=1", browser.Options{}) + if err != nil { + t.Fatal(err) + } + if b != pending { + t.Fatal("started a new browser instead of claiming the pending one") + } + }) + + t.Run("falls back when the id was already claimed", func(t *testing.T) { + other, err := s.Start(ctx, browser.Options{}) + if err != nil { + t.Fatal(err) + } + s.Reserve(other, "q=1") + + b, err := s.ClaimOrStart(ctx, pending.ID, "q=1", browser.Options{}) + if err != nil { + t.Fatal(err) + } + if b != other { + t.Fatal("expected the other pending browser") + } + }) + + t.Run("starts a browser for another query", func(t *testing.T) { + b, err := s.ClaimOrStart(ctx, "", "q=2", browser.Options{}) + if err != nil { + t.Fatal(err) + } + if b == pending { + t.Fatal("claimed a browser with other options") + } + }) +} + +func TestSessionsUnclaimedBrowserIsStopped(t *testing.T) { + s := newSessions(t, SessionsConfig{ClaimTimeout: 50 * time.Millisecond}) + + b, err := s.Start(context.Background(), browser.Options{}) + if err != nil { + t.Fatal(err) + } + s.Reserve(b, "") + + eventually(t, func() bool { return len(s.Browsers()) == 0 }) + <-b.Done() +} + +func TestSessionsExitedBrowserIsRemoved(t *testing.T) { + s := newSessions(t, SessionsConfig{}) + + b, err := s.Start(context.Background(), browser.Options{}) + if err != nil { + t.Fatal(err) + } + b.Stop() + + eventually(t, func() bool { return len(s.Browsers()) == 0 }) +} + +func TestSessionsLimits(t *testing.T) { + ctx := context.Background() + + t.Run("rejects when the queue is full", func(t *testing.T) { + s := newSessions(t, SessionsConfig{Concurrent: 1, Queued: 0}) + + if _, err := s.Start(ctx, browser.Options{}); err != nil { + t.Fatal(err) + } + if _, err := s.Start(ctx, browser.Options{}); !errors.Is(err, ErrQueueFull) { + t.Fatalf("err = %v, want ErrQueueFull", err) + } + }) + + t.Run("queued start waits for a free slot", func(t *testing.T) { + s := newSessions(t, SessionsConfig{Concurrent: 1, Queued: 1}) + + first, err := s.Start(ctx, browser.Options{}) + if err != nil { + t.Fatal(err) + } + + started := make(chan error, 1) + go func() { + _, err := s.Start(ctx, browser.Options{}) + started <- err + }() + + select { + case err := <-started: + t.Fatalf("started without a free slot: %v", err) + case <-time.After(200 * time.Millisecond): + } + + s.Stop(first, "test") + + if err := <-started; err != nil { + t.Fatal(err) + } + }) + + t.Run("queued start gives up when the request is canceled", func(t *testing.T) { + s := newSessions(t, SessionsConfig{Concurrent: 1, Queued: 1}) + + if _, err := s.Start(ctx, browser.Options{}); err != nil { + t.Fatal(err) + } + + canceled, cancel := context.WithTimeout(ctx, 50*time.Millisecond) + defer cancel() + + if _, err := s.Start(canceled, browser.Options{}); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("err = %v, want DeadlineExceeded", err) + } + }) + + t.Run("queued start gives up on shutdown", func(t *testing.T) { + s := newSessions(t, SessionsConfig{Concurrent: 1, Queued: 1}) + + if _, err := s.Start(ctx, browser.Options{}); err != nil { + t.Fatal(err) + } + + started := make(chan error, 1) + go func() { + _, err := s.Start(ctx, browser.Options{}) + started <- err + }() + time.Sleep(50 * time.Millisecond) + + s.StopAll() + + if err := <-started; !errors.Is(err, ErrClosed) { + t.Fatalf("err = %v, want ErrClosed", err) + } + }) +} From b65f9094c61caed379a0b52416e7fa7f65f673fa Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 10:33:47 +0300 Subject: [PATCH 15/23] chore(ci): check formatting, vet and run the Go tests before building the docker images --- .github/workflows/docker.yml | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index b8653b8f..3dc45512 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -13,8 +13,35 @@ on: workflow_dispatch: jobs: + test: + name: Go tests + runs-on: ubuntu-latest + timeout-minutes: 10 + defaults: + run: + working-directory: docker + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Go + uses: actions/setup-go@v5 + with: + go-version-file: docker/go.mod + cache: false + + - name: Check formatting + run: test -z "$(gofmt -l .)" || { gofmt -l .; exit 1; } + + - name: Vet + run: go vet ./... + + - name: Test + run: go test -race ./... + build: name: Build ${{ matrix.browser }} (${{ matrix.arch }}) + needs: test strategy: fail-fast: false matrix: From 7f4c145cc3cdb6660df45da1854cb768119cd0b7 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 10:33:57 +0300 Subject: [PATCH 16/23] test(ci): run the Ferrum suite against the amd64 docker images before publishing them FERRUM_URL points the suite at a driver. CI passes Ferrum's default window size and --disable-web-security in the launch query, so the specs see the same browser as locally. The custom browser path spec needs a local browser and skips remotely. --- .github/workflows/docker.yml | 24 ++++++++++++++++++++++++ spec/browser_spec.rb | 2 ++ spec/spec_helper.rb | 10 ++++++++-- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 3dc45512..ebae3852 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -97,6 +97,30 @@ jobs: docker logs smoke curl -sf http://localhost:3000/json/version | tee /dev/stderr | grep -q webSocketDebuggerUrl + - name: Setup Ruby + if: matrix.arch == 'amd64' + uses: ruby/setup-ruby@v1 + with: + ruby-version: "3.4" + bundler-cache: true + + - name: Run the Ferrum suite against the image + if: matrix.arch == 'amd64' + timeout-minutes: 20 + env: + FERRUM_DEFAULT_TIMEOUT: 15 + FERRUM_PROCESS_TIMEOUT: 25 + run: | + docker rm -f smoke + docker run -d --name suite --network host --user "$(id -u):$(id -g)" -e CONCURRENT=4 \ + -v "$GITHUB_WORKSPACE:$GITHUB_WORKSPACE" -v /tmp:/tmp ${{ matrix.browser }}:test + for i in $(seq 1 30); do curl -sf http://127.0.0.1:3000/health && break; sleep 1; done + FERRUM_URL=http://127.0.0.1:3000 bundle exec rspec --exclude-pattern "spec/unit/**/*_spec.rb" + + - name: Driver logs + if: always() && matrix.arch == 'amd64' + run: docker logs suite 2>&1 | grep -v "browser stderr" | tail -200 || true + - name: Detect browser version run: | mkdir -p /tmp/meta diff --git a/spec/browser_spec.rb b/spec/browser_spec.rb index d870e07a..e97cbd41 100644 --- a/spec/browser_spec.rb +++ b/spec/browser_spec.rb @@ -18,6 +18,8 @@ end it "supports custom chrome path" do + skip "needs a local browser" if ENV["FERRUM_URL"] + original_path = "#{PROJECT_ROOT}/spec/support/chrome_path" File.write(original_path, browser.process.path) diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index f7d328e1..d4e147cb 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -23,8 +23,13 @@ ENV["FERRUM_NEW_WINDOW_WAIT"] ||= "0.8" if ENV["CI"] puts "" -command = Ferrum::Browser::Command.build(Ferrum::Browser::Options.new, nil) -puts `#{Shellwords.escape(command.path)} --version` +remote_url = ENV.fetch("FERRUM_URL", nil) +if remote_url + puts "Remote browser at #{remote_url}" +else + command = Ferrum::Browser::Command.build(Ferrum::Browser::Options.new, nil) + puts `#{Shellwords.escape(command.path)} --version` +end puts "" RSpec.configure do |config| @@ -41,6 +46,7 @@ config.before(:all) do base_url = Ferrum::Server.server.base_url options = { base_url: base_url } + options.merge!(url: ENV["FERRUM_URL"]) if ENV["FERRUM_URL"] options.merge!(headless: false) if ENV["HEADLESS"] == "false" options.merge!(slowmo: ENV["SLOWMO"].to_f) if ENV["SLOWMO"].to_f > 0 From 5f161dcdf455f6c4db320d20a979c9b586a30b45 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 15:48:24 +0300 Subject: [PATCH 17/23] feat(docker): add a HEALTHCHECK on /health that follows PORT --- docker/Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docker/Dockerfile b/docker/Dockerfile index 0bbe76aa..d3f1d49e 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -84,5 +84,7 @@ USER driver WORKDIR /home/driver EXPOSE 3000 STOPSIGNAL SIGTERM +HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \ + CMD curl -fsS "http://127.0.0.1:${PORT:-3000}/health" || exit 1 ENTRYPOINT ["dumb-init", "--", "driver"] From 981f6df2111b3e2d7b81f966275d92a59882e2fc Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 15:48:24 +0300 Subject: [PATCH 18/23] docs(docker): compare the driver with browserless v2 and bring the design status up to date --- docker/DESIGN.md | 161 ++++++++++++++++++++++++++++++----------------- docker/main.go | 6 +- 2 files changed, 105 insertions(+), 62 deletions(-) diff --git a/docker/DESIGN.md b/docker/DESIGN.md index 3712a51f..9aeb1325 100644 --- a/docker/DESIGN.md +++ b/docker/DESIGN.md @@ -1,9 +1,12 @@ # Design +> Better than browserless, not equal to it. + A small Go binary that runs as the entrypoint of a Docker container and gives each client its own Chrome over CDP. -It looks like browserless v2 (checked against v2.57.0) from the outside, but works differently inside. There's no Puppeteer, and nothing -touches the client's targets: bytes go through unchanged. It's self-hosted only (local machines, CI, your own -servers) and Docker only. +It replaces browserless with minimal changes: usually swap the image and connect to `/`. It doesn't copy +browserless, though: where browserless gets in the way, the driver does it differently. There's no Puppeteer, +and nothing touches the client's targets: bytes go through unchanged. It's self-hosted only (local machines, CI, +your own servers) and Docker only. ## Why @@ -18,6 +21,48 @@ Most remote-Chrome bugs Ferrum and Cuprite users report come from browserless be - `quit` doesn't end the session (ferrum#418), and orphaned Chrome trees pile up (ferrum#505). - Zombie processes pile up when the main process is PID 1 (ferrum#364), and there's no native arm64 image (ferrum#270). +## Compared to browserless v2 + +Moving from browserless usually means swapping the image and nothing else, but check the differences below. + +### Shared conventions + +Kept so moving over takes minimal changes: + +- Port 3000, uid 999 and dumb-init. +- Env: `PORT`, `HOST`, `CONCURRENT`, `QUEUED`, `TIMEOUT` and `EXTERNAL` / `PROXY_URL`. +- Query: `launch` (Puppeteer's launch format, JSON or base64) and `timeout`. + +### What's different + +| | browserless v2 | driver | +|---|---|---| +| Runtime | A TypeScript app on Node, using Puppeteer and Playwright | One static Go binary | +| Base image | Ubuntu | `debian:trixie-slim` | +| Browser | Installed by Playwright's installer | Debian's `chromium` package or Google's `.deb` | +| Image size | About 3 GB, 1 GB compressed | About 1 GB, under 500 MB compressed | +| Architectures | `chromium`: amd64 and arm64; `chrome`: amd64 only | Both images on amd64 and arm64 | +| Client's targets | Its own Puppeteer connection sets the viewport and the download folder | Nothing besides the client talks to Chrome | +| Session timeout | 30s by default | None by default | +| `/json/version` | A Chrome started once at boot, with cached metadata and a generic URL without path or query | The session's own Chrome, real metadata, a URL the client can reach, with the query kept | +| `headless: false` | One shared Xvfb `:99` at 1024x768x16, started with the container | One Xvfb per session, sized to the window | +| PID 1 | A bash script; dumb-init runs only the Node process | dumb-init, reaping every orphan | +| `--user ` | `HOME` belongs to `blessuser` | `HOME` is world-writable, any uid works | +| Default flags | Puppeteer's plus browserless's | Ferrum's | +| Auth | `TOKEN` | None, so never publish the port | + +### What's left out + +These are browserless features the driver doesn't have and won't add, because Ferrum and Cuprite drive Chrome +over CDP themselves and files go through bind mounts: + +- the REST APIs (`/screenshot`, `/pdf`, `/content`, `/function`, `/download`, `/scrape`, `/performance`); +- Playwright endpoints, and Firefox, WebKit and Edge images; +- token auth, CORS, `ALLOW_GET` and `ALLOW_FILE_PROTOCOL`; +- the sessions, metrics, pressure and config endpoints, the debugger UI and the alert webhooks; +- ad blocking, stealth, recording, reconnecting to a kept-alive browser; +- the file workspace, `DATA_DIR` and `DOWNLOAD_DIR`. + ## Process model - The container runs dumb-init as PID 1, then `driver`, then one Chrome per session. dumb-init reaps orphans, @@ -30,12 +75,10 @@ Most remote-Chrome bugs Ferrum and Cuprite users report come from browserless be - Both proxy directions are then torn down. - The whole process group gets TERM, then KILL after a grace period, and the profile directory is deleted. - Go's default TCP keepalive on accepted connections catches a client that vanished without closing. -- `CONCURRENT` limits running sessions and defaults to the number of CPUs. Up to `QUEUED` extra connections wait +- `CONCURRENT` limits running sessions and defaults to the number of CPUs. Up to `QUEUED` (default 10) extra connections wait for a free slot. Beyond that they get `429` with `Retry-After`, as a plain HTTP response sent before the - WebSocket upgrade, never bytes on a live socket. The names match browserless v2, so its compose files work - unchanged. -- There's no session timeout by default; browserless v2 defaults to 30s. `TIMEOUT` (env) or `?timeout=` (query), - both in milliseconds, set one. The driver logs why each session ended, so `docker logs` shows it. + WebSocket upgrade, never bytes on a live socket. +- There's no session timeout by default; browserless v2 defaults to 30s. `TIMEOUT` and `?timeout=`, both in milliseconds, set one. The driver logs why each session ended, so `docker logs` shows it. ## HTTP surface @@ -44,22 +87,30 @@ The driver serves everything through `net/http`, and WebSocket upgrades go throu | Route | Behavior | |---|---| | `GET /json/version` | Starts the session's Chrome, proxies Chrome's real `/json/version`, and rewrites `webSocketDebuggerUrl` to `ws(s):///devtools/browser/?` | -| WS `/devtools/browser/` | Attaches to the Chrome started by `/json/version`, `404` if there's none (as in v2) | +| WS `/devtools/browser/` | Attaches to the Chrome started by `/json/version`. If another connection took it, falls back like `/` | | WS `/`, `/chrome`, `/chromium` | Claims a pending Chrome started with the same query, or starts a new one. For `ws_url:` clients and browserless URLs | -| `GET /json/list`, `/json/new`, `/json/protocol`, `/devtools/*`, WS `/devtools/page/` | Forwarded to the Chrome that owns the target, for the DevTools viewer (v2 has the same routes) | +| `GET /json`, `/json/list` | Targets of every running session, with URLs pointing at the driver | +| `PUT /json/new?` | Opens a target, only while exactly one session runs (`409` otherwise) | +| `GET /json/protocol` | Chrome's protocol description, cached. Starts a short-lived Chrome only when none runs | +| WS `/devtools/page/` | Connects to one target of whichever session owns it. Closing it doesn't end the session | +| `GET /devtools/*` | The DevTools frontend, served by a running Chrome | | `GET /health` | Driver liveness. Never starts a Chrome | -Like v2, every route also accepts a trailing slash: `/chrome/` works like `/chrome`. +Every route also accepts a trailing slash: `/json/version/` works like `/json/version`. `/json/version` is answered on the fly, so the client gets real `Browser`, `User-Agent` and protocol data, with no -Chrome started at boot or just for discovery. This differs from v2 on purpose. v2 starts a throwaway Chrome once, -caches its metadata, and always returns its generic address (`EXTERNAL`, or `HOST:PORT`) with no path and no -query, so the client connects to `/`. Clients work against either. Details: - +Chrome started at boot or just for discovery. browserless v2 instead starts a throwaway Chrome once, caches its +metadata, and always returns its generic address (`EXTERNAL`, or `HOST:PORT`) with no path and no query. +Details: + +- **Taken ids.** With `ws_url:`, Ferrum calls `/json/version` and then connects to `/`, which claims any pending + Chrome with the same query. With parallel clients that can be the Chrome another client's `/json/version` + started, so `/devtools/browser/` falls back the same way instead of answering `404`. Chromes started with + the same query are interchangeable. - **Unclaimed sessions.** A Chrome started by `/json/version` is pending until a WebSocket claims it. If none arrives within 30 seconds, for example from a stray `curl` or a misconfigured healthcheck, it's killed. - **Reachable URL.** The URL is built from `Host` / `X-Forwarded-Host` / `X-Forwarded-Proto`, so it's always - reachable from the client (no `0.0.0.0`), and the query string is kept. `EXTERNAL` (or `PROXY_URL`, as in v2) + reachable from the client (no `0.0.0.0`), and the query string is kept. `EXTERNAL` (or `PROXY_URL`) overrides the scheme and host when the driver sits behind a proxy that rewrites them. - **Host header.** Requests forwarded to Chrome carry `Host: 127.0.0.1:`, because Chrome rejects any other host on its HTTP endpoint. @@ -78,33 +129,25 @@ CDP traffic. ## Launch options -The query string from `/json/version` or the WebSocket URL accepts the same parameters as browserless v2: +The query string from `/json/version` or the WebSocket URL accepts: -- `launch`: JSON or base64-encoded JSON, in v2's (Puppeteer launch) format. Supported keys: +- `launch`: JSON or base64-encoded JSON, in Puppeteer's launch format. Supported keys: - `args`: extra Chrome flags, e.g. `--proxy-server=…` or `--accept-lang=…`. A flag with the same name as a default replaces it; - `defaultViewport` `{width, height}`: becomes `--window-size`; - - `ignoreDefaultArgs`: `true`, or a list of default flags to drop, by full flag or by name. - - later: `headless: false` starts Chrome under Xvfb. -- `timeout` (later): the session timeout in milliseconds. + - `ignoreDefaultArgs`: `true`, or a list of default flags to drop, by full flag or by name; + - `headless: false`: Chrome runs with a window on its own Xvfb display, sized to the window and stopped with + the session. Without Xvfb in `PATH` (e.g. on a desktop), Chrome opens a real window. +- `timeout`: the session timeout in milliseconds, overriding `TIMEOUT`. `0` turns it off. `--remote-debugging-port` and `--user-data-dir` are always set by the driver. An invalid `launch` gets `400` -before any Chrome starts. Other parameters (`token`, `blockAds`, `blockAdsInclude`) and other `launch` keys are -ignored, so existing browserless URLs keep working. `driver --help` lists the launch options and default flags. +before any Chrome starts. Other parameters and other `launch` keys are ignored. `driver --help` lists the launch options and default flags. Anyone who can connect can pass flags like `--renderer-cmd-prefix` and run commands in the container. There's no authentication, so never publish the port beyond localhost or a private network. -Default flags: - -- headless; -- `--no-startup-window`, so no implicit context; -- `--no-sandbox`, since the container is the boundary; -- `--disable-dev-shm-usage`; -- `--disable-features=site-per-process`, which makes ferrum#640 less likely. - -Never added: `--disable-web-security`, `--disable-crashpad-for-testing` (the network service crash-loops), or -`NetworkServiceInProcess`. +Default flags are Ferrum's, the same as `Ferrum::Browser.new(dockerize: true)` locally, window size included +(`driver --help` lists them). The one exception is `--remote-allow-origins`: the driver strips `Origin` instead. ## Files @@ -127,25 +170,26 @@ under it, which crashpad needs (ferrum#541, #544). Downloads then belong to the - Two images, `ghcr.io/rubycdp/chrome` (Google's `.deb`) and `ghcr.io/rubycdp/chromium` (Debian's package), each for amd64 and arm64. Tags: `latest`, the browser's major version (`155`) and its full version (`155.0.8059.39`). - The Go binary is cross-compiled in a build stage, so the image has no Node, Playwright or npm. -- `.github/workflows/docker.yml` builds each browser and architecture pair on a native runner, never through - QEMU, which crashes Chrome (ferrum#270). It smoke-tests every image and merges the architectures into one - multi-arch tag by digest. +- `.github/workflows/docker.yml` runs the Go tests, then builds each browser and architecture pair on a native + runner, never through QEMU, which crashes Chrome (ferrum#270). + - It smoke-tests every image. + - On amd64, it runs the Ferrum suite against the image, with `FERRUM_URL` pointing at the driver. + - It merges the architectures into one multi-arch tag by digest. - Fonts: Debian's font packages plus open substitutes for Microsoft fonts. Selawik replaces Segoe UI, Carlito replaces Calibri, Caladea replaces Cambria, and DejaVu replaces Tahoma. -- Later: the Ferrum suite run against each image before it's published. -- Env, with names matching browserless v2: +- Env: - `PORT` (default 3000) and `HOST` (default `0.0.0.0`); - `CONCURRENT`, `QUEUED` and `TIMEOUT`; - `EXTERNAL` / `PROXY_URL`; - plus `BROWSER_PATH`, which the image sets. -- A healthcheck on `/health`, so compose can use `condition: service_healthy`. +- A `HEALTHCHECK` on `/health` (it follows `PORT`), so compose can use `condition: service_healthy`. ## Usage with Cuprite in compose This follows the Evil Martians "System of a test" setup: -- **The `chrome` service.** Use this image instead of `browserless/chrome`. `CONNECTION_TIMEOUT` and the - M1-specific tag aren't needed, and `depends_on` can use `service_healthy`. +- **The `chrome` service.** Use `ghcr.io/rubycdp/chrome` or `chromium`. One tag works on amd64 and arm64, and + `depends_on` can use `service_healthy`. - **Connecting.** `url: ENV["CHROME_URL"]` (`http://chrome:3333`) works as is: one `/json/version`, then the WebSocket. The TCP availability check with fallback to a local Chrome also works as is. - **Sessions.** Cuprite opens one connection per test process and calls `reset` between tests, so it's one Chrome @@ -155,7 +199,7 @@ This follows the Evil Martians "System of a test" setup: `extra_hosts: ["host.docker.internal:host-gateway"]` on Linux. - **WebMock and VCR.** Allow the `chrome` host, because `/json/version` goes through `Net::HTTP`. - **Launch flags.** `browser_options: {"no-sandbox" => nil}` isn't needed. Ferrum doesn't send launch flags to a - remote browser anyway. + remote browser, so put them in the URL instead, e.g. `?launch={"defaultViewport":{"width":1400,"height":1400}}`. - **GitHub Actions.** Service containers start before checkout, so start the driver with `docker run` in a step after checkout when uploads need the workspace mounted. @@ -166,30 +210,29 @@ This follows the Evil Martians "System of a test" setup: 2. `Browser#debug` in remote mode prints the DevTools URL instead of `Process.spawn(process.path, …)`, which fails because the path is nil. 3. Never look for a local binary when `url` or `ws_url` is set (ferrum#554). -4. Optionally, send `window_size` and `browser_options` in v2's `launch` query parameter (`defaultViewport` and +4. Optionally, send `window_size` and `browser_options` in the `launch` query parameter (`defaultViewport` and `args`) when connecting remotely. ## Status Done: -- Config is `BROWSER_PATH`, `HOST` and `PORT`. +- Config: `BROWSER_PATH`, `HOST`, `PORT`, `CONCURRENT`, `QUEUED`, `TIMEOUT`, `EXTERNAL` / `PROXY_URL`. Invalid + values stop the driver at startup. - One Chrome per connection, on `--remote-debugging-port=0`, with the endpoint read from stderr. -- `/json/version` is answered on the fly, with a claim timeout. +- `/json/version` is answered on the fly, with a claim timeout and a fallback for taken ids. - The WebSocket goes through `ReverseProxy`. -- The process group gets TERM, then KILL, and the profile is removed when the connection ends. -- `/health`. +- The process group gets TERM, then KILL, and the profile is removed when the connection ends. Each session logs + why it ended: `connection closed`, `browser exited`, `timeout`, `never claimed` or `shutdown`. +- Session limits with `429` and `Retry-After`; queued connections give up when the client leaves or the driver + shuts down. +- Launch options (`args`, `defaultViewport`, `ignoreDefaultArgs`, `headless`) and `timeout` in the URL. +- The DevTools routes. +- `/health` and the image's `HEALTHCHECK`, trailing-slash routes, `driver --help` and `--version`. - A graceful shutdown stops every Chrome. -- Trailing-slash routes (`/chrome/`, `/chromium/`, `/json/version/`), as in v2. -- The `launch` query parameter (`args`, `defaultViewport`, `ignoreDefaultArgs`). -- `driver --help` and `--version`. - -With `ws_url:`, Ferrum still calls `/json/version` before connecting to `/`. Pending Chromes are therefore matched -by query string, so a connection to `/` claims the one that call started, instead of starting a second. +- Go tests (`go test ./...`), with the test binary acting as a fake Chrome, run in CI before any image is built. +- The Ferrum suite runs against the amd64 images in CI. Not done yet: -- `CONCURRENT`, `QUEUED` and `TIMEOUT`, plus `EXTERNAL`; -- the `timeout` query parameter and `launch.headless`; -- the DevTools proxy (`/json/list`, `/json/new`, `/json/protocol`, `/devtools/*`); -- cleaned-up default flags; -- a `HEALTHCHECK` in the image; -- Go tests beyond the launch options, and a Go job in CI. +- the Ferrum suite on arm64, whose runner has no local Chrome for the specs that start one; +- the zero-copy tunnel; +- the Ferrum changes above. diff --git a/docker/main.go b/docker/main.go index c071d762..658384e1 100644 --- a/docker/main.go +++ b/docker/main.go @@ -22,7 +22,7 @@ const shutdownTimeout = 10 * time.Second const usage = `Usage: driver [options] -Serves one Chrome per CDP connection, compatible with browserless v2 clients. +Serves one Chrome per CDP connection. Each WebSocket gets a fresh Chrome that is killed when the client disconnects. Options: @@ -57,10 +57,10 @@ Launch options (per connection, in the URL of /json/version or the WebSocket): ignoreDefaultArgs true, or a list of default flags to drop headless false runs Chrome with a window, under its own Xvfb timeout= session timeout for this connection, overrides TIMEOUT - Other launch keys and query params (token, blockAds) are ignored. + Other launch keys and query params are ignored. --remote-debugging-port and --user-data-dir are managed by the driver. - http://localhost:3000?launch={"args":["--accept-lang=de-DE"]} + http://localhost:3000?launch={"args":["--accept-lang=de-DE"]}&timeout=60000 ws://localhost:3000/chrome?launch={"defaultViewport":{"width":1280,"height":720}} Default Chrome flags: From 01e825c9a1a437f2549f17752450e76bc7e949b2 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 19:14:14 +0300 Subject: [PATCH 19/23] ref(docker): drop the /chrome and /chromium routes, since an image has only one browser --- docker/DESIGN.md | 3 ++- docker/internal/server/server.go | 2 -- docker/internal/server/server_test.go | 21 ++++----------------- docker/main.go | 6 +++--- 4 files changed, 9 insertions(+), 23 deletions(-) diff --git a/docker/DESIGN.md b/docker/DESIGN.md index 9aeb1325..882d355e 100644 --- a/docker/DESIGN.md +++ b/docker/DESIGN.md @@ -57,6 +57,7 @@ These are browserless features the driver doesn't have and won't add, because Fe over CDP themselves and files go through bind mounts: - the REST APIs (`/screenshot`, `/pdf`, `/content`, `/function`, `/download`, `/scrape`, `/performance`); +- the `/chrome` and `/chromium` paths: an image has one browser, so connect to `/`; - Playwright endpoints, and Firefox, WebKit and Edge images; - token auth, CORS, `ALLOW_GET` and `ALLOW_FILE_PROTOCOL`; - the sessions, metrics, pressure and config endpoints, the debugger UI and the alert webhooks; @@ -88,7 +89,7 @@ The driver serves everything through `net/http`, and WebSocket upgrades go throu |---|---| | `GET /json/version` | Starts the session's Chrome, proxies Chrome's real `/json/version`, and rewrites `webSocketDebuggerUrl` to `ws(s):///devtools/browser/?` | | WS `/devtools/browser/` | Attaches to the Chrome started by `/json/version`. If another connection took it, falls back like `/` | -| WS `/`, `/chrome`, `/chromium` | Claims a pending Chrome started with the same query, or starts a new one. For `ws_url:` clients and browserless URLs | +| WS `/` | Claims a pending Chrome started with the same query, or starts a new one. For `ws_url:` clients | | `GET /json`, `/json/list` | Targets of every running session, with URLs pointing at the driver | | `PUT /json/new?` | Opens a target, only while exactly one session runs (`409` otherwise) | | `GET /json/protocol` | Chrome's protocol description, cached. Starts a short-lived Chrome only when none runs | diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index 3fa54704..e856412c 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -54,8 +54,6 @@ func New(sessions *Sessions, config Config) http.Handler { handle(mux, "GET", "/json/new", s.newTarget) handle(mux, "GET", "/devtools/browser/{id}", s.connect) handle(mux, "GET", "/devtools/page/{id}", s.page) - handle(mux, "GET", "/chrome", s.connect) - handle(mux, "GET", "/chromium", s.connect) mux.HandleFunc("GET /devtools/{path...}", s.frontend) mux.HandleFunc("GET /{$}", s.connect) diff --git a/docker/internal/server/server_test.go b/docker/internal/server/server_test.go index 1c3db89e..c94e6b7f 100644 --- a/docker/internal/server/server_test.go +++ b/docker/internal/server/server_test.go @@ -105,25 +105,11 @@ func TestVersionAndClaim(t *testing.T) { eventually(t, func() bool { return len(srv.sessions.Browsers()) == 0 }) } -func TestConnectRoutes(t *testing.T) { - srv := newServer(t, SessionsConfig{}, Config{}) - - for _, path := range []string{"/", "/chrome", "/chrome/", "/chromium", "/chromium/"} { - t.Run(path, func(t *testing.T) { - conn, resp := dial(t, srv.addr(), path) - if resp.StatusCode != http.StatusSwitchingProtocols { - t.Fatalf("status = %d", resp.StatusCode) - } - conn.roundTrip(t, "hello") - }) - } -} - func TestConnectClaimsTheBrowserStartedByVersion(t *testing.T) { srv := newServer(t, SessionsConfig{}, Config{}) srv.version(t, "?launch=%7B%7D") - conn, _ := dial(t, srv.addr(), "/chrome?launch=%7B%7D") + conn, _ := dial(t, srv.addr(), "/?launch=%7B%7D") conn.roundTrip(t, "hello") if n := len(srv.sessions.Browsers()); n != 1 { @@ -155,7 +141,8 @@ func TestBadRequests(t *testing.T) { for path, status := range map[string]int{ "/json/version?launch=%7Bbroken": http.StatusBadRequest, "/json/version?timeout=soon": http.StatusBadRequest, - "/chrome": http.StatusUpgradeRequired, + "/": http.StatusUpgradeRequired, + "/chrome": http.StatusNotFound, "/devtools/browser/missing": http.StatusUpgradeRequired, "/devtools/page/missing": http.StatusUpgradeRequired, "/unknown": http.StatusNotFound, @@ -210,7 +197,7 @@ func TestQueueLimit(t *testing.T) { t.Error("missing Retry-After") } - _, resp = dial(t, srv.addr(), "/chrome") + _, resp = dial(t, srv.addr(), "/") if resp.StatusCode != http.StatusTooManyRequests { t.Fatalf("websocket status = %d, want 429", resp.StatusCode) } diff --git a/docker/main.go b/docker/main.go index 658384e1..8d9e2f52 100644 --- a/docker/main.go +++ b/docker/main.go @@ -42,7 +42,7 @@ Environment: Endpoints: GET /json/version start a Chrome, return its webSocketDebuggerUrl WS /devtools/browser/ connect to the Chrome started by /json/version - WS /, /chrome, /chromium connect to a new Chrome + WS / connect to a new Chrome GET /json, /json/list targets of every running session PUT /json/new? open a target, when exactly one session runs GET /json/protocol the CDP protocol description @@ -61,7 +61,7 @@ Launch options (per connection, in the URL of /json/version or the WebSocket): --remote-debugging-port and --user-data-dir are managed by the driver. http://localhost:3000?launch={"args":["--accept-lang=de-DE"]}&timeout=60000 - ws://localhost:3000/chrome?launch={"defaultViewport":{"width":1280,"height":720}} + ws://localhost:3000?launch={"defaultViewport":{"width":1280,"height":720}} Default Chrome flags: %s @@ -71,7 +71,7 @@ Examples: BROWSER_PATH=/usr/bin/chromium PORT=9222 driver Ferrum::Browser.new(url: "http://localhost:3000") - Ferrum::Browser.new(ws_url: "ws://localhost:3000/chrome") + Ferrum::Browser.new(ws_url: "ws://localhost:3000") ` func main() { From 9cd5c8e2b2b9fac3c65826d582dff6e0bfe58541 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 20:15:03 +0300 Subject: [PATCH 20/23] feat(docker): serve a dashboard of running sessions with DevTools links at / --- docker/DESIGN.md | 10 ++- docker/internal/server/dashboard.go | 110 ++++++++++++++++++++++++++ docker/internal/server/server.go | 4 +- docker/internal/server/server_test.go | 44 ++++++++++- docker/main.go | 1 + 5 files changed, 162 insertions(+), 7 deletions(-) create mode 100644 docker/internal/server/dashboard.go diff --git a/docker/DESIGN.md b/docker/DESIGN.md index 882d355e..227fa452 100644 --- a/docker/DESIGN.md +++ b/docker/DESIGN.md @@ -3,10 +3,10 @@ > Better than browserless, not equal to it. A small Go binary that runs as the entrypoint of a Docker container and gives each client its own Chrome over CDP. -It replaces browserless with minimal changes: usually swap the image and connect to `/`. It doesn't copy +It replaces browserless with minimal changes: usually swap the image and connect. It doesn't copy browserless, though: where browserless gets in the way, the driver does it differently. There's no Puppeteer, and nothing touches the client's targets: bytes go through unchanged. It's self-hosted only (local machines, CI, -your own servers) and Docker only. +your own servers) at least for now. ## Why @@ -95,6 +95,7 @@ The driver serves everything through `net/http`, and WebSocket upgrades go throu | `GET /json/protocol` | Chrome's protocol description, cached. Starts a short-lived Chrome only when none runs | | WS `/devtools/page/` | Connects to one target of whichever session owns it. Closing it doesn't end the session | | `GET /devtools/*` | The DevTools frontend, served by a running Chrome | +| `GET /` | A dashboard of running sessions and their pages, each linking to DevTools. Refreshes every 3 seconds | | `GET /health` | Driver liveness. Never starts a Chrome | Every route also accepts a trailing slash: `/json/version/` works like `/json/version`. @@ -116,8 +117,8 @@ Details: - **Host header.** Requests forwarded to Chrome carry `Host: 127.0.0.1:`, because Chrome rejects any other host on its HTTP endpoint. - **DevTools URLs.** In the `devtoolsFrontendUrl` returned by `/json`, the `ws=` parameter is rewritten to the - driver's host. Ferrum's `debug_url` and the browser's DevTools then give a live, clickable view of the session. - That's our answer to browserless's debug viewer. + driver's host. The dashboard at `/` links each page to the DevTools frontend served by its Chrome, a live view + of the running test. That's our answer to browserless's debug viewer. ### Zero-copy tunnel (potential) @@ -228,6 +229,7 @@ Done: shuts down. - Launch options (`args`, `defaultViewport`, `ignoreDefaultArgs`, `headless`) and `timeout` in the URL. - The DevTools routes. +- The dashboard at `/`, linked from each "session opened" log line. - `/health` and the image's `HEALTHCHECK`, trailing-slash routes, `driver --help` and `--version`. - A graceful shutdown stops every Chrome. - Go tests (`go test ./...`), with the test binary acting as a fake Chrome, run in CI before any image is built. diff --git a/docker/internal/server/dashboard.go b/docker/internal/server/dashboard.go new file mode 100644 index 00000000..64a6e873 --- /dev/null +++ b/docker/internal/server/dashboard.go @@ -0,0 +1,110 @@ +package server + +import ( + "html/template" + "log/slog" + "net/http" + "net/url" + "slices" + "strings" + + "github.com/rubycdp/ferrum/docker/internal/browser" +) + +var dashboardTemplate = template.Must(template.New("dashboard").Parse(` + + + + + +Ferrum driver + + + +

Ferrum driver

+{{- if .}} +

{{len .}} running {{if eq (len .) 1}}session{{else}}sessions{{end}}

+{{- range .}} +
+

{{.ID}}

+{{- if .Pages}} + +{{- else}} +

No pages

+{{- end}} +
+{{- end}} +{{- else}} +

No running sessions.

+{{- end}} + + +`)) + +type dashboardSession struct { + ID string + Pages []dashboardPage +} + +type dashboardPage struct { + Title string + URL string + DevTools string +} + +func (s *Server) root(w http.ResponseWriter, r *http.Request) { + if isUpgrade(r) { + s.connect(w, r) + return + } + + browsers := s.sessions.Browsers() + slices.SortFunc(browsers, func(a, b *browser.Browser) int { return strings.Compare(a.ID, b.ID) }) + + sessions := make([]dashboardSession, 0, len(browsers)) + for _, b := range browsers { + targets, err := s.targets(b) + if err != nil { + slog.Warn("cannot list browser targets", "id", b.ID, "error", err) + continue + } + + session := dashboardSession{ID: b.ID} + for _, target := range targets { + if target["type"] != "page" { + continue + } + + id, _ := target["id"].(string) + title, _ := target["title"].(string) + url, _ := target["url"].(string) + session.Pages = append(session.Pages, dashboardPage{Title: title, URL: url, DevTools: s.inspectorURL(r, id)}) + } + sessions = append(sessions, session) + } + + w.Header().Set("Content-Type", "text/html; charset=utf-8") + dashboardTemplate.Execute(w, sessions) +} + +func (s *Server) inspectorURL(r *http.Request, id string) string { + ws, _ := url.Parse(s.publicURL(r, "/devtools/page/"+id, "")) + return s.httpURL(r, "/devtools/inspector.html") + "?" + ws.Scheme + "=" + ws.Host + ws.Path +} + +func (s *Server) httpURL(r *http.Request, path string) string { + u, _ := url.Parse(s.publicURL(r, path, "")) + u.Scheme = strings.Replace(u.Scheme, "ws", "http", 1) + return u.String() +} diff --git a/docker/internal/server/server.go b/docker/internal/server/server.go index e856412c..ebaed8ab 100644 --- a/docker/internal/server/server.go +++ b/docker/internal/server/server.go @@ -55,7 +55,7 @@ func New(sessions *Sessions, config Config) http.Handler { handle(mux, "GET", "/devtools/browser/{id}", s.connect) handle(mux, "GET", "/devtools/page/{id}", s.page) mux.HandleFunc("GET /devtools/{path...}", s.frontend) - mux.HandleFunc("GET /{$}", s.connect) + mux.HandleFunc("GET /{$}", s.root) return mux } @@ -124,7 +124,7 @@ func (s *Server) connect(w http.ResponseWriter, r *http.Request) { defer timer.Stop() } - slog.Info("session opened", "id", b.ID, "client", r.RemoteAddr, "timeout", timeout) + slog.Info("session opened", "id", b.ID, "client", r.RemoteAddr, "timeout", timeout, "dashboard", s.httpURL(r, "/")+"#"+b.ID) proxy(b, b.WSPath, "").ServeHTTP(w, r) s.sessions.Stop(b, "connection closed") } diff --git a/docker/internal/server/server_test.go b/docker/internal/server/server_test.go index c94e6b7f..993b7c3e 100644 --- a/docker/internal/server/server_test.go +++ b/docker/internal/server/server_test.go @@ -43,6 +43,17 @@ func (s *testServer) get(t *testing.T, path string) *http.Response { return resp } +func (s *testServer) body(t *testing.T, path string) string { + t.Helper() + + body, err := io.ReadAll(s.get(t, path).Body) + if err != nil { + t.Fatal(err) + } + + return string(body) +} + func (s *testServer) version(t *testing.T, query string) map[string]any { t.Helper() @@ -141,7 +152,6 @@ func TestBadRequests(t *testing.T) { for path, status := range map[string]int{ "/json/version?launch=%7Bbroken": http.StatusBadRequest, "/json/version?timeout=soon": http.StatusBadRequest, - "/": http.StatusUpgradeRequired, "/chrome": http.StatusNotFound, "/devtools/browser/missing": http.StatusUpgradeRequired, "/devtools/page/missing": http.StatusUpgradeRequired, @@ -378,3 +388,35 @@ func TestProtocolWithoutASession(t *testing.T) { t.Fatalf("browsers = %d, want 0", n) } } + +func TestDashboard(t *testing.T) { + srv := newServer(t, SessionsConfig{}, Config{}) + + if body := srv.body(t, "/"); !strings.Contains(body, "No running sessions.") { + t.Fatalf("body = %s", body) + } + + conn, _ := dial(t, srv.addr(), "/") + conn.roundTrip(t, "hello") + + b := srv.sessions.Browsers()[0] + body := srv.body(t, "/") + if !strings.Contains(body, `id="`+b.ID+`"`) { + t.Errorf("session %s missing in %s", b.ID, body) + } + if want := srv.URL + "/devtools/inspector.html?ws=" + srv.addr() + "/devtools/page/page-"; !strings.Contains(body, want) { + t.Errorf("DevTools link %s missing in %s", want, body) + } + + t.Run("behind a proxy", func(t *testing.T) { + external, _ := url.Parse("https://proxy.example.com/chrome") + srv := newServer(t, SessionsConfig{}, Config{External: external}) + conn, _ := dial(t, srv.addr(), "/") + conn.roundTrip(t, "hello") + + want := "https://proxy.example.com/chrome/devtools/inspector.html?wss=proxy.example.com/chrome/devtools/page/page-" + if body := srv.body(t, "/"); !strings.Contains(body, want) { + t.Errorf("DevTools link %s missing in %s", want, body) + } + }) +} diff --git a/docker/main.go b/docker/main.go index 8d9e2f52..1a835dcb 100644 --- a/docker/main.go +++ b/docker/main.go @@ -49,6 +49,7 @@ Endpoints: WS /devtools/page/ connect to one target, e.g. from DevTools GET /devtools/* the DevTools frontend of a running session GET /health liveness, never starts a Chrome + GET / dashboard: running sessions with DevTools links Launch options (per connection, in the URL of /json/version or the WebSocket): launch= From f7957dc3c4ebbe3e354f0cc8d250c1ef9a7632a9 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 20:54:16 +0300 Subject: [PATCH 21/23] test: tag specs that start their own browser and skip them when FERRUM_URL points at a remote one --- spec/browser/options/chrome_spec.rb | 2 +- spec/browser/xvfb_spec.rb | 2 +- spec/browser_spec.rb | 56 ++++++++++++++--------------- spec/client_spec.rb | 14 ++++---- spec/contexts_spec.rb | 6 ++-- spec/cookies_spec.rb | 2 +- spec/downloads_spec.rb | 2 +- spec/frame/runtime_spec.rb | 14 ++++---- spec/page/screenshot_spec.rb | 2 +- spec/spec_helper.rb | 1 + 10 files changed, 50 insertions(+), 51 deletions(-) diff --git a/spec/browser/options/chrome_spec.rb b/spec/browser/options/chrome_spec.rb index ea8a329e..66ef043f 100644 --- a/spec/browser/options/chrome_spec.rb +++ b/spec/browser/options/chrome_spec.rb @@ -27,7 +27,7 @@ end describe ".version" do - it "returns an executable version" do + it "returns an executable version", local_browser: true do expect(described_class.version).to match(/(Chromium|Chrome)(?: for Testing)? \d/) end end diff --git a/spec/browser/xvfb_spec.rb b/spec/browser/xvfb_spec.rb index cbeb7b8a..8bd5a51b 100644 --- a/spec/browser/xvfb_spec.rb +++ b/spec/browser/xvfb_spec.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -describe Ferrum::Browser::Xvfb, skip: !Ferrum::Browser::Binary.find("Xvfb") do +describe Ferrum::Browser::Xvfb, local_browser: true, skip: !Ferrum::Browser::Binary.find("Xvfb") do let(:process) { xvfb_browser.process } let(:xvfb_browser) { Ferrum::Browser.new(default_options.merge(options)) } let(:default_options) { Hash(headless: true, xvfb: true) } diff --git a/spec/browser_spec.rb b/spec/browser_spec.rb index e97cbd41..42258678 100644 --- a/spec/browser_spec.rb +++ b/spec/browser_spec.rb @@ -17,9 +17,7 @@ end end - it "supports custom chrome path" do - skip "needs a local browser" if ENV["FERRUM_URL"] - + it "supports custom chrome path", local_browser: true do original_path = "#{PROJECT_ROOT}/spec/support/chrome_path" File.write(original_path, browser.process.path) @@ -45,7 +43,7 @@ end end - it "supports :logger argument" do + it "supports :logger argument", local_browser: true do browser = Ferrum::Browser.new(logger: logger) browser.go_to(base_url("/console_log")) expect(logger.string).to include("Hello world") @@ -56,7 +54,7 @@ browser&.quit end - it "supports :ignore_default_browser_options argument" do + it "supports :ignore_default_browser_options argument", local_browser: true do defaults = Ferrum::Browser::Options::Chrome.options.except("mute-audio") browser = Ferrum::Browser.new(ignore_default_browser_options: true, browser_options: defaults.merge("no-sandbox" => nil)) @@ -94,7 +92,7 @@ end context ":extensions argument" do - it "extends the browser's world with files" do + it "extends the browser's world with files", local_browser: true do browser = Ferrum::Browser.new(base_url: base_url, extensions: [File.expand_path("support/geolocation.js", __dir__)]) @@ -115,7 +113,7 @@ browser&.quit end - it "extends the browser's world with source" do + it "extends the browser's world with source", local_browser: true do browser = Ferrum::Browser.new(base_url: base_url, extensions: [{ source: "window.secret = 'top'" }]) @@ -126,7 +124,7 @@ browser&.quit end - it "errors when extension is unavailable" do + it "errors when extension is unavailable", local_browser: true do browser = Ferrum::Browser.new(extensions: [File.expand_path("../support/non_existent.js", __dir__)]) expect { browser.go_to }.to raise_error(Errno::ENOENT) ensure @@ -134,7 +132,7 @@ end end - it "supports :port argument" do + it "supports :port argument", local_browser: true do browser = Ferrum::Browser.new(port: 12_345) browser.go_to(base_url) @@ -143,7 +141,7 @@ browser&.quit end - it "supports :url argument" do + it "supports :url argument", local_browser: true do with_external_browser do |url, process| browser = Ferrum::Browser.new(url: url) browser.go_to(base_url) @@ -216,7 +214,7 @@ def proxy_tunnel(socket, request, browser_url) upstream&.close end - it "supports :url argument with a query string" do + it "supports :url argument with a query string", local_browser: true do with_external_browser do |url, _process| with_token_authenticated_proxy(url, token: "secret") do |proxy_url| browser = Ferrum::Browser.new(url: proxy_url) @@ -231,7 +229,7 @@ def proxy_tunnel(socket, request, browser_url) end end - it "raises when :url doesn't answer with a websocket url" do + it "raises when :url doesn't answer with a websocket url", local_browser: true do with_external_browser do |url, _process| with_token_authenticated_proxy(url, token: "secret") do |proxy_url| unauthorized = proxy_url.sub("secret", "wrong") @@ -242,7 +240,7 @@ def proxy_tunnel(socket, request, browser_url) end end - it "supports :ws_url argument" do + it "supports :ws_url argument", local_browser: true do with_external_browser do |url, process| browser = Ferrum::Browser.new(ws_url: web_socket_debugger_url(url)) expect(process.v8_version).not_to be_nil @@ -277,7 +275,7 @@ def proxy_tunnel(socket, request, browser_url) after { proxy.stop } context "without authorization" do - it "works without authorization" do + it "works without authorization", local_browser: true do browser = Ferrum::Browser.new( proxy: { host: proxy.host, port: proxy.port } ) @@ -293,7 +291,7 @@ def proxy_tunnel(socket, request, browser_url) context "with authorization" do let(:options) { Hash(user: "user", password: "pa$$") } - it "works with right password" do + it "works with right password", local_browser: true do browser = Ferrum::Browser.new( proxy: { host: proxy.host, port: proxy.port, **options } ) @@ -305,7 +303,7 @@ def proxy_tunnel(socket, request, browser_url) browser&.quit end - it "breaks with wrong password" do + it "breaks with wrong password", local_browser: true do browser = Ferrum::Browser.new( proxy: { host: proxy.host, port: proxy.port, user: "u1", password: "p1" } ) @@ -345,7 +343,7 @@ def proxy_tunnel(socket, request, browser_url) end end - it "supports :pending_connection_errors argument" do + it "supports :pending_connection_errors argument", local_browser: true do browser = Ferrum::Browser.new(base_url: base_url, pending_connection_errors: true, timeout: 0.5) expect { browser.go_to("/really_slow") }.to raise_error(Ferrum::PendingConnectionsError) @@ -383,7 +381,7 @@ def proxy_tunnel(socket, request, browser_url) context "with local path" do let(:save_path) { "spec/tmp" } - it "raises an error" do + it "raises an error", local_browser: true do expect do browser.go_to("/#{filename}") end.to raise_error(Ferrum::Error, "supply absolute path for `:save_path` option") @@ -392,14 +390,14 @@ def proxy_tunnel(socket, request, browser_url) end context "with error on initialize" do - it "stops process if an error is raised during process start" do + it "stops process if an error is raised during process start", local_browser: true do process = Ferrum::Browser::Process.new(Ferrum::Browser::Options.new(process_timeout: 0)) allow(Ferrum::Browser::Process).to receive(:new).and_return(process) expect { Ferrum::Browser.new }.to raise_error(Ferrum::ProcessTimeoutError) expect(process.pid).to be(nil) end - it "stops process if an error is raised during client creation" do + it "stops process if an error is raised during client creation", local_browser: true do process = Ferrum::Browser::Process.new(Ferrum::Browser::Options.new) allow(Ferrum::Browser::Process).to receive(:new).and_return(process) @@ -410,7 +408,7 @@ def proxy_tunnel(socket, request, browser_url) expect(process.pid).to be(nil) end - it "stops process if an error is raised during contexts creation" do + it "stops process if an error is raised during contexts creation", local_browser: true do process = Ferrum::Browser::Process.new(Ferrum::Browser::Options.new) allow(Ferrum::Browser::Process).to receive(:new).and_return(process) @@ -468,14 +466,14 @@ def proxy_tunnel(socket, request, browser_url) end describe "#quit" do - it "stops silently before go_to call" do + it "stops silently before go_to call", local_browser: true do browser = Ferrum::Browser.new expect { browser.quit }.not_to raise_error ensure browser&.quit end - it "supports stopping the session", skip: Ferrum::Utils::Platform.windows? do + it "supports stopping the session", skip: Ferrum::Utils::Platform.windows?, local_browser: true do browser = Ferrum::Browser.new pid = browser.process.pid @@ -485,7 +483,7 @@ def proxy_tunnel(socket, request, browser_url) expect { Process.kill(0, pid) }.to raise_error(Errno::ESRCH) end - it "returns immediately with wait: false, finishing cleanup once the returned thread is joined", skip: Ferrum::Utils::Platform.windows? do + it "returns immediately with wait: false, finishing cleanup once the returned thread is joined", skip: Ferrum::Utils::Platform.windows?, local_browser: true do browser = Ferrum::Browser.new pid = browser.process.pid @@ -503,7 +501,7 @@ def proxy_tunnel(socket, request, browser_url) end describe "#restart" do - it "blocks until the old process is confirmed dead before returning", skip: Ferrum::Utils::Platform.windows? do + it "blocks until the old process is confirmed dead before returning", skip: Ferrum::Utils::Platform.windows?, local_browser: true do browser = Ferrum::Browser.new old_pid = browser.process.pid @@ -576,7 +574,7 @@ def proxy_tunnel(socket, request, browser_url) end describe "#evaluate_on_new_document" do - it "supports evaluation of JavaScript before page loads" do + it "supports evaluation of JavaScript before page loads", local_browser: true do browser = Ferrum::Browser.new(base_url: base_url) browser.evaluate_on_new_document <<~JS @@ -785,7 +783,7 @@ def proxy_tunnel(socket, request, browser_url) end describe "#debug_url" do - it "parses the devtools frontend url correctly when devtoolsFrontendUrl is relative" do + it "parses the devtools frontend url correctly when devtoolsFrontendUrl is relative", local_browser: true do browser = Ferrum::Browser.new(port: 12_345) uri = instance_double(URI) @@ -795,7 +793,7 @@ def proxy_tunnel(socket, request, browser_url) expect(browser.send(:debug_url)).to eq("http://127.0.0.1:12345/works") end - it "parses the devtools frontend url correctly when devtoolsFrontendUrl is fully qualified" do + it "parses the devtools frontend url correctly when devtoolsFrontendUrl is fully qualified", local_browser: true do browser = Ferrum::Browser.new(port: 12_346) uri = instance_double(URI) @@ -808,7 +806,7 @@ def proxy_tunnel(socket, request, browser_url) end end - context "with pty", if: Ferrum::Utils::Platform.mri? && !Ferrum::Utils::Platform.windows? do + context "with pty", local_browser: true, if: Ferrum::Utils::Platform.mri? && !Ferrum::Utils::Platform.windows? do require "pty" require "timeout" diff --git a/spec/client_spec.rb b/spec/client_spec.rb index c70b426e..b9735129 100644 --- a/spec/client_spec.rb +++ b/spec/client_spec.rb @@ -17,7 +17,7 @@ def messages remote.client.instance_variable_get(:@ws).messages end - it "releases the commands in flight" do + it "releases the commands in flight", local_browser: true do page = remote.create_page Thread.new do sleep(0.2) @@ -29,7 +29,7 @@ def messages expect(Ferrum::Utils::ElapsedTime.elapsed_time(start)).to be < 5 end - it "fails the commands that follow without waiting" do + it "fails the commands that follow without waiting", local_browser: true do page = remote.create_page messages.close start = Ferrum::Utils::ElapsedTime.monotonic_time @@ -38,7 +38,7 @@ def messages expect(Ferrum::Utils::ElapsedTime.elapsed_time(start)).to be < 1 end - it "closes the socket when the browser dies before answering the close frame" do + it "closes the socket when the browser dies before answering the close frame", local_browser: true do remote.create_page socket = remote.client.instance_variable_get(:@ws).instance_variable_get(:@sock) Process.kill("KILL", -remote.process.pid) @@ -48,7 +48,7 @@ def messages expect(socket).to be_closed end - it "reports an unexpected reader error instead of raising in the main thread" do + it "reports an unexpected reader error instead of raising in the main thread", local_browser: true do page = remote.create_page driver = remote.client.instance_variable_get(:@ws).instance_variable_get(:@driver) allow(driver).to receive(:parse).and_raise(Errno::ETIMEDOUT) @@ -76,7 +76,7 @@ def evaluate_in_background(page, expression) end end - it "gives up on a command the browser doesn't take" do + it "gives up on a command the browser doesn't take", local_browser: true do page = remote.create_page Process.kill("STOP", remote.process.pid) @@ -87,7 +87,7 @@ def evaluate_in_background(page, expression) expect(sender.value.message).to match(/accepted no data for 1s/) end - it "fails the commands waiting to be sent behind it" do + it "fails the commands waiting to be sent behind it", local_browser: true do page = remote.create_page Process.kill("STOP", remote.process.pid) @@ -105,7 +105,7 @@ def evaluate_in_background(page, expression) after { remote.quit } - it "keeps dispatching events when a callback raises" do + it "keeps dispatching events when a callback raises", local_browser: true do page = remote.create_page expect do diff --git a/spec/contexts_spec.rb b/spec/contexts_spec.rb index 436d036e..728128fe 100644 --- a/spec/contexts_spec.rb +++ b/spec/contexts_spec.rb @@ -25,7 +25,7 @@ end describe "#default_context" do - it "works in the browser's startup window when it came up with one" do + it "works in the browser's startup window when it came up with one", local_browser: true do with_external_browser(incognito: false) do |url| remote = Ferrum::Browser.new(url: url) @@ -36,7 +36,7 @@ end end - it "creates a context of its own when the browser has no startup window" do + it "creates a context of its own when the browser has no startup window", local_browser: true do with_external_browser do |url| remote = Ferrum::Browser.new(url: url) remote.create_page @@ -51,7 +51,7 @@ end end - it "ignores contexts another client created in the same browser" do + it "ignores contexts another client created in the same browser", local_browser: true do with_external_browser do |url| first = Ferrum::Browser.new(url: url) context = first.contexts.create diff --git a/spec/cookies_spec.rb b/spec/cookies_spec.rb index 4abf4271..6e1b4f86 100644 --- a/spec/cookies_spec.rb +++ b/spec/cookies_spec.rb @@ -217,7 +217,7 @@ expect(browser.body).to include("127.0.0.1") end - it "sets cookies correctly with :domain option when base_url isn't set" do + it "sets cookies correctly with :domain option when base_url isn't set", local_browser: true do browser = Ferrum::Browser.new browser.cookies.set(name: "stealth", value: "123456", domain: "localhost") diff --git a/spec/downloads_spec.rb b/spec/downloads_spec.rb index 7cd33879..7564e37a 100644 --- a/spec/downloads_spec.rb +++ b/spec/downloads_spec.rb @@ -71,7 +71,7 @@ end context "with the browser's implicit context" do - it "saves an attachment" do + it "saves an attachment", local_browser: true do with_external_browser(incognito: false) do |url| remote = Ferrum::Browser.new(url: url, base_url: base_url, save_path: save_path) remote_page = remote.create_page diff --git a/spec/frame/runtime_spec.rb b/spec/frame/runtime_spec.rb index f8406eeb..32e238b6 100644 --- a/spec/frame/runtime_spec.rb +++ b/spec/frame/runtime_spec.rb @@ -16,7 +16,7 @@ after { browser.quit } - it "propagates a Javascript error to a ruby exception" do + it "propagates a Javascript error to a ruby exception", local_browser: true do expect do browser.execute(%(throw new Error("zomg"))) end.to raise_error(Ferrum::JavaScriptError) { |e| @@ -24,7 +24,7 @@ } end - it "propagates an asynchronous Javascript error on the page to a ruby exception" do + it "propagates an asynchronous Javascript error on the page to a ruby exception", local_browser: true do expect do browser.execute "setTimeout(function() { omg }, 0)" sleep 0.01 @@ -32,13 +32,13 @@ end.to raise_error(Ferrum::JavaScriptError, /ReferenceError.*omg/) end - it "propagates a synchronous Javascript error on the page to a ruby exception" do + it "propagates a synchronous Javascript error on the page to a ruby exception", local_browser: true do expect do browser.execute "omg" end.to raise_error(Ferrum::JavaScriptError, /ReferenceError.*omg/) end - it "does not re-raise a Javascript error if it is rescued" do + it "does not re-raise a Javascript error if it is rescued", local_browser: true do expect do browser.execute "setTimeout(function() { omg }, 0)" sleep 0.01 @@ -49,11 +49,11 @@ expect(browser.evaluate("1+1")).to eq(2) end - it "propagates a Javascript error during page load to a ruby exception" do + it "propagates a Javascript error during page load to a ruby exception", local_browser: true do expect { browser.go_to("/js_error") }.to raise_error(Ferrum::JavaScriptError) end - it "does not propagate a Javascript error to ruby if error raising disabled" do + it "does not propagate a Javascript error to ruby if error raising disabled", local_browser: true do browser = Ferrum::Browser.new(base_url: base_url, js_errors: false) browser.go_to("/js_error") browser.execute "setTimeout(function() { omg }, 0)" @@ -63,7 +63,7 @@ browser&.quit end - it "does not propagate a Javascript error to ruby if error raising disabled and client restarted" do + it "does not propagate a Javascript error to ruby if error raising disabled and client restarted", local_browser: true do browser = Ferrum::Browser.new(base_url: base_url, js_errors: false) browser.restart browser.go_to("/js_error") diff --git a/spec/page/screenshot_spec.rb b/spec/page/screenshot_spec.rb index efd3ec95..6d48ead9 100644 --- a/spec/page/screenshot_spec.rb +++ b/spec/page/screenshot_spec.rb @@ -489,7 +489,7 @@ def create_screenshot(path:, **options) expect(browser.viewport_size).to eq([1024, 681]) end - it "supports specifying viewport size with an option" do + it "supports specifying viewport size with an option", local_browser: true do browser = Ferrum::Browser.new(window_size: [800, 600]) browser.go_to(base_url) expect(browser.viewport_size).to eq([800, 513]) diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index d4e147cb..a724876d 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -33,6 +33,7 @@ puts "" RSpec.configure do |config| + config.filter_run_excluding local_browser: true if remote_url config.wait_timeout = 1 config.wait_delay = 0.2 From 30c800e2e10c0a0db6b9a3f2fda082d24006939a Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 21:04:56 +0300 Subject: [PATCH 22/23] chore(ci): check for new browser versions daily and build only the browsers that changed A scheduled run compares the newest Chrome (Google's apt repository) and Chromium (Debian's apt-cache policy) with the tags on GHCR. The version is a build arg, so a new release busts the layer cache, and the image is labeled with the driver's commit. --- .github/workflows/docker.yml | 60 +++++++++++++++++++++++++++++++++--- docker/DESIGN.md | 12 ++++++-- docker/Dockerfile | 4 +++ 3 files changed, 70 insertions(+), 6 deletions(-) diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index ebae3852..6c54f226 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -10,8 +10,14 @@ on: paths: - "docker/**" - ".github/workflows/docker.yml" + schedule: + - cron: "0 6 * * *" workflow_dispatch: +concurrency: + group: docker-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + jobs: test: name: Go tests @@ -39,13 +45,57 @@ jobs: - name: Test run: go test -race ./... + versions: + name: Browser versions + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + packages: read + outputs: + versions: ${{ steps.versions.outputs.versions }} + browsers: ${{ steps.versions.outputs.browsers }} + steps: + - name: Login to registry + if: github.event_name == 'schedule' + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Find the newest versions + id: versions + env: + OWNER: ${{ github.repository_owner }} + EVENT: ${{ github.event_name }} + run: | + chrome=$(curl -fsS https://dl.google.com/linux/chrome/deb/dists/stable/main/binary-amd64/Packages \ + | awk '/^Package: google-chrome-stable$/ { found = 1 } found && /^Version:/ { print $2; exit }' | sed 's/-.*//') + chromium=$(docker run --rm debian:trixie-slim sh -c 'apt-get update -qq && apt-cache policy chromium' \ + | awk '/Candidate:/ { print $2 }' | sed 's/-.*//') + echo "chrome $chrome, chromium $chromium" + + browsers=() + for browser in chrome chromium; do + version=${!browser} + if [ "$EVENT" = "schedule" ] && docker manifest inspect "ghcr.io/$OWNER/$browser:$version" > /dev/null 2>&1; then + echo "$browser $version is already published" + else + browsers+=("\"$browser\"") + fi + done + + echo "versions={\"chrome\":\"$chrome\",\"chromium\":\"$chromium\"}" >> "$GITHUB_OUTPUT" + echo "browsers=[$(IFS=,; echo "${browsers[*]}")]" >> "$GITHUB_OUTPUT" + build: name: Build ${{ matrix.browser }} (${{ matrix.arch }}) - needs: test + needs: [test, versions] + if: needs.versions.outputs.browsers != '[]' strategy: fail-fast: false matrix: - browser: [chrome, chromium] + browser: ${{ fromJSON(needs.versions.outputs.browsers) }} arch: [amd64, arm64] include: - arch: amd64 @@ -81,6 +131,7 @@ jobs: platforms: linux/${{ matrix.arch }} build-args: | BROWSER=${{ matrix.browser }} + BROWSER_VERSION=${{ fromJSON(needs.versions.outputs.versions)[matrix.browser] }} VERSION=${{ github.sha }} load: true tags: ${{ matrix.browser }}:test @@ -137,6 +188,7 @@ jobs: platforms: linux/${{ matrix.arch }} build-args: | BROWSER=${{ matrix.browser }} + BROWSER_VERSION=${{ fromJSON(needs.versions.outputs.versions)[matrix.browser] }} VERSION=${{ github.sha }} cache-from: type=gha,scope=${{ matrix.browser }}-${{ matrix.arch }} outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true @@ -160,10 +212,10 @@ jobs: publish: name: Publish ${{ matrix.browser }} if: github.event_name != 'pull_request' - needs: build + needs: [versions, build] strategy: matrix: - browser: [chrome, chromium] + browser: ${{ fromJSON(needs.versions.outputs.browsers) }} runs-on: ubuntu-latest permissions: packages: write diff --git a/docker/DESIGN.md b/docker/DESIGN.md index 227fa452..0cd6b546 100644 --- a/docker/DESIGN.md +++ b/docker/DESIGN.md @@ -171,12 +171,20 @@ under it, which crashpad needs (ferrum#541, #544). Downloads then belong to the Ubuntu's `chromium` package is a snap, which doesn't work in Docker. - Two images, `ghcr.io/rubycdp/chrome` (Google's `.deb`) and `ghcr.io/rubycdp/chromium` (Debian's package), each for amd64 and arm64. Tags: `latest`, the browser's major version (`155`) and its full version (`155.0.8059.39`). + The driver's commit is in the `org.opencontainers.image.revision` label. - The Go binary is cross-compiled in a build stage, so the image has no Node, Playwright or npm. - `.github/workflows/docker.yml` runs the Go tests, then builds each browser and architecture pair on a native runner, never through QEMU, which crashes Chrome (ferrum#270). - It smoke-tests every image. - On amd64, it runs the Ferrum suite against the image, with `FERRUM_URL` pointing at the driver. - - It merges the architectures into one multi-arch tag by digest. + - It merges the architectures into one multi-arch tag by digest. A failed build or suite publishes nothing, so + `latest` stays where it was. +- Builds happen on driver changes pushed to `main` and on a daily check for new browser versions: Google's apt + repository for Chrome, `apt-cache policy chromium` for Chromium. The check builds a browser only when its + version isn't published yet. +- Only the current browser version gets builds. Older tags stay on GHCR as they were, with the driver of their + time: Google and Debian only serve the newest packages, so an old version can't be rebuilt. Rebuilding the same + version with a newer driver moves its tags; pin a digest to keep an exact image. - Fonts: Debian's font packages plus open substitutes for Microsoft fonts. Selawik replaces Segoe UI, Carlito replaces Calibri, Caladea replaces Cambria, and DejaVu replaces Tahoma. - Env: @@ -233,7 +241,7 @@ Done: - `/health` and the image's `HEALTHCHECK`, trailing-slash routes, `driver --help` and `--version`. - A graceful shutdown stops every Chrome. - Go tests (`go test ./...`), with the test binary acting as a fake Chrome, run in CI before any image is built. -- The Ferrum suite runs against the amd64 images in CI. +- The Ferrum suite runs against the amd64 images in CI, and a daily check builds new browser versions. Not done yet: - the Ferrum suite on arm64, whose runner has no local Chrome for the specs that start one; diff --git a/docker/Dockerfile b/docker/Dockerfile index d3f1d49e..47bf0ae6 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -58,18 +58,22 @@ COPY deploy/fonts.conf /etc/fonts/local.conf RUN fc-cache -f FROM base AS chromium +ARG BROWSER_VERSION RUN apt-get update && apt-get install -y chromium \ && rm -rf /var/lib/apt/lists/* ENV BROWSER_PATH=/usr/bin/chromium FROM base AS chrome ARG TARGETARCH +ARG BROWSER_VERSION RUN curl -fsSL --retry 5 --retry-all-errors -o /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_${TARGETARCH}.deb \ && apt-get update && apt-get install -y /tmp/chrome.deb \ && rm -rf /var/lib/apt/lists/* /tmp/chrome.deb ENV BROWSER_PATH=/usr/bin/google-chrome FROM ${BROWSER} +ARG VERSION=dev +LABEL org.opencontainers.image.revision=${VERSION} RUN mkdir -p /home/driver /tmp/.X11-unix && chmod 1777 /home/driver /tmp/.X11-unix \ && useradd --uid 999 --home-dir /home/driver --no-create-home --shell /usr/sbin/nologin driver From 46f8b9786789c7d7b82aca7b1c71b531b146367d Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sat, 17 Oct 2026 21:04:56 +0300 Subject: [PATCH 23/23] test(ci): run the Ferrum suite against the arm64 images too and drop the smoke test --- .github/workflows/docker.yml | 15 +-------------- docker/DESIGN.md | 7 +++---- 2 files changed, 4 insertions(+), 18 deletions(-) diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 6c54f226..b1e8f532 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -138,38 +138,25 @@ jobs: cache-from: type=gha,scope=${{ matrix.browser }}-${{ matrix.arch }} cache-to: type=gha,mode=max,scope=${{ matrix.browser }}-${{ matrix.arch }} - - name: Smoke test - run: | - docker run -d --name smoke -p 3000:3000 ${{ matrix.browser }}:test - for i in $(seq 1 30); do - curl -sf http://localhost:3000/json/version | grep -q webSocketDebuggerUrl && break - sleep 1 - done - docker logs smoke - curl -sf http://localhost:3000/json/version | tee /dev/stderr | grep -q webSocketDebuggerUrl - - name: Setup Ruby - if: matrix.arch == 'amd64' uses: ruby/setup-ruby@v1 with: ruby-version: "3.4" bundler-cache: true - name: Run the Ferrum suite against the image - if: matrix.arch == 'amd64' timeout-minutes: 20 env: FERRUM_DEFAULT_TIMEOUT: 15 FERRUM_PROCESS_TIMEOUT: 25 run: | - docker rm -f smoke docker run -d --name suite --network host --user "$(id -u):$(id -g)" -e CONCURRENT=4 \ -v "$GITHUB_WORKSPACE:$GITHUB_WORKSPACE" -v /tmp:/tmp ${{ matrix.browser }}:test for i in $(seq 1 30); do curl -sf http://127.0.0.1:3000/health && break; sleep 1; done FERRUM_URL=http://127.0.0.1:3000 bundle exec rspec --exclude-pattern "spec/unit/**/*_spec.rb" - name: Driver logs - if: always() && matrix.arch == 'amd64' + if: always() run: docker logs suite 2>&1 | grep -v "browser stderr" | tail -200 || true - name: Detect browser version diff --git a/docker/DESIGN.md b/docker/DESIGN.md index 0cd6b546..f92f465e 100644 --- a/docker/DESIGN.md +++ b/docker/DESIGN.md @@ -175,8 +175,8 @@ under it, which crashpad needs (ferrum#541, #544). Downloads then belong to the - The Go binary is cross-compiled in a build stage, so the image has no Node, Playwright or npm. - `.github/workflows/docker.yml` runs the Go tests, then builds each browser and architecture pair on a native runner, never through QEMU, which crashes Chrome (ferrum#270). - - It smoke-tests every image. - - On amd64, it runs the Ferrum suite against the image, with `FERRUM_URL` pointing at the driver. + - It runs the Ferrum suite against every image, with `FERRUM_URL` pointing at the driver. Specs tagged + `local_browser`, which start their own Chrome, are skipped. - It merges the architectures into one multi-arch tag by digest. A failed build or suite publishes nothing, so `latest` stays where it was. - Builds happen on driver changes pushed to `main` and on a daily check for new browser versions: Google's apt @@ -241,9 +241,8 @@ Done: - `/health` and the image's `HEALTHCHECK`, trailing-slash routes, `driver --help` and `--version`. - A graceful shutdown stops every Chrome. - Go tests (`go test ./...`), with the test binary acting as a fake Chrome, run in CI before any image is built. -- The Ferrum suite runs against the amd64 images in CI, and a daily check builds new browser versions. +- The Ferrum suite runs against every image in CI, and a daily check builds new browser versions. Not done yet: -- the Ferrum suite on arm64, whose runner has no local Chrome for the specs that start one; - the zero-copy tunnel; - the Ferrum changes above.