diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f2503b2..2bc64b01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,9 @@ ### Added ### Changed +- Behind a proxy with `:user` and `:password`, Ferrum stops continuing intercepted requests once you subscribe to + `on(:request)`, like it already does without a proxy: your callbacks have to continue or abort every request, or it + stays paused until the page times out [#332] - Browser details (user agent, versions) come from `Browser.getVersion` over CDP for every browser, local or remote. `/json/version` is only asked for the websocket url when `url:` is given, so `ws_url:` works against services without an HTTP endpoint and a local launch makes no HTTP call @@ -15,6 +18,8 @@ - A `Ferrum::Node` found before a navigation or reload kept its cached CDP node id, which Chrome reuses for the new document, so `focus`, `click`, `scroll_into_view` and friends silently hit another element. Node ids are now dropped on `DOM.documentUpdated` and such nodes raise `Ferrum::NodeNotFoundError` [#86] +- Behind a proxy with credentials, `request.abort` in `on(:request)` and `network.blacklist`/`whitelist` were ignored + because Ferrum's own proxy handler continued every request first. It now only continues requests nobody else handles [#332] ### Removed - `Browser::Process#default_user_agent`, `#browser_version`, `#protocol_version`, `#v8_version` and `#webkit_version`, diff --git a/docs/7-network.md b/docs/7-network.md index cbbcc08e..e3c4b71f 100644 --- a/docs/7-network.md +++ b/docs/7-network.md @@ -133,6 +133,9 @@ Set request interception for given options. This method is only sets request interception, you should use `on` callback to catch requests and abort or continue them. +Every paused request has to be continued, aborted or responded to by one of your `on(:request)` callbacks, including +ones that only log or inspect it. A request nobody handles stays paused, and the page waits for it until it times out. + * options `Hash` * :pattern `String` \* by default * :resource_type `Symbol` one of the [resource types](https://chromedevtools.github.io/devtools-protocol/tot/Network#type-ResourceType) diff --git a/docs/9-proxy.md b/docs/9-proxy.md index 6dc1501f..6d0c6103 100644 --- a/docs/9-proxy.md +++ b/docs/9-proxy.md @@ -34,6 +34,25 @@ browser.create_page(proxy: { host: "y.y.y.y", port: 31337, user: "user", passwor end ``` +## Intercepting requests behind a proxy + +A proxy with `:user` and `:password` is authorized through request interception, so Ferrum continues every request +for you as long as nothing else subscribes to `on(:request)`. Once you subscribe, or set `network.blacklist` or +`network.whitelist`, your code takes over and has to continue or abort every request: + +```ruby +browser = Ferrum::Browser.new(proxy: { host: "x.x.x.x", port: "8800", user: "user", password: "pa$$" }) +browser.network.intercept +browser.on(:request) do |request| + if request.resource_type == "Image" + request.abort + else + request.continue + end +end +browser.go_to("https://github.com/") +``` + ## Built-in proxy server Ferrum also ships a small forwarding proxy server, `Ferrum::Proxy`, which is handy diff --git a/lib/ferrum/page.rb b/lib/ferrum/page.rb index 3ed20a3e..6d51c09f 100644 --- a/lib/ferrum/page.rb +++ b/lib/ferrum/page.rb @@ -569,8 +569,8 @@ def prepare_page if use_authorized_proxy? network.authorize(user: @proxy_user, password: @proxy_password, - type: :proxy) do |request, _index, _total| - request.continue + type: :proxy) do |request, _index, total| + request.continue if total == 1 end end diff --git a/lib/ferrum/proxy.rb b/lib/ferrum/proxy.rb index dfed4284..a7bac649 100644 --- a/lib/ferrum/proxy.rb +++ b/lib/ferrum/proxy.rb @@ -1,5 +1,6 @@ # frozen_string_literal: true +require "logger" require "tempfile" begin diff --git a/spec/browser_spec.rb b/spec/browser_spec.rb index 68831265..2c34f71a 100644 --- a/spec/browser_spec.rb +++ b/spec/browser_spec.rb @@ -342,6 +342,35 @@ def proxy_tunnel(socket, request, browser_url) ensure browser&.quit end + + it "lets requests be aborted", local_browser: true do + browser = Ferrum::Browser.new( + proxy: { host: proxy.host, port: proxy.port, **options } + ) + browser.network.intercept + browser.on(:request) { |request| request.match?(/example\.com/) ? request.abort : request.continue } + + expect { browser.go_to("https://example.com") }.to raise_error( + Ferrum::StatusError, + "Request to https://example.com failed (net::ERR_BLOCKED_BY_CLIENT)" + ) + ensure + browser&.quit + end + + it "supports blacklist", local_browser: true do + browser = Ferrum::Browser.new( + proxy: { host: proxy.host, port: proxy.port, **options } + ) + browser.network.blacklist = [/example\.com/] + + expect { browser.go_to("https://example.com") }.to raise_error( + Ferrum::StatusError, + "Request to https://example.com failed (net::ERR_BLOCKED_BY_CLIENT)" + ) + ensure + browser&.quit + end end context "with rotation", skip: "Think how to make it working on CI" do