From 93c050dc46a5e0e576375865a790ebcfb38aeb94 Mon Sep 17 00:00:00 2001 From: Dmitry Vorotilin Date: Sun, 18 Oct 2026 19:21:10 +0300 Subject: [PATCH] feat(keyboard): hold modifiers for keyboard and mouse input inside a block Chrome doesn't carry a pressed modifier over to later input events, so a shift+drag needed raw CDP commands. Keyboard#hold presses the given modifiers, makes every keyboard and mouse event in the block carry them and releases them when the block returns or raises. State lives only for the block, so a missing keyup can't leak into later input. Keyboard#down and #up raise ArgumentError for a modifier and point to hold: down used to crash with NoMethodError and up sent an empty keyup event. Mouse#move accepts modifiers: like down and up, and Mouse#click sends its modifiers with the preceding mousemove too. Refs #581. --- CHANGELOG.md | 5 ++ docs/10-mouse-keyboard.md | 44 ++++++++++++---- lib/ferrum/keyboard.rb | 103 +++++++++++++++++++++++++++++++++++--- lib/ferrum/mouse.rb | 24 ++++++--- sig/ferrum/keyboard.rbs | 17 +++++++ sig/ferrum/mouse.rbs | 2 +- spec/keyboard_spec.rb | 83 ++++++++++++++++++++++++++++++ spec/mouse_spec.rb | 17 +++++++ 8 files changed, 269 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bc64b01..8e94696c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,12 @@ ## [Unreleased](https://github.com/rubycdp/ferrum/compare/v0.19.1...main) ## ### Added +- `keyboard.hold(:shift) { }` holds modifiers for every keyboard and mouse event in the block, e.g. a shift+drag, and + releases them afterwards. `mouse.move` accepts `modifiers:` like `down` and `up` [#581] ### Changed +- `keyboard.down` and `keyboard.up` raise `ArgumentError` for a modifier such as `:shift` and point to `hold`. `down` + used to crash with `NoMethodError`, `up` sent an empty keyup event [#581] - Behind a proxy with `:user` and `:password`, Ferrum stops continuing intercepted requests once you subscribe to `on(:request)`, like it already does without a proxy: your callbacks have to continue or abort every request, or it stays paused until the page times out [#332] @@ -20,6 +24,7 @@ on `DOM.documentUpdated` and such nodes raise `Ferrum::NodeNotFoundError` [#86] - Behind a proxy with credentials, `request.abort` in `on(:request)` and `network.blacklist`/`whitelist` were ignored because Ferrum's own proxy handler continued every request first. It now only continues requests nobody else handles [#332] +- `mouse.click(modifiers:)` sent its `mousemove` without the modifiers [#581] ### Removed - `Browser::Process#default_user_agent`, `#browser_version`, `#protocol_version`, `#v8_version` and `#webkit_version`, diff --git a/docs/10-mouse-keyboard.md b/docs/10-mouse-keyboard.md index bf450f53..4ed7b390 100644 --- a/docs/10-mouse-keyboard.md +++ b/docs/10-mouse-keyboard.md @@ -32,7 +32,8 @@ Click given coordinates, fires mouse move, down and up events. * :delay `Float` defaults to 0. Delay between mouse down and mouse up events * :button `Symbol` :left | :right, defaults to :left * :count `Integer` defaults to 1 - * :modifiers `Integer` bitfield for key modifiers. See`keyboard.modifiers` + * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. + If omitted, uses the keys held by `keyboard.hold`, or none outside of it #### down(\*\*options) : `Mouse` @@ -41,7 +42,8 @@ Mouse down for given coordinates. * options `Hash` * :button `Symbol` :left | :right, defaults to :left * :count `Integer` defaults to 1 - * :modifiers `Integer` bitfield for key modifiers. See`keyboard.modifiers` + * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. + If omitted, uses the keys held by `keyboard.hold`, or none outside of it #### up(\*\*options) : `Mouse` @@ -50,9 +52,10 @@ Mouse up for given coordinates. * options `Hash` * :button `Symbol` :left | :right, defaults to :left * :count `Integer` defaults to 1 - * :modifiers `Integer` bitfield for key modifiers. See`keyboard.modifiers` + * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. + If omitted, uses the keys held by `keyboard.hold`, or none outside of it -#### move(x:, y:, steps: 1) : `Mouse` +#### move(x:, y:, steps: 1, modifiers: nil) : `Mouse` Mouse move to given x and y. @@ -60,6 +63,8 @@ Mouse move to given x and y. * :x `Integer` * :y `Integer` * :steps `Integer` defaults to 1. Sends intermediate mousemove events. + * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. + If omitted, uses the keys held by `keyboard.hold`, or none outside of it ## Keyboard @@ -69,13 +74,15 @@ Mouse move to given x and y. Dispatches a keydown event. -* key `String` | `Symbol` Name of key such as "a", :enter, :backspace +* key `String` | `Symbol` Name of key such as "a", :enter, :backspace. Raises `ArgumentError` for a modifier, + use `hold` instead #### up(key) : `Keyboard` Dispatches a keyup event. -* key `String` | `Symbol` Name of key such as "b", :enter, :backspace +* key `String` | `Symbol` Name of key such as "b", :enter, :backspace. Raises `ArgumentError` for a modifier, + use `hold` instead #### type(\*keys) : `Keyboard` @@ -84,6 +91,27 @@ Sends a keydown, keypress/input, and keyup event for each character in the text. * text `String` | `Array | Array` A text to type into a focused element, `[:Shift, "s"], "tring"` +#### hold(\*keys) { } : `Object` + +Holds modifier keys down while the block runs. Every keyboard and mouse event inside the block carries them, and they +are released when the block returns or raises. Returns the block's value. + +* keys `Symbol` :alt | :ctrl | :control | :meta | :command | :shift + +```ruby +page.keyboard.hold(:shift) do + page.mouse.move(x: 10, y: 10).down.move(x: 200, y: 200, steps: 10).up +end +``` + +#### held_modifiers : `Integer` + +Modifier keys currently pressed by `hold`, in the same format as `modifiers` returns. It's 0 outside a `hold` block. + +```ruby +page.keyboard.hold(:shift) { page.keyboard.held_modifiers } # => 8 +``` + #### modifiers(keys) : `Integer` Returns bitfield for a given keys @@ -103,7 +131,5 @@ page.keyboard.type("Hello World") page.keyboard.type(:Enter) # Use keyboard shortcuts -page.keyboard.down(:Shift) -page.keyboard.type("h", "e", "l", "l", "o") -page.keyboard.up(:Shift) +page.keyboard.hold(:shift) { page.keyboard.type("hello") } ``` diff --git a/lib/ferrum/keyboard.rb b/lib/ferrum/keyboard.rb index 6370e404..63d386c0 100644 --- a/lib/ferrum/keyboard.rb +++ b/lib/ferrum/keyboard.rb @@ -32,8 +32,15 @@ class Keyboard f10: "F10", f11: "F11", f12: "F12", meta: "Meta", command: "Meta" }.freeze + # Modifier keys currently pressed by {#hold}, in the same format as + # {#modifiers} returns. It's 0 outside a {#hold} block. + # + # @return [Integer] + attr_reader :held_modifiers + def initialize(page) @page = page + @held_modifiers = 0 end # @@ -44,7 +51,10 @@ def initialize(page) # # @return [self] # + # @raise [ArgumentError] If the key is a modifier, use {#hold} instead. + # def down(key) + reject_modifier(key) key = normalize_keys(Array(key)).first type = key[:text] ? "keyDown" : "rawKeyDown" @page.command("Input.dispatchKeyEvent", slowmoable: true, type: type, **key) @@ -59,7 +69,10 @@ def down(key) # # @return [self] # + # @raise [ArgumentError] If the key is a modifier, use {#hold} instead. + # def up(key) + reject_modifier(key) key = normalize_keys(Array(key)).first @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key) self @@ -86,6 +99,44 @@ def type(*keys) self end + # + # Holds modifier keys down while the block runs: every keyboard and mouse + # event dispatched inside it carries them, and they are released when the + # block returns or raises. + # + # @param [Array<:alt, :ctrl, :control, :meta, :command, :shift>] keys + # + # @yield The input to dispatch with the modifiers held. + # + # @return [Object] The block's value. + # + # @raise [ArgumentError] If no block is given or a key isn't a modifier. + # + # @example + # page.keyboard.hold(:shift) do + # page.mouse.move(x: 10, y: 10).down.move(x: 200, y: 200, steps: 10).up + # end + # + def hold(*keys) + raise ArgumentError, "block is required" unless block_given? + + keys = keys.map { |key| modifier_name(key) || raise(ArgumentError, "not a modifier key: #{key.inspect}") } + previous = @held_modifiers + pressed = [] + + begin + keys.each { |key| press_modifier(key, pressed) } + result = yield + release_modifiers(pressed) + result + rescue StandardError + release_modifiers(pressed, quietly: true) + raise + ensure + @held_modifiers = previous + end + end + # # Returns bitfield for a given keys. # @@ -99,6 +150,42 @@ def modifiers(keys) private + def modifier_name(key) + name = key.to_s.downcase + name if MODIFIERS.key?(name) + end + + def reject_modifier(key) + return unless key.is_a?(Symbol) && modifier_name(key) + + raise ArgumentError, "#{key.inspect} is a modifier, use keyboard.hold(#{key.inspect}) { ... } instead" + end + + def press_modifier(name, pressed) + bit = MODIFIERS[name] + return if @held_modifiers.anybits?(bit) + + @held_modifiers |= bit + dispatch_modifier("rawKeyDown", name) + pressed.push(name) + end + + def release_modifiers(pressed, quietly: false) + while (name = pressed.pop) + @held_modifiers &= ~MODIFIERS[name] + begin + dispatch_modifier("keyUp", name) + rescue Error + raise unless quietly + end + end + end + + def dispatch_modifier(type, name) + key = to_options(KEYS.fetch(KEYS_MAPPING[name.to_sym])) + @page.command("Input.dispatchKeyEvent", slowmoable: true, type: type, modifiers: @held_modifiers, **key) + end + # rubocop:disable Metrics/PerceivedComplexity # rubocop:disable Metrics/CyclomaticComplexity def normalize_keys(keys, pressed_keys = [], memo = []) @@ -120,33 +207,34 @@ def normalize_keys(keys, pressed_keys = [], memo = []) nil else key = KEYS.fetch(KEYS_MAPPING[key.to_sym] || key.to_sym) - key[:modifiers] = pressed_keys.flatten.map { |k| MODIFIERS[k] }.reduce(0, :|) + key[:modifiers] = pressed_keys.flatten.map { |k| MODIFIERS[k] }.reduce(@held_modifiers, :|) to_options(key) end when String raise ArgumentError, "empty keys passed" if keys.empty? pressed = pressed_keys.flatten + mask = pressed.map { |k| MODIFIERS[k] }.reduce(@held_modifiers, :|) keys.each_char.map do |char| key = KEYS[char] || {} - if pressed.empty? + if mask.zero? key = key.merge(text: char, unmodifiedText: char) [to_options(key)] else - text = pressed == ["shift"] ? char.upcase : char + text = mask == MODIFIERS["shift"] ? char.upcase : char key = key.merge( text: text, unmodifiedText: text, isKeypad: key["location"] == 3, - modifiers: pressed.map { |k| MODIFIERS[k] }.reduce(0, :|) + modifiers: mask ) # Synthetic key events alone do not trigger the browser's editing # shortcuts (e.g. select-all). When the chord matches a platform # accelerator, name the editing command explicitly so Chrome # executes it via the CDP `commands` field. - command = editing_command(pressed, char) + command = editing_command(mask, char) key = key.merge(commands: [command]) if command modifiers = pressed.map { |k| to_options(KEYS.fetch(KEYS_MAPPING[k.to_sym])) } @@ -175,12 +263,11 @@ def to_options(hash) # is not an editing accelerator. The accelerator modifier is Cmd on macOS # and Ctrl elsewhere, matching Chrome's own key bindings. Currently only # select-all is mapped; extend the guard for further commands as needed. - def editing_command(pressed, char) + def editing_command(mask, char) return unless char.casecmp?("a") - return if pressed.include?("shift") accelerator = Utils::Platform.mac? ? MODIFIERS["meta"] : MODIFIERS["ctrl"] - "selectAll" if pressed.map { |k| MODIFIERS[k] }.compact.uniq == [accelerator] + "selectAll" if mask == accelerator end end end diff --git a/lib/ferrum/mouse.rb b/lib/ferrum/mouse.rb index f34985e5..355f2536 100644 --- a/lib/ferrum/mouse.rb +++ b/lib/ferrum/mouse.rb @@ -81,12 +81,13 @@ def scroll_to(top, left) # @option options [Integer] :count (1) # # @option options [Integer] :modifiers - # Bitfield for key modifiers. See`keyboard.modifiers`. + # Bitfield for key modifiers. See `keyboard.modifiers`. + # If omitted, uses the keys held by `keyboard.hold`, or none outside of it. # # @return [self] # def click(x:, y:, delay: 0, wait: CLICK_WAIT, **options) - move(x: x, y: y) + move(x: x, y: y, modifiers: options[:modifiers]) down(**options) sleep(delay) # Potential wait because if some network event is triggered then we have @@ -107,7 +108,8 @@ def click(x:, y:, delay: 0, wait: CLICK_WAIT, **options) # @option options [Integer] :count (1) # # @option options [Integer] :modifiers - # Bitfield for key modifiers. See`keyboard.modifiers`. + # Bitfield for key modifiers. See `keyboard.modifiers`. + # If omitted, uses the keys held by `keyboard.hold`, or none outside of it. # # @return [self] # @@ -127,7 +129,8 @@ def down(**options) # @option options [Integer] :count (1) # # @option options [Integer] :modifiers - # Bitfield for key modifiers. See`keyboard.modifiers`. + # Bitfield for key modifiers. See `keyboard.modifiers`. + # If omitted, uses the keys held by `keyboard.hold`, or none outside of it. # # @return [self] # @@ -145,9 +148,13 @@ def up(**options) # @param [Integer] steps # Sends intermediate mousemove events. # + # @param [Integer, nil] modifiers + # Bitfield for key modifiers. See `keyboard.modifiers`. + # If omitted, uses the keys held by `keyboard.hold`, or none outside of it. + # # @return [self] # - def move(x:, y:, steps: 1) + def move(x:, y:, steps: 1, modifiers: nil) from_x = @x from_y = @y @x = x @@ -164,7 +171,8 @@ def move(x:, y:, steps: 1) x: new_x, y: new_y, button: button, - buttons: @buttons) + buttons: @buttons, + modifiers: modifiers || @page.keyboard.held_modifiers) end self @@ -175,8 +183,8 @@ def move(x:, y:, steps: 1) def mouse_event(type:, button: :left, count: 1, modifiers: nil, wait: 0) button = validate_button(button) register_event_button(type, button) - options = { x: @x, y: @y, type: type, button: button, buttons: @buttons, clickCount: count } - options.merge!(modifiers: modifiers) if modifiers + options = { x: @x, y: @y, type: type, button: button, buttons: @buttons, clickCount: count, + modifiers: modifiers || @page.keyboard.held_modifiers } @page.command("Input.dispatchMouseEvent", wait: wait, slowmoable: true, **options) end diff --git a/sig/ferrum/keyboard.rbs b/sig/ferrum/keyboard.rbs index 2ce38756..57f1a4a6 100644 --- a/sig/ferrum/keyboard.rbs +++ b/sig/ferrum/keyboard.rbs @@ -5,6 +5,9 @@ module Ferrum KEYS_MAPPING: Hash[Symbol, String] @page: Page + @held_modifiers: ::Integer + + attr_reader held_modifiers: ::Integer def initialize: (Page page) -> void @@ -14,14 +17,28 @@ module Ferrum def type: (*(String | Symbol | ::Array[String | Symbol])) -> self + def hold: [T] (*(String | Symbol) keys) { () -> T } -> T + def modifiers: (::Array[Symbol]) -> ::Integer private + def modifier_name: (String | Symbol key) -> String? + + def reject_modifier: (String | Symbol | ::Array[String | Symbol] key) -> void + + def press_modifier: (String name, ::Array[String] pressed) -> void + + def release_modifiers: (::Array[String] pressed, ?quietly: bool) -> void + + def dispatch_modifier: (String type, String name) -> void + def normalize_keys: ((::Array[untyped] | Symbol | String) keys, ?::Array[::Array[String]] pressed_keys, ?::Array[Hash[Symbol, untyped]] memo) -> ::Array[Hash[Symbol, untyped]] def combine_strings: (::Array[untyped] keys) -> ::Array[untyped] def to_options: (Hash[String, untyped] hash) -> Hash[Symbol, untyped] + + def editing_command: (::Integer mask, String char) -> String? end end diff --git a/sig/ferrum/mouse.rbs b/sig/ferrum/mouse.rbs index 7d87eeeb..5c9050e3 100644 --- a/sig/ferrum/mouse.rbs +++ b/sig/ferrum/mouse.rbs @@ -21,7 +21,7 @@ module Ferrum def up: (?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer, ?wait: ::Numeric) -> self - def move: (x: ::Numeric, y: ::Numeric, ?steps: ::Integer) -> self + def move: (x: ::Numeric, y: ::Numeric, ?steps: ::Integer, ?modifiers: ::Integer?) -> self private diff --git a/spec/keyboard_spec.rb b/spec/keyboard_spec.rb index 17ef08a0..0b24b0bb 100644 --- a/spec/keyboard_spec.rb +++ b/spec/keyboard_spec.rb @@ -11,6 +11,10 @@ expect { browser.keyboard.down(:page_down) }.not_to raise_error expect { browser.keyboard.down(1) }.to raise_error(ArgumentError, "unexpected argument") end + + it "rejects a modifier key" do + expect { browser.keyboard.down(:shift) }.to raise_error(ArgumentError, /use keyboard.hold\(:shift\)/) + end end describe "#up" do @@ -23,5 +27,84 @@ expect { browser.keyboard.up(:page_down) }.not_to raise_error expect { browser.keyboard.up(1) }.to raise_error(ArgumentError, "unexpected argument") end + + it "rejects a modifier key" do + expect { browser.keyboard.up(:Control) }.to raise_error(ArgumentError, /use keyboard.hold\(:Control\)/) + end + end + + describe "#hold" do + before do + browser.go_to("/type") + browser.execute(<<~JS) + window.events = []; + ["keydown", "keyup", "mousemove", "mousedown", "mouseup"].forEach(type => + document.addEventListener(type, e => events.push(`${type}:${e.shiftKey}:${e.ctrlKey}`))); + JS + end + + it "holds a modifier for mouse events and releases it after the block" do + browser.keyboard.hold(:shift) do + browser.mouse.move(x: 10, y: 10).down.move(x: 50, y: 50, steps: 2).up + end + browser.mouse.move(x: 60, y: 60) + + expect(browser.evaluate("events")).to eq( + %w[keydown:true:false mousemove:true:false mousedown:true:false mousemove:true:false + mousemove:true:false mouseup:true:false keyup:false:false mousemove:false:false] + ) + expect(browser.keyboard.held_modifiers).to eq(0) + end + + it "presses nested modifiers once" do + browser.keyboard.hold(:shift) do + browser.keyboard.hold(:shift, :ctrl) { browser.mouse.move(x: 10, y: 10) } + browser.mouse.move(x: 20, y: 20) + end + + expect(browser.evaluate("events")).to eq( + %w[keydown:true:false keydown:true:true mousemove:true:true keyup:true:false + mousemove:true:false keyup:false:false] + ) + end + + it "releases modifiers when the block raises" do + expect do + browser.keyboard.hold(:shift) { raise "boom" } + end.to raise_error(RuntimeError, "boom") + browser.mouse.move(x: 10, y: 10) + + expect(browser.evaluate("events")).to eq(%w[keydown:true:false keyup:false:false mousemove:false:false]) + expect(browser.keyboard.held_modifiers).to eq(0) + end + + it "returns the block value" do + expect(browser.keyboard.hold(:shift) { 42 }).to eq(42) + end + + it "types with the held modifier" do + input = browser.at_css("#empty_input") + + input.focus + browser.keyboard.hold(:shift) { browser.keyboard.type("abc") } + + expect(input.value).to eq("ABC") + end + + it "selects all with the held command modifier" do + input = browser.at_css("#filled_input") + modifier = Ferrum::Utils::Platform.mac? ? :meta : :ctrl + + input.focus + browser.keyboard.hold(modifier) { browser.keyboard.type("a") } + browser.keyboard.type(:backspace) + + expect(input.value).to eq("") + end + + it "rejects keys that are not modifiers" do + expect { browser.keyboard.hold(:enter) { nil } }.to raise_error(ArgumentError, "not a modifier key: :enter") + expect { browser.keyboard.hold(:shift) }.to raise_error(ArgumentError, "block is required") + end end end diff --git a/spec/mouse_spec.rb b/spec/mouse_spec.rb index df683516..3b292ff8 100644 --- a/spec/mouse_spec.rb +++ b/spec/mouse_spec.rb @@ -57,6 +57,23 @@ end describe "#move" do + it "sends modifiers" do + browser.go_to("/simple") + browser.execute(<<~JS) + window.events = []; + ["mousemove", "mousedown", "mouseup"].forEach(type => + document.addEventListener(type, e => events.push(`${type}:${e.shiftKey}:${e.altKey}`))); + JS + + browser.mouse.move(x: 10, y: 10, modifiers: 8) + browser.keyboard.hold(:shift) { browser.mouse.move(x: 20, y: 20, modifiers: 1) } + browser.mouse.click(x: 30, y: 30, modifiers: 8) + + expect(browser.evaluate("events")).to eq( + %w[mousemove:true:false mousemove:false:true mousemove:true:false mousedown:true:false mouseup:true:false] + ) + end + it "drags and drops with a button held" do browser.go_to("/html5_drag")