feat(keyboard): hold modifiers for keyboard and mouse input inside a block

CHANGELOG.md

@@ -1,8 +1,12 @@
 ## [Unreleased](https://github.com/rubycdp/ferrum/compare/v0.19.1...main) ##
 
 ### Added
+- `keyboard.hold(:shift) { }` holds modifiers for every keyboard and mouse event in the block, e.g. a shift+drag, and
+  releases them afterwards. `mouse.move` accepts `modifiers:` like `down` and `up` [#581]
 
 ### Changed
+- `keyboard.down` and `keyboard.up` raise `ArgumentError` for a modifier such as `:shift` and point to `hold`. `down`
+  used to crash with `NoMethodError`, `up` sent an empty keyup event [#581]
 - Behind a proxy with `:user` and `:password`, Ferrum stops continuing intercepted requests once you subscribe to
   `on(:request)`, like it already does without a proxy: your callbacks have to continue or abort every request, or it
   stays paused until the page times out [#332]
@@ -20,6 +24,7 @@
   on `DOM.documentUpdated` and such nodes raise `Ferrum::NodeNotFoundError` [#86]
 - Behind a proxy with credentials, `request.abort` in `on(:request)` and `network.blacklist`/`whitelist` were ignored
   because Ferrum's own proxy handler continued every request first. It now only continues requests nobody else handles [#332]
+- `mouse.click(modifiers:)` sent its `mousemove` without the modifiers [#581]
 
 ### Removed
 - `Browser::Process#default_user_agent`, `#browser_version`, `#protocol_version`, `#v8_version` and `#webkit_version`,

docs/10-mouse-keyboard.md

@@ -32,7 +32,8 @@ Click given coordinates, fires mouse move, down and up events.
     * :delay `Float` defaults to 0. Delay between mouse down and mouse up events
     * :button `Symbol` :left | :right, defaults to :left
     * :count `Integer` defaults to 1
-    * :modifiers `Integer` bitfield for key modifiers. See`keyboard.modifiers`
+    * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`.
+      If omitted, uses the keys held by `keyboard.hold`, or none outside of it
 
 #### down(\*\*options) : `Mouse`
 
@@ -41,7 +42,8 @@ Mouse down for given coordinates.
 * options `Hash`
     * :button `Symbol` :left | :right, defaults to :left
     * :count `Integer` defaults to 1
-    * :modifiers `Integer` bitfield for key modifiers. See`keyboard.modifiers`
+    * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`.
+      If omitted, uses the keys held by `keyboard.hold`, or none outside of it
 
 #### up(\*\*options) : `Mouse`
 
@@ -50,16 +52,19 @@ Mouse up for given coordinates.
 * options `Hash`
     * :button `Symbol` :left | :right, defaults to :left
     * :count `Integer` defaults to 1
-    * :modifiers `Integer` bitfield for key modifiers. See`keyboard.modifiers`
+    * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`.
+      If omitted, uses the keys held by `keyboard.hold`, or none outside of it
 
-#### move(x:, y:, steps: 1) : `Mouse`
+#### move(x:, y:, steps: 1, modifiers: nil) : `Mouse`
 
 Mouse move to given x and y.
 
 * options `Hash`
     * :x `Integer`
     * :y `Integer`
     * :steps `Integer` defaults to 1. Sends intermediate mousemove events.
+    * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`.
+      If omitted, uses the keys held by `keyboard.hold`, or none outside of it
 
 ## Keyboard
 
@@ -69,13 +74,15 @@ Mouse move to given x and y.
 
 Dispatches a keydown event.
 
-* key `String` | `Symbol` Name of key such as "a", :enter, :backspace
+* key `String` | `Symbol` Name of key such as "a", :enter, :backspace. Raises `ArgumentError` for a modifier,
+  use `hold` instead
 
 #### up(key) : `Keyboard`
 
 Dispatches a keyup event.
 
-* key `String` | `Symbol` Name of key such as "b", :enter, :backspace
+* key `String` | `Symbol` Name of key such as "b", :enter, :backspace. Raises `ArgumentError` for a modifier,
+  use `hold` instead
 
 #### type(\*keys) : `Keyboard`
 
@@ -84,6 +91,27 @@ Sends a keydown, keypress/input, and keyup event for each character in the text.
 * text `String` | `Array<String> | Array<Symbol>` A text to type into a focused
   element, `[:Shift, "s"], "tring"`
 
+#### hold(\*keys) { } : `Object`
+
+Holds modifier keys down while the block runs. Every keyboard and mouse event inside the block carries them, and they
+are released when the block returns or raises. Returns the block's value.
+
+* keys `Symbol` :alt | :ctrl | :control | :meta | :command | :shift
+
+```ruby
+page.keyboard.hold(:shift) do
+  page.mouse.move(x: 10, y: 10).down.move(x: 200, y: 200, steps: 10).up
+end
+```
+
+#### held_modifiers : `Integer`
+
+Modifier keys currently pressed by `hold`, in the same format as `modifiers` returns. It's 0 outside a `hold` block.
+
+```ruby
+page.keyboard.hold(:shift) { page.keyboard.held_modifiers } # => 8
+```
+
 #### modifiers(keys) : `Integer`
 
 Returns bitfield for a given keys
@@ -103,7 +131,5 @@ page.keyboard.type("Hello World")
 page.keyboard.type(:Enter)
 
 # Use keyboard shortcuts
-page.keyboard.down(:Shift)
-page.keyboard.type("h", "e", "l", "l", "o")
-page.keyboard.up(:Shift)
+page.keyboard.hold(:shift) { page.keyboard.type("hello") }
 ```

lib/ferrum/keyboard.rb

@@ -32,8 +32,15 @@ class Keyboard
       f10: "F10", f11: "F11", f12: "F12", meta: "Meta", command: "Meta"
     }.freeze
 
+    # Modifier keys currently pressed by {#hold}, in the same format as
+    # {#modifiers} returns. It's 0 outside a {#hold} block.
+    #
+    # @return [Integer]
+    attr_reader :held_modifiers
+
     def initialize(page)
       @page = page
+      @held_modifiers = 0
     end
 
     #
@@ -44,7 +51,10 @@ def initialize(page)
     #
     # @return [self]
     #
+    # @raise [ArgumentError] If the key is a modifier, use {#hold} instead.
+    #
     def down(key)
+      reject_modifier(key)
       key = normalize_keys(Array(key)).first
       type = key[:text] ? "keyDown" : "rawKeyDown"
       @page.command("Input.dispatchKeyEvent", slowmoable: true, type: type, **key)
@@ -59,7 +69,10 @@ def down(key)
     #
     # @return [self]
     #
+    # @raise [ArgumentError] If the key is a modifier, use {#hold} instead.
+    #
     def up(key)
+      reject_modifier(key)
       key = normalize_keys(Array(key)).first
       @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key)
       self
@@ -86,6 +99,44 @@ def type(*keys)
       self
     end
 
+    #
+    # Holds modifier keys down while the block runs: every keyboard and mouse
+    # event dispatched inside it carries them, and they are released when the
+    # block returns or raises.
+    #
+    # @param [Array<:alt, :ctrl, :control, :meta, :command, :shift>] keys
+    #
+    # @yield The input to dispatch with the modifiers held.
+    #
+    # @return [Object] The block's value.
+    #
+    # @raise [ArgumentError] If no block is given or a key isn't a modifier.
+    #
+    # @example
+    #   page.keyboard.hold(:shift) do
+    #     page.mouse.move(x: 10, y: 10).down.move(x: 200, y: 200, steps: 10).up
+    #   end
+    #
+    def hold(*keys)
+      raise ArgumentError, "block is required" unless block_given?
+
+      keys = keys.map { |key| modifier_name(key) || raise(ArgumentError, "not a modifier key: #{key.inspect}") }
+      previous = @held_modifiers
+      pressed = []
+
+      begin
+        keys.each { |key| press_modifier(key, pressed) }
+        result = yield
+        release_modifiers(pressed)
+        result
+      rescue StandardError
+        release_modifiers(pressed, quietly: true)
+        raise
+      ensure
+        @held_modifiers = previous
+      end
+    end
+
     #
     # Returns bitfield for a given keys.
     #
@@ -99,6 +150,42 @@ def modifiers(keys)
 
     private
 
+    def modifier_name(key)
+      name = key.to_s.downcase
+      name if MODIFIERS.key?(name)
+    end
+
+    def reject_modifier(key)
+      return unless key.is_a?(Symbol) && modifier_name(key)
+
+      raise ArgumentError, "#{key.inspect} is a modifier, use keyboard.hold(#{key.inspect}) { ... } instead"
+    end
+
+    def press_modifier(name, pressed)
+      bit = MODIFIERS[name]
+      return if @held_modifiers.anybits?(bit)
+
+      @held_modifiers |= bit
+      dispatch_modifier("rawKeyDown", name)
+      pressed.push(name)
+    end
+
+    def release_modifiers(pressed, quietly: false)
+      while (name = pressed.pop)
+        @held_modifiers &= ~MODIFIERS[name]
+        begin
+          dispatch_modifier("keyUp", name)
+        rescue Error
+          raise unless quietly
+        end
+      end
+    end
+
+    def dispatch_modifier(type, name)
+      key = to_options(KEYS.fetch(KEYS_MAPPING[name.to_sym]))
+      @page.command("Input.dispatchKeyEvent", slowmoable: true, type: type, modifiers: @held_modifiers, **key)
+    end
+
     # rubocop:disable Metrics/PerceivedComplexity
     # rubocop:disable Metrics/CyclomaticComplexity
     def normalize_keys(keys, pressed_keys = [], memo = [])
@@ -120,33 +207,34 @@ def normalize_keys(keys, pressed_keys = [], memo = [])
           nil
         else
           key = KEYS.fetch(KEYS_MAPPING[key.to_sym] || key.to_sym)
-          key[:modifiers] = pressed_keys.flatten.map { |k| MODIFIERS[k] }.reduce(0, :|)
+          key[:modifiers] = pressed_keys.flatten.map { |k| MODIFIERS[k] }.reduce(@held_modifiers, :|)
           to_options(key)
         end
       when String
         raise ArgumentError, "empty keys passed" if keys.empty?
 
         pressed = pressed_keys.flatten
+        mask = pressed.map { |k| MODIFIERS[k] }.reduce(@held_modifiers, :|)
         keys.each_char.map do |char|
           key = KEYS[char] || {}
 
-          if pressed.empty?
+          if mask.zero?
             key = key.merge(text: char, unmodifiedText: char)
             [to_options(key)]
           else
-            text = pressed == ["shift"] ? char.upcase : char
+            text = mask == MODIFIERS["shift"] ? char.upcase : char
             key = key.merge(
               text: text,
               unmodifiedText: text,
               isKeypad: key["location"] == 3,
-              modifiers: pressed.map { |k| MODIFIERS[k] }.reduce(0, :|)
+              modifiers: mask
             )
 
             # Synthetic key events alone do not trigger the browser's editing
             # shortcuts (e.g. select-all). When the chord matches a platform
             # accelerator, name the editing command explicitly so Chrome
             # executes it via the CDP `commands` field.
-            command = editing_command(pressed, char)
+            command = editing_command(mask, char)
             key = key.merge(commands: [command]) if command
 
             modifiers = pressed.map { |k| to_options(KEYS.fetch(KEYS_MAPPING[k.to_sym])) }
@@ -175,12 +263,11 @@ def to_options(hash)
     # is not an editing accelerator. The accelerator modifier is Cmd on macOS
     # and Ctrl elsewhere, matching Chrome's own key bindings. Currently only
     # select-all is mapped; extend the guard for further commands as needed.
-    def editing_command(pressed, char)
+    def editing_command(mask, char)
       return unless char.casecmp?("a")
-      return if pressed.include?("shift")
 
       accelerator = Utils::Platform.mac? ? MODIFIERS["meta"] : MODIFIERS["ctrl"]
-      "selectAll" if pressed.map { |k| MODIFIERS[k] }.compact.uniq == [accelerator]
+      "selectAll" if mask == accelerator
     end
   end
 end

lib/ferrum/mouse.rb

@@ -81,12 +81,13 @@ def scroll_to(top, left)
     # @option options [Integer] :count (1)
     #
     # @option options [Integer] :modifiers
-    #   Bitfield for key modifiers. See`keyboard.modifiers`.
+    #   Bitfield for key modifiers. See `keyboard.modifiers`.
+    #   If omitted, uses the keys held by `keyboard.hold`, or none outside of it.
     #
     # @return [self]
     #
     def click(x:, y:, delay: 0, wait: CLICK_WAIT, **options)
-      move(x: x, y: y)
+      move(x: x, y: y, modifiers: options[:modifiers])
       down(**options)
       sleep(delay)
       # Potential wait because if some network event is triggered then we have
@@ -107,7 +108,8 @@ def click(x:, y:, delay: 0, wait: CLICK_WAIT, **options)
     # @option options [Integer] :count (1)
     #
     # @option options [Integer] :modifiers
-    #   Bitfield for key modifiers. See`keyboard.modifiers`.
+    #   Bitfield for key modifiers. See `keyboard.modifiers`.
+    #   If omitted, uses the keys held by `keyboard.hold`, or none outside of it.
     #
     # @return [self]
     #
@@ -127,7 +129,8 @@ def down(**options)
     # @option options [Integer] :count (1)
     #
     # @option options [Integer] :modifiers
-    #   Bitfield for key modifiers. See`keyboard.modifiers`.
+    #   Bitfield for key modifiers. See `keyboard.modifiers`.
+    #   If omitted, uses the keys held by `keyboard.hold`, or none outside of it.
     #
     # @return [self]
     #
@@ -145,9 +148,13 @@ def up(**options)
     # @param [Integer] steps
     #   Sends intermediate mousemove events.
     #
+    # @param [Integer, nil] modifiers
+    #   Bitfield for key modifiers. See `keyboard.modifiers`.
+    #   If omitted, uses the keys held by `keyboard.hold`, or none outside of it.
+    #
     # @return [self]
     #
-    def move(x:, y:, steps: 1)
+    def move(x:, y:, steps: 1, modifiers: nil)
       from_x = @x
       from_y = @y
       @x = x
@@ -164,7 +171,8 @@ def move(x:, y:, steps: 1)
                       x: new_x,
                       y: new_y,
                       button: button,
-                      buttons: @buttons)
+                      buttons: @buttons,
+                      modifiers: modifiers || @page.keyboard.held_modifiers)
       end
 
       self
@@ -175,8 +183,8 @@ def move(x:, y:, steps: 1)
     def mouse_event(type:, button: :left, count: 1, modifiers: nil, wait: 0)
       button = validate_button(button)
       register_event_button(type, button)
-      options = { x: @x, y: @y, type: type, button: button, buttons: @buttons, clickCount: count }
-      options.merge!(modifiers: modifiers) if modifiers
+      options = { x: @x, y: @y, type: type, button: button, buttons: @buttons, clickCount: count,
+                  modifiers: modifiers || @page.keyboard.held_modifiers }
       @page.command("Input.dispatchMouseEvent", wait: wait, slowmoable: true, **options)
     end