Skip to content

FelixClements/ClickFlow

Repository files navigation

ClickFlow

ClickFlow is a Python desktop automation tool that finds UI elements by matching reference images on screen, then runs actions such as click, type, drag, scroll, wait, and keyboard shortcuts. It is useful for applications and RDP sessions where DOM or API automation is not available because it works from what is visible on the screen.

The project currently includes a CustomTkinter desktop GUI, CLI tools, YAML workflow execution, reference image capture, CSV queue processing, and action recording.

Screenshots

Run Tab

ClickFlow Run tab

Workflows Tab

ClickFlow Workflows tab

Queue Tab

ClickFlow Queue tab

Capture Tab

ClickFlow Capture tab

Contents

Quick Start

pip install -r requirements.txt
python gui/app.py

Recommended first workflow:

  1. Open the GUI with python gui/app.py.
  2. Go to Workflows and create or select a workflow.
  3. Go to Capture and capture the UI elements used by the workflow.
  4. Go to Images to review, rename, delete, or retake reference images.
  5. Go to Run, adjust parameters and overrides, then run the workflow.

How It Works

ClickFlow workflows are YAML files stored under workflows/. Each step usually finds a named image in an images directory, then performs an action at the matched location.

For example, this step looks for images/search_field.png, clicks it, and pastes text:

- description: "Search for user"
  action: type
  image: search_field
  text: "{param:username}"

Reference images are small screenshots of buttons, fields, labels, menu items, or other stable UI regions. Image names are referenced without the file extension.

Desktop GUI

Launch the GUI:

python gui/app.py

The GUI saves window state, selected tab, selected workflow, and theme in ~/.clickflow/config.json.

Tabs

Tab Purpose
Run Select a workflow, enter parameters, set confidence/timeout/delay/images directory, dry-run, save workflow settings, and execute.
Queue Process the active workflow against a CSV queue, with resume/reset, limit, progress, failed-item retry, and progress JSON tracking.
Capture Capture a screen region into the active images directory after an optional countdown.
Images Browse reference images, filter by name, preview metadata, rename, delete, or retake.
Workflows Browse, create, preview, edit, and run YAML workflows. Includes Blank, Template, and Generic Queue starters.
Record Record clicks, typing, scrolling, and key presses into a generated YAML workflow.

Shortcuts

Shortcut Tab
Ctrl+R Run
Ctrl+Shift+Q Queue
Ctrl+Shift+C Capture
Ctrl+Shift+I Images
Ctrl+Shift+W Workflows
Ctrl+Shift+R Record

Workflow Files

Workflow files live in workflows/ and are discovered recursively by the GUI.

Minimal example:

name: "Example Workflow"
description: "Find a field, type text, and confirm."

settings:
  confidence: 0.8
  timeout: 10
  delay: 0.5
  fail_fast: true
  images_dir: ./images

params:
  username: ""

steps:
  - description: "Type username"
    action: type
    image: username_field
    text: "{param:username}"

  - description: "Confirm"
    action: click
    image: confirm_button
    fallback:
      - action: press
        key: enter

See workflows/template.yaml for a broader action template and workflows/generic_queue.yaml for a queue-oriented starter.

Building Workflow Actions

A workflow action is one item under steps:. Most actions follow this pattern:

- description: "Human-readable step name"
  action: click
  image: button_ok
  timeout: 10
  confidence: 0.85

The action field decides which operation ClickFlow runs. The rest of the fields are the inputs for that action. For image-based actions, the image field is the reference image name without .png. For example, image: button_ok points to button_ok.png inside the active images_dir.

Action Building Workflow

  1. Capture the target UI element in the Capture tab or with cli/capture_images.py.
  2. Give the image a stable, descriptive name such as login_button, search_field, or save_menu_item.
  3. Confirm the image is in the workflow's settings.images_dir folder.
  4. Add a step in the Workflows tab or edit the YAML directly.
  5. Pick the action type and fill only the fields that action needs.
  6. Add timeout or confidence on the step only when it needs different matching behavior than the global settings.
  7. Add fallback actions for important steps where a keyboard shortcut can recover from image matching failure.
  8. Test with one or two steps first, then expand the workflow.

Creating Actions In The GUI

The Workflows tab is the main place to create and maintain workflow actions without editing YAML by hand.

To create a workflow and add actions in the GUI:

  1. Open Workflows.
  2. Click + New.
  3. Fill in Name, Description, Template, Confidence, Timeout, Delay, Fail fast, and Images dir.
  4. Choose Blank if you want to build every action yourself, Template for a simple starter workflow, or Generic Queue for a starter that uses queue placeholders.
  5. Click Create.
  6. Select the workflow from the list on the left.
  7. Click + Add Step.
  8. In the step editor, choose the Action type.
  9. Fill in Description and the action-specific fields shown under Action Fields.
  10. Use Browse next to image fields to pick from the workflow's image directory.
  11. Add per-step Timeout or Confidence only when this action needs different matching settings.
  12. Add Fallbacks when there is a keyboard or alternate action that can recover from failure.
  13. Click Save in the step editor.
  14. Click Save Steps in the Workflows tab to write the changes to the YAML file.
  15. Click Run this or switch to the Run tab to test the workflow.

Existing actions appear as step cards. Use Up and Down to reorder them, Edit to change a step, and X to delete a step. Show YAML lets you inspect the generated YAML, but the step editor is usually safer for normal editing.

The action editor only shows fields relevant to the selected action. For example, click shows an image picker and click options, type shows text plus an optional image picker, press shows a key field, and wait or assert show image plus present/absent state.

GUI Action Field Examples

Goal GUI action type Fields to fill
Click a button click Image name = save_button
Type into a field type Image name = search_field, Text to type = {param:search}
Type queue data type Image name = search_field, Text to type = {queue:id}
Press Enter press Key = enter
Use Ctrl+S hotkey Keys = ctrl, s
Wait for spinner to disappear wait Image name = loading_spinner, State = absent
Verify success message assert Image name = success_message, State = present
Pause between actions delay Seconds = 1

When entering multiple hotkey keys in the GUI, separate them with commas, for example ctrl, s or ctrl, shift, esc.

Step Anatomy

Field Required Purpose
description No Label shown in logs and progress output.
action Yes Action name such as click, type, press, wait, or assert.
image Action-dependent Reference image name without .png. Used by actions that need to find something on screen.
text For type and type_slow Text to enter. Can include {param:...} or {queue:...} placeholders.
key For press Key name such as enter, tab, escape, f5, up, or down.
keys For hotkey List of keys, for example [ctrl, s].
timeout No Per-step seconds to wait for an image. Overrides settings.timeout.
confidence No Per-step image match threshold. Overrides settings.confidence.
fallback No One or more backup steps to try if the primary step fails.
id No Stable result key for step results. Useful when debugging.
unless No Basic skip condition. Currently supports checks such as param:name.

Common Action Examples

Click a button:

- description: "Click Save"
  action: click
  image: save_button

Type into a field. type first clicks the optional image, then pastes the text through the clipboard:

- description: "Enter user ID"
  action: type
  image: user_id_field
  text: "{param:user_id}"

Type slowly when clipboard paste is blocked:

- description: "Enter code slowly"
  action: type_slow
  image: code_field
  text: "{param:code}"
  interval: 0.05

Press a single key without image matching:

- description: "Confirm dialog"
  action: press
  key: enter

Use a keyboard shortcut:

- description: "Save with Ctrl+S"
  action: hotkey
  keys: [ctrl, s]

Wait until loading disappears:

- description: "Wait for loading spinner"
  action: wait
  image: loading_spinner
  state: absent
  timeout: 30

Assert that the expected result is visible:

- description: "Verify success"
  action: assert
  image: success_message
  state: present
  timeout: 10

Drag from one image to another:

- description: "Move item to target"
  action: drag
  image_from: draggable_item
  image_to: drop_target
  duration: 0.5

Add a fallback when image matching might fail:

- description: "Open File menu"
  action: click
  image: file_menu
  timeout: 5
  fallback:
    - action: hotkey
      keys: [alt, f]
    - action: press
      key: enter

Choosing Good Reference Images

Capture the smallest stable region that uniquely identifies the target. Good examples are a button with its label, a field label plus a bit of the field border, or a unique icon with nearby text. Avoid blank corners, generic borders, or large screenshots that include changing content. If two screens use similar controls, put each app's images in its own folder and set settings.images_dir for that workflow.

Queue Processing

ClickFlow can run the same workflow once per row in a CSV file. This is useful when the same screen process must be repeated for many users, IDs, files, tickets, products, or other records.

The queue runner reads one CSV row, injects the row values into the workflow as placeholders, runs the workflow, records progress, then moves to the next row.

Queue CSV Basics

A queue CSV must have a header row. Each header becomes a placeholder that can be used in workflow steps.

id,name,department,action,status
USR001,John Smith,Engineering,activate,pending
USR002,Jane Doe,Marketing,deactivate,pending
USR003,Bob Wilson,Finance,update_role,pending

In the workflow, those columns are available as:

CSV column Placeholder forms Example value
id {queue:id} or {queue.id} USR001
name {queue:name} or {queue.name} John Smith
department {queue:department} or {queue.department} Engineering
action {queue:action} or {queue.action} activate

The first CSV column is used as the unique item key unless key_column is set. The key is what appears in the progress table and progress JSON. Pick a stable unique column such as id, ticket, filename, or account_number.

The status column is optional. If present, rows marked completed, skipped, or done are ignored by the queue loader unless you reset progress. Rows with pending or blank status are processed.

Queue Data Versus Reference Images

Queue values are normally used for fields like text, description, or filenames. The image field should usually stay fixed because it points to a reference screenshot file.

Good:

- action: type
  image: search_field
  text: "{queue:id}"

Avoid this unless you intentionally captured separate images for every possible CSV value:

- action: click
  image: "{queue:button_name}"

GUI Queue Setup

The GUI Queue tab reads queue settings from the active workflow's queue: block. CSV paths are resolved from the queues/ directory, and .csv is appended automatically if omitted.

To set up a GUI queue:

  1. Create a CSV file under queues/, for example queues/users.csv.
  2. Add a queue: block to the workflow or create the workflow from the Workflows tab with a Queue CSV value.
  3. Use {queue:column_name} placeholders in the workflow steps.
  4. Select the workflow in the GUI.
  5. Open the Queue tab and click Refresh to preview rows.
  6. Enter a small Limit such as 1 for the first test run.
  7. Click PROCESS QUEUE.
  8. Use Resume after an interruption, Reset to start over, and Retry for failed rows.

Creating Queue CSVs In The GUI

The Workflows tab can create the workflow and the queue CSV starter for you.

To create a queued workflow from the GUI:

  1. Open Workflows.
  2. Click + New.
  3. Enter the workflow name and choose Generic Queue if you want queue-ready starter actions.
  4. Set Images dir to the folder where this workflow's reference images will live.
  5. In Queue CSV, either type a CSV name such as users.csv or click Create CSV.
  6. In Create Queue CSV, enter a filename and column names such as id,name,department,action.
  7. Choose the Key column, usually id or another unique value.
  8. Click Create in the CSV dialog. ClickFlow writes the CSV under queues/ and fills the workflow's Queue CSV and Key column fields.
  9. Set CSV delimiter, On error, Max retries, and Rate limit (s).
  10. Click Create in the workflow form.
  11. Edit the generated steps so they use your actual reference images and queue placeholders.
  12. Open Queue, click Refresh, check the row preview, set Limit to 1, then click PROCESS QUEUE.

The CSV dialog creates the header row only. Add queue rows in the CSV file afterward using Excel, a text editor, or another export process. The GUI Queue tab will preview one pending item per row after you click Refresh.

Queue options in the GUI map to the workflow YAML like this:

GUI field YAML field Meaning
Queue CSV queue.csv CSV file under queues/.
Key column queue.key_column Unique row identifier used in progress tracking.
CSV delimiter queue.delimiter Separator used by the CSV file.
On error queue.on_error Stop at first failed row or continue.
Max retries queue.max_retries Attempts per row before marking failed.
Rate limit (s) queue.rate_limit Delay between processed rows.

Using Queue Values In GUI Actions

Queue placeholders can be typed directly into action fields in the GUI step editor. If your CSV has a column named id, type {queue:id} into a text field. If it has department, type {queue:department}. These values are replaced separately for each row while the queue runs.

Typical GUI step editor values for a queued action:

Action type Field Value
type Image name search_field
type Text to type {queue:id}
click Image name action_button
assert Image name success_message
assert State present

Keep reference image fields fixed unless you intentionally have many image files named from CSV values. Queue data usually belongs in text fields, descriptions, and output filenames, not in image names.

GUI Queue Configuration

Minimal GUI queue workflow:

name: "Queued Workflow"

queue:
  csv: workflow3.csv
  key_column: id
  delimiter: ","
  on_error: stop
  max_retries: 1
  rate_limit: 0

settings:
  confidence: 0.8
  timeout: 10
  delay: 0.5
  images_dir: ./images/app3

steps:
  - description: "Search for {queue:id}"
    action: type
    image: search_field
    text: "{queue:id}"

  - description: "Confirm"
    action: click
    image: confirm_button

Example CSV in queues/workflow3.csv:

id,name,action
USR001,John Smith,activate
USR002,Jane Doe,deactivate

Progress is stored next to the CSV as <queue-name>.progress.json. Use Resume to skip completed rows, Reset to start over, and Retry on failed rows after a run has stopped.

Queue Configuration Fields

Field Required Used by Purpose
csv Yes for GUI Queue tab GUI CSV file under queues/. .csv may be omitted.
key_column No GUI and CLI equivalent flag CSV column used as each row's unique key. Defaults to first column.
delimiter No GUI and CLI equivalent flag CSV delimiter. Use ,, ;, or tab depending on the file.
on_error No GUI and CLI equivalent flag stop stops at first failed item. continue moves to the next item.
max_retries No GUI and CLI equivalent flag Number of attempts for each row. Defaults to 1.
rate_limit No GUI Seconds to wait between queue items.

Complete Queue Example

CSV file at queues/users.csv:

id,name,department,action,status
USR001,John Smith,Engineering,activate,pending
USR002,Jane Doe,Marketing,deactivate,pending

Workflow file at workflows/process_users.yaml:

name: "Process Users"
description: "Search each user from queues/users.csv and apply an action."

queue:
  csv: users.csv
  key_column: id
  delimiter: ","
  on_error: stop
  max_retries: 2
  rate_limit: 1

settings:
  confidence: 0.8
  timeout: 10
  delay: 0.5
  images_dir: ./images/users_app

steps:
  - description: "Search for {queue:id} - {queue:name}"
    action: type
    image: search_field
    text: "{queue:id}"
    timeout: 5
    fallback:
      - action: hotkey
        keys: [ctrl, f]
      - action: type_slow
        text: "{queue:id}"

  - description: "Open selected user"
    action: double_click
    image: search_result_row
    timeout: 10

  - description: "Apply requested action: {queue:action}"
    action: click
    image: action_button
    fallback:
      - action: press
        key: enter

  - description: "Wait for save to finish"
    action: wait
    image: saving_spinner
    state: absent
    timeout: 30

  - description: "Verify success for {queue:id}"
    action: assert
    image: success_message
    state: present
    timeout: 10

  - description: "Pause before next queue row"
    action: delay
    seconds: 1

For the first run, use a GUI limit of 1 or run the CLI with --limit 1 --dry-run to confirm the queue resolves correctly.

CLI Queue Processing

The CLI runner accepts any CSV path:

python cli/queue_runner.py -w workflows/generic_queue.yaml -q workflows/examples/items.csv --key-column id
python cli/queue_runner.py -w workflows/generic_queue.yaml -q workflows/examples/items.csv --key-column id --resume
python cli/queue_runner.py -w workflows/generic_queue.yaml -q workflows/examples/items.csv --limit 1 --dry-run

Useful queue flags:

Flag Purpose
--resume Skip completed/skipped rows from the progress file.
--reset Ignore existing progress and reprocess rows.
--limit N Process only the first N pending rows.
--key-column id Use a specific CSV column as the unique item key.
`--on-error stop continue`
--max-retries N Retry each item up to N attempts.
--delay N Wait N seconds between items.
--delimiter ; Use a custom CSV delimiter.

CLI Tools

The GUI is the primary interface, but the CLI tools remain available.

Run A Workflow

python cli/automate.py workflows/template.yaml
python cli/automate.py workflows/template.yaml --param my_var=hello
python cli/automate.py workflows/template.yaml --dry-run
python cli/automate.py workflows/template.yaml --debug
python cli/automate.py workflows/template.yaml --confidence 0.7 --timeout 15
python cli/automate.py workflows/template.yaml --no-fail-fast
python cli/automate.py --list-images

Capture Reference Images

python cli/capture_images.py
python cli/capture_images.py --bulk-capture button_ok search_field confirm_button
python cli/capture_images.py --list
python cli/capture_images.py --retake button_ok
python cli/capture_images.py --delete button_old
python cli/capture_images.py --images-dir images/app1

Record Actions

python cli/recorder.py workflows/recorded.yaml
python cli/recorder.py workflows/recorded.yaml --images-dir images/app1

During CLI recording, press F6 to stop and save or Esc to cancel.

Portable Windows Build

ClickFlow can be packaged as a portable Windows zip containing ClickFlow.exe. End users can unzip it and run the app without installing Python.

The project uses PyInstaller in onedir mode because CustomTkinter includes theme, font, and JSON assets that must be shipped with the executable.

Build The Zip

From the repository root on Windows:

powershell -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Clean -Version 0.1.0

The build script will:

  1. Create .venv-build/ if needed.
  2. Install requirements.txt and build-requirements.txt.
  3. Run PyInstaller with ClickFlow.spec.
  4. Create dist/ClickFlow/ with ClickFlow.exe and bundled runtime files.
  5. Copy starter workflows/, images/, and README_FIRST.txt into the release folder.
  6. Create dist/ClickFlow-0.1.0-windows-x64.zip.

Use -Version dev or omit -Version to create dist/ClickFlow-windows-x64.zip.

Zip Layout

ClickFlow-windows-x64.zip
`-- ClickFlow/
    |-- ClickFlow.exe
    |-- README_FIRST.txt
    |-- _internal/
    |-- workflows/
    |-- images/
    |-- queues/
    `-- debug_screenshots/

Users should extract the zip first, then double-click ClickFlow.exe. They should keep workflows/, images/, and queues/ next to the exe because ClickFlow reads and writes those folders at runtime.

Rolling Main Release

The GitHub Actions workflow at .github/workflows/release-latest-main.yml builds a new Windows zip on every push to main and updates one rolling prerelease named latest-main.

This keeps the GitHub Releases page clean:

  • The latest-main tag is moved to the newest main commit.
  • The release title and notes are updated with the latest commit and workflow run.
  • The ClickFlow-latest-main-windows-x64.zip asset is replaced using --clobber.
  • Older main builds are not kept as separate releases.

Use latest-main for internal testing builds. For public/stable releases, create a separate versioned release such as v1.0.0.

Release Smoke Test

Before publishing a zip, test it on a Windows machine that does not have Python installed:

  1. Extract the zip.
  2. Double-click ClickFlow.exe.
  3. Open every tab.
  4. Create a workflow in the GUI.
  5. Capture or browse an image.
  6. Run a dry-run workflow.
  7. Create a queue CSV and process with Limit = 1.
  8. Confirm queues/*.progress.json is written.
  9. Start and stop the recorder.

Unsigned PyInstaller executables can trigger Windows SmartScreen warnings. Code signing is recommended for public distribution.

YAML Reference

Settings

settings:
  confidence: 0.8       # Match threshold from 0.0 to 1.0.
  timeout: 10            # Seconds to wait for each image.
  delay: 0.5             # Pause around actions.
  fail_fast: true        # Stop workflow on first failed step.
  images_dir: ./images   # Directory containing reference images.

Actions

Action Description Common fields
click Click an image center or absolute x/y. image, x, y, button, clicks, interval
double_click Double-click an image center. image
right_click Right-click an image center. image
hover Move the mouse over an image. image
drag Drag from one image to another or to coordinates. image_from, image_to, to_x, to_y, duration
scroll Scroll at current position or at an image. clicks, image
type Click an optional image, then paste text using the clipboard. text, image
type_slow Click an optional image, then type character by character. text, image, interval
press Press one key. key
hotkey Press a key combination. keys
wait Wait for an image to be present or absent. image, state
assert Fail if an image is not in the expected state. image, state
delay Wait a fixed number of seconds. seconds
screenshot Save a screenshot. filename
focus_rdp Bring an RDP window to the foreground when pywin32 is installed. title_contains

Each step may also include description, id, timeout, confidence, unless, and fallback.

Placeholders

Placeholder Source Example
{param:name} CLI --param, GUI parameter rows, or workflow params. {param:username}
{queue:column} Current queue CSV row. {queue:id}
{queue.column} Current queue CSV row, dot notation. {queue.name}
{env:NAME} Environment variable. {env:USERNAME}
{timestamp} Current timestamp as YYYYMMDD_HHMMSS. output_{timestamp}.png

Fallbacks

Fallback steps run only if the primary step fails.

- description: "Open menu"
  action: click
  image: menu_button
  timeout: 5
  fallback:
    - action: hotkey
      keys: [alt, f]
    - action: press
      key: enter

Project Layout

ClickFlow/
|-- cli/
|   |-- automate.py        # Run one YAML workflow.
|   |-- capture_images.py  # Capture/list/retake/delete reference images.
|   |-- queue_runner.py    # Process CSV queues from the command line.
|   `-- recorder.py        # Record mouse/keyboard events to YAML.
|-- core/
|   |-- actions.py         # Mouse, keyboard, wait, assert, screenshot actions.
|   |-- engine.py          # YAML workflow parser and executor.
|   |-- exceptions.py      # ClickFlow exceptions.
|   `-- matcher.py         # OpenCV image matching and screenshots.
|-- gui/
|   |-- app.py             # CustomTkinter application shell.
|   |-- tabs/              # Run, Queue, Capture, Images, Workflows, Record.
|   |-- utils/             # Config and action schema helpers.
|   |-- widgets/           # Reusable dialogs and widgets.
|   `-- workers/           # Background workflow and queue workers.
|-- images/                # Reference images. Usually workflow/app specific.
|-- queues/                # Local queue CSVs and progress JSON files.
|-- workflows/             # YAML workflows and templates.
|-- requirements.txt
`-- README.md

Troubleshooting

Image Not Found

  • Lower confidence, for example --confidence 0.7, or set confidence: 0.7 on a step.
  • Retake the reference image with a tighter crop and less surrounding noise.
  • Keep the target window at the same size and scale used when capturing images.
  • Use --debug to save debug screenshots in debug_screenshots/.

Wrong Element Clicked

  • Avoid generic images such as blank button corners.
  • Capture nearby label text or a larger unique region.
  • Use workflow-specific image directories with settings.images_dir.

Text Does Not Type Correctly

  • type uses clipboard paste, which is usually best over RDP.
  • If the target blocks paste, use type_slow.
  • Click or tab into the field before typing.

Queue Tab Says No CSV Configured

  • Add a queue: block to the active workflow.
  • Put the CSV under queues/ when using the GUI Queue tab.
  • Make sure queue.csv matches the file name, with or without .csv.

RDP Focus Problems

  • Keep the RDP window visible and foregrounded.
  • Do not resize the RDP window after capturing reference images.
  • Install pywin32 if you want to use the focus_rdp action on Windows.

Emergency Stop

PyAutoGUI fail-safe is enabled. Move the mouse to the top-left corner of the screen to abort automation.

About

Automate repetitive desktop tasks with visual recognition, reusable workflows, and queue processing.

Resources

License

Stars

0 stars

Watchers

0 watching

Forks

Packages

 
 
 

Contributors