ClickFlow is a Python desktop automation tool that finds UI elements by matching reference images on screen, then runs actions such as click, type, drag, scroll, wait, and keyboard shortcuts. It is useful for applications and RDP sessions where DOM or API automation is not available because it works from what is visible on the screen.
The project currently includes a CustomTkinter desktop GUI, CLI tools, YAML workflow execution, reference image capture, CSV queue processing, and action recording.
- Screenshots
- Quick Start
- How It Works
- Desktop GUI
- Workflow Files
- Building Workflow Actions
- Queue Processing
- CLI Tools
- Portable Windows Build
- YAML Reference
- Project Layout
- Troubleshooting
pip install -r requirements.txt
python gui/app.pyRecommended first workflow:
- Open the GUI with
python gui/app.py. - Go to Workflows and create or select a workflow.
- Go to Capture and capture the UI elements used by the workflow.
- Go to Images to review, rename, delete, or retake reference images.
- Go to Run, adjust parameters and overrides, then run the workflow.
ClickFlow workflows are YAML files stored under workflows/. Each step usually finds a named image in an images directory, then performs an action at the matched location.
For example, this step looks for images/search_field.png, clicks it, and pastes text:
- description: "Search for user"
action: type
image: search_field
text: "{param:username}"Reference images are small screenshots of buttons, fields, labels, menu items, or other stable UI regions. Image names are referenced without the file extension.
Launch the GUI:
python gui/app.pyThe GUI saves window state, selected tab, selected workflow, and theme in ~/.clickflow/config.json.
| Tab | Purpose |
|---|---|
| Run | Select a workflow, enter parameters, set confidence/timeout/delay/images directory, dry-run, save workflow settings, and execute. |
| Queue | Process the active workflow against a CSV queue, with resume/reset, limit, progress, failed-item retry, and progress JSON tracking. |
| Capture | Capture a screen region into the active images directory after an optional countdown. |
| Images | Browse reference images, filter by name, preview metadata, rename, delete, or retake. |
| Workflows | Browse, create, preview, edit, and run YAML workflows. Includes Blank, Template, and Generic Queue starters. |
| Record | Record clicks, typing, scrolling, and key presses into a generated YAML workflow. |
| Shortcut | Tab |
|---|---|
Ctrl+R |
Run |
Ctrl+Shift+Q |
Queue |
Ctrl+Shift+C |
Capture |
Ctrl+Shift+I |
Images |
Ctrl+Shift+W |
Workflows |
Ctrl+Shift+R |
Record |
Workflow files live in workflows/ and are discovered recursively by the GUI.
Minimal example:
name: "Example Workflow"
description: "Find a field, type text, and confirm."
settings:
confidence: 0.8
timeout: 10
delay: 0.5
fail_fast: true
images_dir: ./images
params:
username: ""
steps:
- description: "Type username"
action: type
image: username_field
text: "{param:username}"
- description: "Confirm"
action: click
image: confirm_button
fallback:
- action: press
key: enterSee workflows/template.yaml for a broader action template and workflows/generic_queue.yaml for a queue-oriented starter.
A workflow action is one item under steps:. Most actions follow this pattern:
- description: "Human-readable step name"
action: click
image: button_ok
timeout: 10
confidence: 0.85The action field decides which operation ClickFlow runs. The rest of the fields are the inputs for that action. For image-based actions, the image field is the reference image name without .png. For example, image: button_ok points to button_ok.png inside the active images_dir.
- Capture the target UI element in the Capture tab or with
cli/capture_images.py. - Give the image a stable, descriptive name such as
login_button,search_field, orsave_menu_item. - Confirm the image is in the workflow's
settings.images_dirfolder. - Add a step in the Workflows tab or edit the YAML directly.
- Pick the action type and fill only the fields that action needs.
- Add
timeoutorconfidenceon the step only when it needs different matching behavior than the global settings. - Add
fallbackactions for important steps where a keyboard shortcut can recover from image matching failure. - Test with one or two steps first, then expand the workflow.
The Workflows tab is the main place to create and maintain workflow actions without editing YAML by hand.
To create a workflow and add actions in the GUI:
- Open Workflows.
- Click + New.
- Fill in Name, Description, Template, Confidence, Timeout, Delay, Fail fast, and Images dir.
- Choose Blank if you want to build every action yourself, Template for a simple starter workflow, or Generic Queue for a starter that uses queue placeholders.
- Click Create.
- Select the workflow from the list on the left.
- Click + Add Step.
- In the step editor, choose the Action type.
- Fill in Description and the action-specific fields shown under Action Fields.
- Use Browse next to image fields to pick from the workflow's image directory.
- Add per-step Timeout or Confidence only when this action needs different matching settings.
- Add Fallbacks when there is a keyboard or alternate action that can recover from failure.
- Click Save in the step editor.
- Click Save Steps in the Workflows tab to write the changes to the YAML file.
- Click Run this or switch to the Run tab to test the workflow.
Existing actions appear as step cards. Use Up and Down to reorder them, Edit to change a step, and X to delete a step. Show YAML lets you inspect the generated YAML, but the step editor is usually safer for normal editing.
The action editor only shows fields relevant to the selected action. For example, click shows an image picker and click options, type shows text plus an optional image picker, press shows a key field, and wait or assert show image plus present/absent state.
| Goal | GUI action type | Fields to fill |
|---|---|---|
| Click a button | click |
Image name = save_button |
| Type into a field | type |
Image name = search_field, Text to type = {param:search} |
| Type queue data | type |
Image name = search_field, Text to type = {queue:id} |
| Press Enter | press |
Key = enter |
| Use Ctrl+S | hotkey |
Keys = ctrl, s |
| Wait for spinner to disappear | wait |
Image name = loading_spinner, State = absent |
| Verify success message | assert |
Image name = success_message, State = present |
| Pause between actions | delay |
Seconds = 1 |
When entering multiple hotkey keys in the GUI, separate them with commas, for example ctrl, s or ctrl, shift, esc.
| Field | Required | Purpose |
|---|---|---|
description |
No | Label shown in logs and progress output. |
action |
Yes | Action name such as click, type, press, wait, or assert. |
image |
Action-dependent | Reference image name without .png. Used by actions that need to find something on screen. |
text |
For type and type_slow |
Text to enter. Can include {param:...} or {queue:...} placeholders. |
key |
For press |
Key name such as enter, tab, escape, f5, up, or down. |
keys |
For hotkey |
List of keys, for example [ctrl, s]. |
timeout |
No | Per-step seconds to wait for an image. Overrides settings.timeout. |
confidence |
No | Per-step image match threshold. Overrides settings.confidence. |
fallback |
No | One or more backup steps to try if the primary step fails. |
id |
No | Stable result key for step results. Useful when debugging. |
unless |
No | Basic skip condition. Currently supports checks such as param:name. |
Click a button:
- description: "Click Save"
action: click
image: save_buttonType into a field. type first clicks the optional image, then pastes the text through the clipboard:
- description: "Enter user ID"
action: type
image: user_id_field
text: "{param:user_id}"Type slowly when clipboard paste is blocked:
- description: "Enter code slowly"
action: type_slow
image: code_field
text: "{param:code}"
interval: 0.05Press a single key without image matching:
- description: "Confirm dialog"
action: press
key: enterUse a keyboard shortcut:
- description: "Save with Ctrl+S"
action: hotkey
keys: [ctrl, s]Wait until loading disappears:
- description: "Wait for loading spinner"
action: wait
image: loading_spinner
state: absent
timeout: 30Assert that the expected result is visible:
- description: "Verify success"
action: assert
image: success_message
state: present
timeout: 10Drag from one image to another:
- description: "Move item to target"
action: drag
image_from: draggable_item
image_to: drop_target
duration: 0.5Add a fallback when image matching might fail:
- description: "Open File menu"
action: click
image: file_menu
timeout: 5
fallback:
- action: hotkey
keys: [alt, f]
- action: press
key: enterCapture the smallest stable region that uniquely identifies the target. Good examples are a button with its label, a field label plus a bit of the field border, or a unique icon with nearby text. Avoid blank corners, generic borders, or large screenshots that include changing content. If two screens use similar controls, put each app's images in its own folder and set settings.images_dir for that workflow.
ClickFlow can run the same workflow once per row in a CSV file. This is useful when the same screen process must be repeated for many users, IDs, files, tickets, products, or other records.
The queue runner reads one CSV row, injects the row values into the workflow as placeholders, runs the workflow, records progress, then moves to the next row.
A queue CSV must have a header row. Each header becomes a placeholder that can be used in workflow steps.
id,name,department,action,status
USR001,John Smith,Engineering,activate,pending
USR002,Jane Doe,Marketing,deactivate,pending
USR003,Bob Wilson,Finance,update_role,pendingIn the workflow, those columns are available as:
| CSV column | Placeholder forms | Example value |
|---|---|---|
id |
{queue:id} or {queue.id} |
USR001 |
name |
{queue:name} or {queue.name} |
John Smith |
department |
{queue:department} or {queue.department} |
Engineering |
action |
{queue:action} or {queue.action} |
activate |
The first CSV column is used as the unique item key unless key_column is set. The key is what appears in the progress table and progress JSON. Pick a stable unique column such as id, ticket, filename, or account_number.
The status column is optional. If present, rows marked completed, skipped, or done are ignored by the queue loader unless you reset progress. Rows with pending or blank status are processed.
Queue values are normally used for fields like text, description, or filenames. The image field should usually stay fixed because it points to a reference screenshot file.
Good:
- action: type
image: search_field
text: "{queue:id}"Avoid this unless you intentionally captured separate images for every possible CSV value:
- action: click
image: "{queue:button_name}"The GUI Queue tab reads queue settings from the active workflow's queue: block. CSV paths are resolved from the queues/ directory, and .csv is appended automatically if omitted.
To set up a GUI queue:
- Create a CSV file under
queues/, for examplequeues/users.csv. - Add a
queue:block to the workflow or create the workflow from the Workflows tab with a Queue CSV value. - Use
{queue:column_name}placeholders in the workflow steps. - Select the workflow in the GUI.
- Open the Queue tab and click Refresh to preview rows.
- Enter a small Limit such as
1for the first test run. - Click PROCESS QUEUE.
- Use Resume after an interruption, Reset to start over, and Retry for failed rows.
The Workflows tab can create the workflow and the queue CSV starter for you.
To create a queued workflow from the GUI:
- Open Workflows.
- Click + New.
- Enter the workflow name and choose Generic Queue if you want queue-ready starter actions.
- Set Images dir to the folder where this workflow's reference images will live.
- In Queue CSV, either type a CSV name such as
users.csvor click Create CSV. - In Create Queue CSV, enter a filename and column names such as
id,name,department,action. - Choose the Key column, usually
idor another unique value. - Click Create in the CSV dialog. ClickFlow writes the CSV under
queues/and fills the workflow's Queue CSV and Key column fields. - Set CSV delimiter, On error, Max retries, and Rate limit (s).
- Click Create in the workflow form.
- Edit the generated steps so they use your actual reference images and queue placeholders.
- Open Queue, click Refresh, check the row preview, set Limit to
1, then click PROCESS QUEUE.
The CSV dialog creates the header row only. Add queue rows in the CSV file afterward using Excel, a text editor, or another export process. The GUI Queue tab will preview one pending item per row after you click Refresh.
Queue options in the GUI map to the workflow YAML like this:
| GUI field | YAML field | Meaning |
|---|---|---|
| Queue CSV | queue.csv |
CSV file under queues/. |
| Key column | queue.key_column |
Unique row identifier used in progress tracking. |
| CSV delimiter | queue.delimiter |
Separator used by the CSV file. |
| On error | queue.on_error |
Stop at first failed row or continue. |
| Max retries | queue.max_retries |
Attempts per row before marking failed. |
| Rate limit (s) | queue.rate_limit |
Delay between processed rows. |
Queue placeholders can be typed directly into action fields in the GUI step editor. If your CSV has a column named id, type {queue:id} into a text field. If it has department, type {queue:department}. These values are replaced separately for each row while the queue runs.
Typical GUI step editor values for a queued action:
| Action type | Field | Value |
|---|---|---|
type |
Image name | search_field |
type |
Text to type | {queue:id} |
click |
Image name | action_button |
assert |
Image name | success_message |
assert |
State | present |
Keep reference image fields fixed unless you intentionally have many image files named from CSV values. Queue data usually belongs in text fields, descriptions, and output filenames, not in image names.
Minimal GUI queue workflow:
name: "Queued Workflow"
queue:
csv: workflow3.csv
key_column: id
delimiter: ","
on_error: stop
max_retries: 1
rate_limit: 0
settings:
confidence: 0.8
timeout: 10
delay: 0.5
images_dir: ./images/app3
steps:
- description: "Search for {queue:id}"
action: type
image: search_field
text: "{queue:id}"
- description: "Confirm"
action: click
image: confirm_buttonExample CSV in queues/workflow3.csv:
id,name,action
USR001,John Smith,activate
USR002,Jane Doe,deactivateProgress is stored next to the CSV as <queue-name>.progress.json. Use Resume to skip completed rows, Reset to start over, and Retry on failed rows after a run has stopped.
| Field | Required | Used by | Purpose |
|---|---|---|---|
csv |
Yes for GUI Queue tab | GUI | CSV file under queues/. .csv may be omitted. |
key_column |
No | GUI and CLI equivalent flag | CSV column used as each row's unique key. Defaults to first column. |
delimiter |
No | GUI and CLI equivalent flag | CSV delimiter. Use ,, ;, or tab depending on the file. |
on_error |
No | GUI and CLI equivalent flag | stop stops at first failed item. continue moves to the next item. |
max_retries |
No | GUI and CLI equivalent flag | Number of attempts for each row. Defaults to 1. |
rate_limit |
No | GUI | Seconds to wait between queue items. |
CSV file at queues/users.csv:
id,name,department,action,status
USR001,John Smith,Engineering,activate,pending
USR002,Jane Doe,Marketing,deactivate,pendingWorkflow file at workflows/process_users.yaml:
name: "Process Users"
description: "Search each user from queues/users.csv and apply an action."
queue:
csv: users.csv
key_column: id
delimiter: ","
on_error: stop
max_retries: 2
rate_limit: 1
settings:
confidence: 0.8
timeout: 10
delay: 0.5
images_dir: ./images/users_app
steps:
- description: "Search for {queue:id} - {queue:name}"
action: type
image: search_field
text: "{queue:id}"
timeout: 5
fallback:
- action: hotkey
keys: [ctrl, f]
- action: type_slow
text: "{queue:id}"
- description: "Open selected user"
action: double_click
image: search_result_row
timeout: 10
- description: "Apply requested action: {queue:action}"
action: click
image: action_button
fallback:
- action: press
key: enter
- description: "Wait for save to finish"
action: wait
image: saving_spinner
state: absent
timeout: 30
- description: "Verify success for {queue:id}"
action: assert
image: success_message
state: present
timeout: 10
- description: "Pause before next queue row"
action: delay
seconds: 1For the first run, use a GUI limit of 1 or run the CLI with --limit 1 --dry-run to confirm the queue resolves correctly.
The CLI runner accepts any CSV path:
python cli/queue_runner.py -w workflows/generic_queue.yaml -q workflows/examples/items.csv --key-column id
python cli/queue_runner.py -w workflows/generic_queue.yaml -q workflows/examples/items.csv --key-column id --resume
python cli/queue_runner.py -w workflows/generic_queue.yaml -q workflows/examples/items.csv --limit 1 --dry-runUseful queue flags:
| Flag | Purpose |
|---|---|
--resume |
Skip completed/skipped rows from the progress file. |
--reset |
Ignore existing progress and reprocess rows. |
--limit N |
Process only the first N pending rows. |
--key-column id |
Use a specific CSV column as the unique item key. |
| `--on-error stop | continue` |
--max-retries N |
Retry each item up to N attempts. |
--delay N |
Wait N seconds between items. |
--delimiter ; |
Use a custom CSV delimiter. |
The GUI is the primary interface, but the CLI tools remain available.
python cli/automate.py workflows/template.yaml
python cli/automate.py workflows/template.yaml --param my_var=hello
python cli/automate.py workflows/template.yaml --dry-run
python cli/automate.py workflows/template.yaml --debug
python cli/automate.py workflows/template.yaml --confidence 0.7 --timeout 15
python cli/automate.py workflows/template.yaml --no-fail-fast
python cli/automate.py --list-imagespython cli/capture_images.py
python cli/capture_images.py --bulk-capture button_ok search_field confirm_button
python cli/capture_images.py --list
python cli/capture_images.py --retake button_ok
python cli/capture_images.py --delete button_old
python cli/capture_images.py --images-dir images/app1python cli/recorder.py workflows/recorded.yaml
python cli/recorder.py workflows/recorded.yaml --images-dir images/app1During CLI recording, press F6 to stop and save or Esc to cancel.
ClickFlow can be packaged as a portable Windows zip containing ClickFlow.exe. End users can unzip it and run the app without installing Python.
The project uses PyInstaller in onedir mode because CustomTkinter includes theme, font, and JSON assets that must be shipped with the executable.
From the repository root on Windows:
powershell -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Clean -Version 0.1.0The build script will:
- Create
.venv-build/if needed. - Install
requirements.txtandbuild-requirements.txt. - Run PyInstaller with
ClickFlow.spec. - Create
dist/ClickFlow/withClickFlow.exeand bundled runtime files. - Copy starter
workflows/,images/, andREADME_FIRST.txtinto the release folder. - Create
dist/ClickFlow-0.1.0-windows-x64.zip.
Use -Version dev or omit -Version to create dist/ClickFlow-windows-x64.zip.
ClickFlow-windows-x64.zip
`-- ClickFlow/
|-- ClickFlow.exe
|-- README_FIRST.txt
|-- _internal/
|-- workflows/
|-- images/
|-- queues/
`-- debug_screenshots/
Users should extract the zip first, then double-click ClickFlow.exe. They should keep workflows/, images/, and queues/ next to the exe because ClickFlow reads and writes those folders at runtime.
The GitHub Actions workflow at .github/workflows/release-latest-main.yml builds a new Windows zip on every push to main and updates one rolling prerelease named latest-main.
This keeps the GitHub Releases page clean:
- The
latest-maintag is moved to the newestmaincommit. - The release title and notes are updated with the latest commit and workflow run.
- The
ClickFlow-latest-main-windows-x64.zipasset is replaced using--clobber. - Older main builds are not kept as separate releases.
Use latest-main for internal testing builds. For public/stable releases, create a separate versioned release such as v1.0.0.
Before publishing a zip, test it on a Windows machine that does not have Python installed:
- Extract the zip.
- Double-click
ClickFlow.exe. - Open every tab.
- Create a workflow in the GUI.
- Capture or browse an image.
- Run a dry-run workflow.
- Create a queue CSV and process with
Limit = 1. - Confirm
queues/*.progress.jsonis written. - Start and stop the recorder.
Unsigned PyInstaller executables can trigger Windows SmartScreen warnings. Code signing is recommended for public distribution.
settings:
confidence: 0.8 # Match threshold from 0.0 to 1.0.
timeout: 10 # Seconds to wait for each image.
delay: 0.5 # Pause around actions.
fail_fast: true # Stop workflow on first failed step.
images_dir: ./images # Directory containing reference images.| Action | Description | Common fields |
|---|---|---|
click |
Click an image center or absolute x/y. |
image, x, y, button, clicks, interval |
double_click |
Double-click an image center. | image |
right_click |
Right-click an image center. | image |
hover |
Move the mouse over an image. | image |
drag |
Drag from one image to another or to coordinates. | image_from, image_to, to_x, to_y, duration |
scroll |
Scroll at current position or at an image. | clicks, image |
type |
Click an optional image, then paste text using the clipboard. | text, image |
type_slow |
Click an optional image, then type character by character. | text, image, interval |
press |
Press one key. | key |
hotkey |
Press a key combination. | keys |
wait |
Wait for an image to be present or absent. | image, state |
assert |
Fail if an image is not in the expected state. | image, state |
delay |
Wait a fixed number of seconds. | seconds |
screenshot |
Save a screenshot. | filename |
focus_rdp |
Bring an RDP window to the foreground when pywin32 is installed. |
title_contains |
Each step may also include description, id, timeout, confidence, unless, and fallback.
| Placeholder | Source | Example |
|---|---|---|
{param:name} |
CLI --param, GUI parameter rows, or workflow params. |
{param:username} |
{queue:column} |
Current queue CSV row. | {queue:id} |
{queue.column} |
Current queue CSV row, dot notation. | {queue.name} |
{env:NAME} |
Environment variable. | {env:USERNAME} |
{timestamp} |
Current timestamp as YYYYMMDD_HHMMSS. |
output_{timestamp}.png |
Fallback steps run only if the primary step fails.
- description: "Open menu"
action: click
image: menu_button
timeout: 5
fallback:
- action: hotkey
keys: [alt, f]
- action: press
key: enterClickFlow/
|-- cli/
| |-- automate.py # Run one YAML workflow.
| |-- capture_images.py # Capture/list/retake/delete reference images.
| |-- queue_runner.py # Process CSV queues from the command line.
| `-- recorder.py # Record mouse/keyboard events to YAML.
|-- core/
| |-- actions.py # Mouse, keyboard, wait, assert, screenshot actions.
| |-- engine.py # YAML workflow parser and executor.
| |-- exceptions.py # ClickFlow exceptions.
| `-- matcher.py # OpenCV image matching and screenshots.
|-- gui/
| |-- app.py # CustomTkinter application shell.
| |-- tabs/ # Run, Queue, Capture, Images, Workflows, Record.
| |-- utils/ # Config and action schema helpers.
| |-- widgets/ # Reusable dialogs and widgets.
| `-- workers/ # Background workflow and queue workers.
|-- images/ # Reference images. Usually workflow/app specific.
|-- queues/ # Local queue CSVs and progress JSON files.
|-- workflows/ # YAML workflows and templates.
|-- requirements.txt
`-- README.md
- Lower confidence, for example
--confidence 0.7, or setconfidence: 0.7on a step. - Retake the reference image with a tighter crop and less surrounding noise.
- Keep the target window at the same size and scale used when capturing images.
- Use
--debugto save debug screenshots indebug_screenshots/.
- Avoid generic images such as blank button corners.
- Capture nearby label text or a larger unique region.
- Use workflow-specific image directories with
settings.images_dir.
typeuses clipboard paste, which is usually best over RDP.- If the target blocks paste, use
type_slow. - Click or tab into the field before typing.
- Add a
queue:block to the active workflow. - Put the CSV under
queues/when using the GUI Queue tab. - Make sure
queue.csvmatches the file name, with or without.csv.
- Keep the RDP window visible and foregrounded.
- Do not resize the RDP window after capturing reference images.
- Install
pywin32if you want to use thefocus_rdpaction on Windows.
PyAutoGUI fail-safe is enabled. Move the mouse to the top-left corner of the screen to abort automation.



