RL environments: Windows desktop packs (UI Automation and pixels-only RDP)

packages/flow/environments/openenv/_windows/README.md

@@ -0,0 +1,108 @@
+# Windows desktop environments
+
+Four OpenEnv environments on two Windows line-of-business apps, each backed by
+a SQLite database. The reward is verify-run's verdict on that database, read
+outside the agent's session, so an agent earns reward only when the record
+shows what it reports.
+
+| Environment | App | Surface | What the agent observes |
+|---|---|---|---|
+| `win-patient-chart` | Patient Chart (WinForms) | UI Automation | Screen and the window's UI Automation elements |
+| `win-patient-chart-rdp` | Patient Chart (WinForms) | Remote Desktop | Screen pixels only |
+| `win-claims-desk` | Claims Desk (WPF) | UI Automation | Screen and the window's UI Automation elements |
+| `win-claims-desk-rdp` | Claims Desk (WPF) | Remote Desktop | Screen pixels only |
+
+The two variants of an app share one task set, so a policy's score on pixels
+alone compares directly with its score with an accessibility tree.
+
+## Tasks
+
+Each app has 24 public tasks, generated from one seed by
+[`make_tasks.py`](make_tasks.py):
+
+- 12 feasible tasks: write a chart note on a patient, or record a decision
+  (status, allowed amount, note) on a claim. In three, the first save is
+  lost while the screen says "saved"; the reference solution re-opens the
+  record, sees the count unchanged and saves again. Three show a misleading
+  banner, such as "Note saved for Nell Sorensen (MRN-104812)" or "Batch
+  posting complete: all decisions saved". Three use a reordered list.
+- 12 tasks with a planted write fault. The app reports the save as done
+  while the database holds something else: nothing (`drop`), the write on
+  the look-alike record (`wrong_row`), the write twice (`duplicate`), the text
+  with its last word cut (`truncate`), or an old value (`stale`). The correct
+  outcome is to report failure, after trying the save: a failure report pays
+  only when the app was asked to save the task's values.
+
+Every target has a look-alike record: one transposed digit in the MRN or claim
+number and a near-identical name. Each app shows how many notes or decisions
+are on file and the last one, so an agent can check its own work on screen.
+
+Task ids are opaque and tags never name the fault. `tasks/MANIFEST.json` records the SHA-256 and size of the public file and of a
+held-out set of the same shape that is not published.
+
+## Windows is yours
+
+Nothing here contains Windows. Like Windows Agent Arena, the environments run
+on a Windows machine you provide: a VM built from your own Windows image or
+evaluation ISO, or an Azure or AWS Windows template.
+
+1. Run [`provision/setup_guest.ps1`](provision/setup_guest.ps1) on it as an
+   administrator. It turns on WinRM and Remote Desktop, sets the session host
+   to prefer H.264/AVC444, installs CPython and the in-guest agent's packages,
+   and creates `C:\oa\pack`.
+2. Sign the desktop account in to its desktop.
+3. On the server, set `OPENADAPT_WINENV_HOST`, `OPENADAPT_WINENV_USER`,
+   `OPENADAPT_WINENV_PASSWORD` and `OPENADAPT_WINENV_PYTHON` (the path the
+   script printed), then serve an environment:
+
+```bash
+pip install 'openadapt-flow[openenv]' pywinrm aardwolf
+python -m openadapt_flow.rl_env.desktop environments/openenv/win-patient-chart --port 8000
+```
+
+The server copies the apps in, then resets each episode in one WinRM call:
+it stops the app, restores the database from its pristine copy, sets the
+episode's fault and banner, and starts the app. It compares the restored
+database's digest with a digest it computes from the same seed and refuses
+the episode if they differ.
+
+## Surfaces
+
+- **UI Automation**: input goes through the in-guest agent (`win_agent`) over
+  TLS with a pinned certificate. Observations carry the screenshot and the
+  window's elements (role, name, automation id, bounds), without field values.
+- **Remote Desktop**: input and frames go over an RDP session. The client is
+  FreeRDP 3 under Xvfb, asking for H.264/AVC444 (`rdp.avc444: true`), as
+  against a production session host. It needs Linux; see
+  [`rdp-client/Containerfile`](rdp-client/Containerfile). On any other host,
+  `OPENADAPT_WINENV_RDP_TRANSPORT=aardwolf` uses a pure-Python client that
+  negotiates RDP's bitmap codecs instead.
+
+Actions are `click`, `double_click`, `right_click` and `scroll` at session
+coordinates, `type`, `press` and `done` with a claim. The Windows key and the
+Ctrl+Esc and Ctrl+Shift+Esc chords are refused. The agent has no network
+action.
+
+## Results
+
+On a Windows 11 Pro x64 machine (2026-11-04), all four environments passed every check:
+- every reference solution earned 1, and every feasible task earned 0 with a write fault planted;
+- random actions earned 0, and an agent trusting the "saved" message earned exactly the feasible half;
+- no state carried between episodes.
+
+Median reset was 6.6–7.3 s on the UI Automation surface and 3.2–3.3 s over Remote Desktop with AVC444. Details and the full episode logs are in [`results/windows11-x64-2026-11-04/`](results/windows11-x64-2026-11-04/README.md).
+
+## Validate
+
+```bash
+OPENADAPT_ENV_REFERENCE=1 python -m openadapt_flow.rl_env.validate_desktop \
+    environments/openenv/win-patient-chart --out results.json
+```
+
+The run checks that every task's reference solution earns 1, that an agent
+trusting the "saved" message earns 1 only on feasible tasks, that each
+feasible task's solution earns 0 with a write fault planted, that random
+actions stay at least 0.1 below the maximum, and that no state carries from
+one episode to the next. It also records each reset's duration.
+`OPENADAPT_ENV_REFERENCE=1` lets the reference solutions name elements by
+automation id on the pixels-only surface; never set it for an agent.

packages/flow/environments/openenv/_windows/apps/claims_desk.ps1

@@ -0,0 +1,203 @@
+# Claims Desk (WPF): find a claim, adjudicate it (status, allowed amount, note), save.
+#
+# A self-built line-of-business app for the Windows environment pack, the
+# WPF counterpart of patient_chart.ps1 (WinForms). Persistence goes through
+# pack_db.py (SQLite); the oracle reads the database outside the session. The
+# app never reads the planted fault and always reports the save as done.
+#
+# UI knobs come from C:\oa\pack\data\claims_desk_ui.json, written at reset:
+#   banner : text in a yellow strip at the top (a misleading notice).
+#   window : "maximized" (default: full screen over the taskbar) or
+#            "windowed"; size "WxH" forces windowed.
+#
+# Usage:  powershell -STA -ExecutionPolicy Bypass -File claims_desk.ps1
+
+param(
+    [string]$Python = "",
+    [string]$DbCli  = "C:\oa\pack\pack_db.py"
+)
+
+Add-Type -AssemblyName PresentationFramework
+Add-Type -AssemblyName PresentationCore
+Add-Type -AssemblyName WindowsBase
+
+if (-not $Python) { $Python = (Get-Content "C:\oa\pack\python.txt" -Raw).Trim() }
+
+$cfg = @{ banner = ""; window = "maximized"; size = "" }
+$cfgPath = "C:\oa\pack\data\claims_desk_ui.json"
+if (Test-Path $cfgPath) {
+    try {
+        $j = Get-Content $cfgPath -Raw | ConvertFrom-Json
+        foreach ($k in @("banner", "window", "size")) {
+            if ($null -ne $j.$k) { $cfg[$k] = $j.$k }
+        }
+    } catch {}
+}
+
+function Invoke-Db {
+    param([string[]]$DbArgs)
+    $psi = New-Object System.Diagnostics.ProcessStartInfo
+    $psi.FileName = $Python
+    $psi.Arguments = (@("-I", "`"$DbCli`"") + $DbArgs) -join ' '
+    $psi.RedirectStandardOutput = $true
+    $psi.UseShellExecute = $false
+    $psi.CreateNoWindow = $true
+    $p = [System.Diagnostics.Process]::Start($psi)
+    $out = $p.StandardOutput.ReadToEnd()
+    $p.WaitForExit()
+    return $out
+}
+
+[xml]$xaml = @"
+<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
+        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
+        Title="Claims Desk" Name="claimsDeskWindow" Width="980" Height="660"
+        Left="40" Top="40" Topmost="True" FontFamily="Segoe UI" FontSize="13" Background="White">
+  <Grid Margin="20,8,20,12">
+    <Grid.RowDefinitions>
+      <RowDefinition Height="30"/>
+      <RowDefinition Height="40"/>
+      <RowDefinition Height="250"/>
+      <RowDefinition Height="52"/>
+      <RowDefinition Height="36"/>
+      <RowDefinition Height="40"/>
+      <RowDefinition Height="100"/>
+      <RowDefinition Height="44"/>
+    </Grid.RowDefinitions>
+    <TextBlock Grid.Row="0" Name="noticeBanner" Background="#FFF1A8" Padding="6,4" Visibility="Collapsed"/>
+    <StackPanel Grid.Row="1" Orientation="Horizontal" Margin="0,4,0,4">
+      <TextBox Name="searchBox" Width="520" Height="28" AutomationProperties.Name="Search claims"/>
+      <Button Name="searchButton" Content="Search" Width="100" Height="30" Margin="10,0,0,0"/>
+    </StackPanel>
+    <DataGrid Grid.Row="2" Name="claimsGrid" AutoGenerateColumns="False" IsReadOnly="True"
+              SelectionMode="Single" SelectionUnit="FullRow" HeadersVisibility="Column"
+              RowHeight="34" ColumnHeaderHeight="32" CanUserAddRows="False"
+              AutomationProperties.Name="Claims">
+      <DataGrid.Columns>
+        <DataGridTextColumn Header="Claim" Binding="{Binding claim_no}" Width="170"/>
+        <DataGridTextColumn Header="Member" Binding="{Binding member}" Width="190"/>
+        <DataGridTextColumn Header="Member ID" Binding="{Binding member_id}" Width="130"/>
+        <DataGridTextColumn Header="Service date" Binding="{Binding service_date}" Width="130"/>
+        <DataGridTextColumn Header="Billed" Binding="{Binding billed}" Width="100"/>
+        <DataGridTextColumn Header="Status" Binding="{Binding status}" Width="*"/>
+      </DataGrid.Columns>
+    </DataGrid>
+    <StackPanel Grid.Row="3" VerticalAlignment="Center">
+      <TextBlock Name="claimHeader" FontWeight="SemiBold"/>
+      <TextBlock Name="historyText" Foreground="DimGray"/>
+    </StackPanel>
+    <StackPanel Grid.Row="4" Orientation="Horizontal" VerticalAlignment="Center">
+      <TextBlock Text="Decision:" Width="90" VerticalAlignment="Center"/>
+      <RadioButton Name="statusApproved" Content="Approved" GroupName="decision" Margin="0,0,24,0" VerticalAlignment="Center"/>
+      <RadioButton Name="statusDenied" Content="Denied" GroupName="decision" Margin="0,0,24,0" VerticalAlignment="Center"/>
+      <RadioButton Name="statusPended" Content="Pended" GroupName="decision" VerticalAlignment="Center"/>
+    </StackPanel>
+    <StackPanel Grid.Row="5" Orientation="Horizontal" VerticalAlignment="Center">
+      <TextBlock Text="Allowed:" Width="90" VerticalAlignment="Center"/>
+      <TextBox Name="amountBox" Width="160" Height="28" AutomationProperties.Name="Allowed amount"/>
+    </StackPanel>
+    <TextBox Grid.Row="6" Name="noteBox" AcceptsReturn="True" TextWrapping="Wrap" AutomationProperties.Name="Adjudication note"/>
+    <DockPanel Grid.Row="7" LastChildFill="False" Margin="0,8,0,0">
+      <TextBlock Name="statusText" Text="Ready" DockPanel.Dock="Left" VerticalAlignment="Center"/>
+      <Button Name="saveButton" Content="Save Decision" Width="130" Height="34" DockPanel.Dock="Right"/>
+    </DockPanel>
+  </Grid>
+</Window>
+"@
+
+$window = [Windows.Markup.XamlReader]::Load((New-Object System.Xml.XmlNodeReader $xaml))
+$c = @{}
+foreach ($n in @("noticeBanner", "searchBox", "searchButton", "claimsGrid", "claimHeader",
+                 "statusApproved", "statusDenied", "statusPended", "amountBox", "noteBox",
+                 "statusText", "saveButton", "historyText")) { $c[$n] = $window.FindName($n) }
+
+if ([string]$cfg["banner"]) {
+    $c.noticeBanner.Text = [string]$cfg["banner"]
+    $c.noticeBanner.Visibility = "Visible"
+}
+# Full screen by default: no chrome and maximized, so the window covers the
+# taskbar and an agent's clicks cannot reach the Start menu or another app.
+if ([string]$cfg["window"] -ne "windowed" -and -not [string]$cfg["size"]) {
+    $window.WindowStyle = "None"
+    $window.ResizeMode = "NoResize"
+    $window.WindowState = "Maximized"
+}
+if ([string]$cfg["size"]) {
+    $wh = ([string]$cfg["size"]) -split 'x'
+    $window.Width = [int]$wh[0]; $window.Height = [int]$wh[1]
+}
+
+$script:current = $null
+
+function Clear-Detail {
+    $script:current = $null
+    $c.claimHeader.Text = ""
+    $c.historyText.Text = ""
+    foreach ($r in @("statusApproved", "statusDenied", "statusPended")) { $c[$r].IsChecked = $false }
+    $c.amountBox.Text = ""
+    $c.noteBox.Text = ""
+}
+
+function Load-Claims {
+    param([string]$Filter)
+    $json = Invoke-Db -DbArgs @("cd-list", "`"$Filter`"")
+    $rows = @()
+    if ($json.Trim().Length -gt 0) { $rows = @($json | ConvertFrom-Json) }
+    $c.claimsGrid.ItemsSource = $rows
+    $c.claimsGrid.UnselectAll()
+    Clear-Detail
+    $c.statusText.Text = "Found " + $rows.Count + " claims"
+}
+
+$c.searchButton.Add_Click({ Load-Claims -Filter $c.searchBox.Text })
+$c.searchBox.Add_KeyDown({
+    if ($_.Key -eq "Return") { Load-Claims -Filter $c.searchBox.Text; $_.Handled = $true }
+})
+
+function Open-Claim {
+    param([string]$ClaimNo)
+    $p = (Invoke-Db -DbArgs @("cd-get", "$ClaimNo")) | ConvertFrom-Json
+    if ($p) {
+        Clear-Detail
+        $script:current = $p.claim_no
+        $c.claimHeader.Text = "Claim " + $p.claim_no + "  " + $p.member + "  " + $p.member_id + "  billed " + $p.billed + "  (" + $p.status + ")"
+        if ($p.last_decision) {
+            $c.historyText.Text = "" + $p.decision_count + " decisions on file. Last: " + $p.last_decision.status + " " + $p.last_decision.allowed_amount + " - " + $p.last_decision.note
+        } else { $c.historyText.Text = "No decisions on file" }
+    }
+}
+
+$c.claimsGrid.Add_SelectionChanged({
+    $item = $c.claimsGrid.SelectedItem
+    if ($null -ne $item) { Open-Claim $item.claim_no }
+})
+
+# A click on the row that is already selected raises no SelectionChanged;
+# re-open the claim anyway, so its history and an empty form show again.
+$c.claimsGrid.Add_PreviewMouseLeftButtonUp({
+    $node = $_.OriginalSource
+    while ($null -ne $node -and -not ($node -is [System.Windows.Controls.DataGridRow])) {
+        if ($node -is [System.Windows.Media.Visual]) { $node = [System.Windows.Media.VisualTreeHelper]::GetParent($node) }
+        else { $node = $node.Parent }
+    }
+    if ($null -ne $node -and $node.IsSelected -and $null -ne $node.Item) {
+        if ($script:current -eq $node.Item.claim_no) { Open-Claim $node.Item.claim_no }
+    }
+})
+
+$c.saveButton.Add_Click({
+    if (-not $script:current) { $c.statusText.Text = "Open a claim first"; return }
+    $decision = ""
+    if ($c.statusApproved.IsChecked) { $decision = "Approved" }
+    elseif ($c.statusDenied.IsChecked) { $decision = "Denied" }
+    elseif ($c.statusPended.IsChecked) { $decision = "Pended" }
+    if (-not $decision) { $c.statusText.Text = "Choose a decision"; return }
+    $amount = $c.amountBox.Text.Trim()
+    if (-not ($amount -match '^\d+(\.\d{1,2})?$')) { $c.statusText.Text = "Allowed amount must be a number"; return }
+    $b64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($c.noteBox.Text))
+    [void](Invoke-Db -DbArgs @("cd-save", "$($script:current)", $decision, $amount, "$b64"))
+    $c.statusText.Text = "Decision saved for $($script:current)"
+})
+
+Load-Claims -Filter ""
+[void]$window.ShowDialog()