Skip to content

test(xtask): close the pwsh contract's edge cases from #658's review (ROCMAI-680) - #717

Merged
rominf merged 5 commits into
mainfrom
test/workflow-pwsh-contract-edge-cases
Nov 6, 2026
Merged

rominf merged 5 commits into
mainfrom
test/workflow-pwsh-contract-edge-cases

Conversation

@rominf

@rominf rominf commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Follows up juhovainio's three non-blocking notes on #658. The shell: pwsh contract in xtask/src/workflow_contract.rs fails a step whose native command can fail without the step noticing. Each note was a case where it misjudged a step: either it flagged a step whose status was read, or it never looked at a step that should be checked.

A $LASTEXITCODE check later in the step. Until now a check counted only in the statement directly after the native command, so putting a Write-Output or an assignment before the check flagged a command whose status was in fact read. A later check now counts if every statement in between leaves $LASTEXITCODE alone. The rule is deliberately narrow. Text inside single quotes is ignored, since it is literal. Outside it, the statement may not contain ( (a $(…) or (…) runs a command, even inside double quotes), { (a script block, or a hashtable whose values run), & (the call operator, &&) or an expanding @" here-string. It must also be a pipeline that starts with a plain value, or an assignment of one, or with one of a short list of output/file cmdlets (Write-Output, Write-Host, Out-File, Add-Content, …), and every later element must be one of those cmdlets. Assigning $LASTEXITCODE is not a read. Anything else counts as possibly resetting the status, including exit, a function call, a loop's closing } and another native command. Reaching the end of the body also counts as a read. GitHub's pwsh wrapper appends if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }, so a last native command followed only by a Write-Output is no longer flagged.

Every run: block form. run_block matched only the literal run: | header. A pwsh step written as run: |-, run: > or run: >- fell through to the one-line reading, whose value is the bare indicator, so the step escaped the contract. Every block-scalar header now matches, with any chomping or indentation indicator and an optional trailing comment. A folded > block is read the way YAML folds it: two lines at the block's indent become one line of script, so they are one command. The folding was compared against PyYAML on the same fixtures. Two more ways a step escaped turned up while doing this, and they are fixed in a separate commit: shell: or run: written as the step's first key (- shell: pwsh, - run: |). run_block is shared, so the other workflow checks that use it read these forms too. No workflow in the repository uses any of them, so those checks behave as before.

Case and keywords. PowerShell variable names are case-insensitive, so $LastExitCode now counts as a check. break, continue, begin, process, end, param and trap join PWSH_KEYWORDS. pwsh's own tokenizer confirms that every entry is a keyword.

Verification:

  • Each fix has a test that fails with the fix reverted. Mutation runs on the new rule also caught removing each forbidden character, the scoped-assignment check, chained-assignment re-checking ($a = $b = git …), the end-of-body read, and both the single- and double-quote handling. One mutant survives: allowing a plain value after a pipe. pwsh rejects that at parse time ("Expressions are only allowed as the first element of a pipeline"), so no real step can take that form.
  • Every case on both sides of the later-check rule was run through pwsh 7.4. A native command exited 3, then the in-between statements ran, then the status was printed. Every accepted case still printed 3. Every rejected case lost it, except Write-Output 'x' || git …, which is rejected conservatively.
  • Over all real workflows the contract finds the same 26 pwsh steps and the same three unchecked commands as before. Those are the three allow-listed steps, so nothing is newly flagged and no allow-list entry goes stale.

Out of scope, and still not seen by the contract: a step that inherits pwsh from defaults: run: shell: pwsh (no workflow sets one); a native command whose output is assigned ($out = cargo metadata), which is not classed as native; and the literal-only run: | matching in multiline_run_blocks and preflight_step_shape, which are separate from this contract.

This is test-only (xtask contract tests) and changes nothing a CLI user can observe, so it needs no Gherkin scenario.

Review follow-up: a statement with an unclosed quote is never treated as leaving $LASTEXITCODE alone; $LASTEXITCODE counts as read only outside single quotes and as the whole name; a folded run: >N block folds at the indent its indicator sets; and the comments and messages that said only run: | is read now name every block-scalar form.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No ticket and no xfail rows involved.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (Not applicable.)
  • Every new or changed user-facing message was read against the code path that runs after it. (The only changed message is the contract's own failure text. It now says to check $LASTEXITCODE before anything that could run another program or reset it, which is the rule the tests exercise.)

@rominf rominf changed the title test(xtask): close the pwsh contract's edge cases from #658's review test(xtask): close the pwsh contract's edge cases from #658's review (ROCMAI-680) Oct 26, 2026
@rominf
rominf force-pushed the test/workflow-pwsh-contract-edge-cases branch from e6dff6f to 93ba189 Compare November 3, 2026 09:16
PowerShell variable names are case-insensitive, so a check spelled
`$LastExitCode` reads the same status as `$LASTEXITCODE`; the pwsh
contract only matched the upper-case spelling and would flag the native
command before it.

`break`, `continue`, `begin`, `process`, `end`, `param` and `trap`
are pwsh keywords, not programs, but a statement opening with one read as
a native command. They join PWSH_KEYWORDS, all of whose entries pwsh's
tokenizer reports as keywords.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`run_block` only recognised the literal `run: |` header, so a pwsh step
written with `run: |-`, `run: >` or `run: >-` fell through to the
one-line reading, whose value is the bare indicator, and escaped the
contract whatever its script did.

Every block-scalar header now counts, with any chomping or indentation
indicator and an optional trailing comment. A folded `>` block is read
as YAML folds it before the statements are split: two lines at the
block's indent are one line of script, so they are one command, not a
native command followed by an unchecked one.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The pwsh contract matched `shell: pwsh` and `run: |` only as sibling
keys. Written as the step's first key, `- shell: pwsh` was never found
as a pwsh step, and `- run: |` read as no script at all; both took the
step out of the contract. A quoted `shell: 'pwsh'` is matched too.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…empt

The pwsh contract accepted an exit-code check only in the statement right
after a native command, so a `Write-Output` or a plain assignment before
the check flagged a command whose status was in fact read.

A check further on now counts when every statement between leaves
`$LASTEXITCODE` alone. That is defined narrowly: outside single quotes,
no `(`, `{`, `&` or expanding `@"` here-string, and a pipeline of a
plain value or assignment of one, followed only by cmdlets that write
output or files. An assignment to `$LASTEXITCODE` is not a read. Running
off the end of the body is one, since GitHub's wrapper exits with
`$LASTEXITCODE`, so a last native command followed only by a
`Write-Output` is no longer flagged either. Every case in the tests,
on both sides of the rule, was run through pwsh first.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the test/workflow-pwsh-contract-edge-cases branch from 93ba189 to db2b44c Compare November 5, 2026 14:09
@rominf
rominf marked this pull request as ready for review November 5, 2026 14:09
@rominf
rominf requested a review from a team as a code owner November 5, 2026 14:09
@rominf
rominf requested a review from johnl-amd November 5, 2026 14:09

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The rule is narrow and conservative, both sides of it are tested, and the four commits stay on one theme. Nothing here blocks merge; a few non-blocking suggestions:

  • Unterminated single quote (main one). without_single_quoted_text drops everything after a ' that never closes. The closing line of a here-string with a pipe, e.g. '@ | git hash-object --stdin, reduces to just ', which is_value_or_assignment accepts. A native command before it then reads as checked even though git resets $LASTEXITCODE. That cuts against the "errs towards might" intent. Returning "might reset" on an open quote, plus a reject case next to the accepted '@ | Add-Content ... one, would close it. The same hole hides (, {, & and later pipe elements on any line with an odd '.
  • reads_last_exit_code also matches inside single-quoted text (Write-Output '$LASTEXITCODE') and as a prefix ($LASTEXITCODEX). Pre-existing, but a read now ends the walk as "read", so it is a bit more reachable.
  • The doc comment after run_script still says run: | only. run_block_header also matches run: at any indent (nested keys under with:), which is fine for this repo but worth a note.
  • Since this is xtask-only, the PR body could say explicitly that no Gherkin scenario applies because the change is internal.

This is based on reading the diff; CI (the Rust test jobs in particular) was still running when I looked.

Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
From review of the pwsh contract:

- `without_single_quoted_text` dropped everything after a `'` that never
  closed, so a stray `'@ | git hash-object --stdin` reduced to a lone `'`,
  read as a plain value, and a native command before it counted as
  checked although git resets `$LASTEXITCODE`. A statement with a quote
  left open is now never inert, and never a read.
- `$LASTEXITCODE` counted as a read inside single quotes and as the
  prefix of a longer name (`$LASTEXITCODEX`). Only the whole name outside
  single quotes counts now.
- A folded `run: >2` block folded at its first line's indent. It now
  folds at the `run` key's column plus the indicator, as YAML does;
  PyYAML reads the new fixtures the same way.
- The statements and messages that #598 wrote while `run_block` read only
  `run: |` now say that every block-scalar form is read, and
  `run_block_header` notes that it takes a `run:` key at any depth.

No workflow in the repository changes verdict: the contract still finds
the same pwsh steps and the same allow-listed commands.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. All four points are addressed in 0e13ba7:

  • The unterminated single quote: see the inline thread.
  • reads_last_exit_code now only counts $LASTEXITCODE outside single quotes and as the whole name, so Write-Output '$LASTEXITCODE' and $LASTEXITCODEX no longer end the walk as a read. Both have reject cases that go red with the change reverted.
  • The doc comments after run_script and on pwsh_statements now say every block-scalar form is read; two failure messages and one doc comment from test(xtask): read wrapped step names, and say which run form is checked (ROCMAI-679) #598 that still said only run: | was read are fixed too. run_block_header's doc now says it takes a run: key at any depth, so a run: nested under with: would be read as the step's script; no workflow here has one.
  • No Gherkin scenario: this is an internal, test-only change to the xtask contract tests and changes nothing a CLI user can observe.

@rominf
rominf added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit 59dfabd Nov 6, 2026
30 checks passed
@rominf
rominf deleted the test/workflow-pwsh-contract-edge-cases branch November 6, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants