A bounded list, not an umbrella. #17 was closed today for being open-ended, and this is deliberately not that: five named tests or test groups, each with the specific reason it cannot fail and the measurement that shows it. Nothing here is a request to "add more tests".
1. tests/New-PSBuildFileCatalog.tests.ps1 — two tests swallow any failure that is not the expected one
Calls New-FileCatalog with correct parameters and Returns a FileInfo object both wrap the real call like this:
try {
$result = New-PSBuildFileCatalog -ModulePath $testModulePath -CatalogFilePath $catalogPath -CatalogVersion 2
$result | Should -Not -BeNullOrEmpty
Test-Path $catalogPath | Should -BeTrue
} catch {
# If New-FileCatalog isn't available, just verify the function exists and accepts the params
if ($_.Exception.Message -match 'New-FileCatalog') {
$command = Get-Command New-PSBuildFileCatalog
$command.Parameters.ContainsKey('CatalogVersion') | Should -BeTrue
}
}If the call fails for any reason whose message does not mention New-FileCatalog — a denied path, a locked file, a changed parameter name, an operating system error — the catch runs, the if is false, and the test passes having asserted nothing at all. Measured with a probe that throws Access to the path is denied. from that position: Tests Passed: 1, Failed: 0.
On this machine the happy path is currently taken and coverage of New-PSBuildFileCatalog.ps1 is 15 of 15 commands, so the tests do work today. The defect is that they are constructed to stop working silently.
A third test, Defaults CatalogVersion to 2 (SHA256), asserts nothing about the default it is named for. Its own comment admits this:
$parameter = $command.Parameters['CatalogVersion']
# The default should be set in the function, we'll check by the ValidateRange attribute
$parameter | Should -Not -BeNullOrEmpty
$parameter is the ParameterMetadata for a parameter the previous test already established exists. The default value of 2 is never read.
Fix shape: drop the try/catch and let a real failure be a real failure — the containing context is already -Skip: guarded for non-Windows, which is the only condition the catch was defending against. For the third test, read the default off the function's abstract syntax tree, or assert that the catalog file the function produces is SHA256.
2. tests/Test-PSBuildScriptAnalysis.tests.ps1 — full instruction coverage, no message ever asserted
43 tests, 55 of 55 commands covered. The assertions are 8 bare Should -Throw and 12 bare Should -Not -Throw, with zero -ExpectedMessage and zero -Because anywhere in the file. Test-PSBuildScriptAnalysis has five throw sites drawing on three localized messages, and no test can tell them apart.
Measured by swapping two of the three messages in the module's localized data — ScriptAnalyzerErrors and ScriptAnalyzerIssues — so the Error threshold now reports "issues" and the Any threshold now reports "errors":
RESULT with swapped messages: Passed=43 Failed=0
That matters because the message is what a consumer sees. The three are not interchangeable: ScriptAnalyzerErrors means "your build has error-severity findings", ScriptAnalyzerWarnings covers the Warning and Information thresholds, and ScriptAnalyzerIssues means "any diagnostic at all was returned, at the Any threshold you asked for". A consumer reading "One or more ScriptAnalyzer errors were found!" after configuring SeverityThreshold = 'Any' would go looking for errors that are not there.
Honourable exception, and the model to copy: the three tests in Analyzer rule crash retry use Should -Invoke -CommandName 'Invoke-ScriptAnalyzer' -Times N -Exactly to pin the retry count at 2, 3, and 1 respectively. Those are the strongest tests in the suite — they assert a specific observable count rather than the absence of an exception, and they would fail on exactly the regression they were written for. Analyzes the supplied path recursively is the same shape with a -ParameterFilter.
Fix shape: the file already imports the module, so the $LocalizedData values can be read and used as Should -Throw -ExpectedMessage. Adding the expected message to the eight throwing tests is a mechanical change and would make all five sites distinguishable.
3. tests/Build-PSBuildModule.tests.ps1 — two "Emits no warning" assertions that are structurally unfalsifiable
Line 265 Context 'Building without compilation' It 'Emits no warning'
Line 557 Context 'Building without compilation from a scaffold loader' It 'Emits no warning'
Both assert $script:buildWarning | Should -BeNullOrEmpty after a build with no -Compile. Every Write-Warning in Build-PSBuildModule — the Export-ModuleMember warning at line 174 and NoScriptsToCompile at line 214 — is inside if ($Compile.IsPresent), which begins at line 140 and closes into the else at line 253. A non-compile build cannot emit either one, so neither assertion can fail no matter what happens to the function.
The two sibling assertions in compile contexts are fine: Emits no warning for a root module that does not call Export-ModuleMember (line 381) and Emits no warning for a mention inside a block comment (line 582) both run with -Compile, where the warning is reachable, and the second one specifically pins that a comment mentioning Export-ModuleMember is not reported as a call.
All four arrived in #205, merged earlier today alongside the Export-ModuleMember warning itself. This is not a criticism of that change — it is what happens when a "does not warn" assertion is copied into a context where warning was never possible.
Fix shape: delete the two non-compile assertions, or give them something falsifiable to assert. The genuinely interesting claim for the non-compile path is the one the comment at line 535 makes — "nothing is wrong with the loader; compilation is what breaks it" — and that is already asserted by Builds a module that exports its public functions immediately above it.
4. tests/Build-PSBuildHelp.tests.ps1 — the HelpInfoUri guard test would pass with the guard deleted
declines when the manifest declares no HelpInfoUri (line 250) strips HelpInfoUri from a scenario manifest, calls Build-PSBuildUpdatableHelp, and asserts:
$result.Threw | Should -BeFalse
$scenario.UpdatableHelpPath | Should -Not -Exist
The nouri scenario never has Build-PSBuildMarkdown run against it, and tests/fixtures/PSBuildTestFixture contains no docs directory, so DocsPath does not exist. Get-PSBuildHelpLocale finds nothing, $cabinetWork is empty, and the function returns at if (-not $cabinetWork) { return } — which is after the HelpInfoUri guard but still before OutputPath is created.
Measured, running the same shape twice with only the manifest differing:
Case 1 (HelpInfoUri PRESENT, guard does not fire, no docs tree): OutputPath exists = False
Case 2 (HelpInfoUri ABSENT, guard fires): OutputPath exists = False
Both assertions hold identically either way. The test also never asserts the HelpInfoUriRequired warning, which is the guard's only observable effect and the thing that tells a consumer what to fix.
Fix shape: give the scenario a real docs tree — generate markdown for it the way the cab context does — so that removing the guard would actually produce a cabinet, and assert on the warning as well as on the absent output.
5. tests/Help.tests.ps1 — the link-validation block currently generates no tests
if ($helpLinks) {
It "Help link <_> is valid" -ForEach $helpLinks {
(Invoke-WebRequest -Uri $_ -UseBasicParsing).StatusCode | Should -Be '200'
}
}No command in the module has a .LINK section. Measured across all twelve exported commands: the link count is 0 for every one, so this block expands to zero tests on every run, and the guard makes that invisible — a suite reporting hundreds of passing help tests contains no link check at all. If a .LINK is ever added and later removed, its coverage disappears the same way.
This is the mildest item in the list, and the guard itself is not wrong — its comment explains that it preserves Pester 5's empty -ForEach behaviour on Pester 6, which is a real problem it solves correctly.
Explicitly not claimed: the neighbouring if ($commandParameters) guard is not a hole. $commandParameters is derived from the command's own parameter metadata rather than from its help, so the block skips only a command that genuinely has no parameters, and the parameter-help assertions inside it do go red when help is missing — measured, twelve failures on Expected a value, but got $null or empty at line 167 when help is unavailable. An earlier reading of this file claimed that total help loss passes green; it does not, and Help is not auto-generated, Has description, and Has example code all fail in that case.
Fix shape: either add .LINK sections to the public commands so the block does something, or drop the block and record that the module does not use .LINK. A one-line note either way is enough.
Why these five, and not a general "improve the tests" issue
Each of the five is a test that reports success while being unable to report failure, which is worse than no test: it occupies the slot where real coverage would go, and it tells a reader the behaviour is checked. Items 2 and 3 are the ones with live consequences — a wrong analyzer message reaches consumers directly, and item 3 sits in a file that is otherwise among the strongest in the suite. Item 5 is a note rather than a defect, and is included so the list is complete rather than because it is urgent.
Related: #17 (closed today as too open-ended — this list is scoped to avoid the same fate), #205 (which added item 3's contexts and much of the surrounding good coverage), #96 (which added the analyzer test file), #149 (which added the help test file).
A bounded list, not an umbrella. #17 was closed today for being open-ended, and this is deliberately not that: five named tests or test groups, each with the specific reason it cannot fail and the measurement that shows it. Nothing here is a request to "add more tests".
1.
tests/New-PSBuildFileCatalog.tests.ps1— two tests swallow any failure that is not the expected oneCalls New-FileCatalog with correct parametersandReturns a FileInfo objectboth wrap the real call like this:If the call fails for any reason whose message does not mention
New-FileCatalog— a denied path, a locked file, a changed parameter name, an operating system error — thecatchruns, theifis false, and the test passes having asserted nothing at all. Measured with a probe that throwsAccess to the path is denied.from that position:Tests Passed: 1, Failed: 0.On this machine the happy path is currently taken and coverage of
New-PSBuildFileCatalog.ps1is 15 of 15 commands, so the tests do work today. The defect is that they are constructed to stop working silently.A third test,
Defaults CatalogVersion to 2 (SHA256), asserts nothing about the default it is named for. Its own comment admits this:$parameteris theParameterMetadatafor a parameter the previous test already established exists. The default value of2is never read.Fix shape: drop the
try/catchand let a real failure be a real failure — the containing context is already-Skip:guarded for non-Windows, which is the only condition thecatchwas defending against. For the third test, read the default off the function's abstract syntax tree, or assert that the catalog file the function produces is SHA256.2.
tests/Test-PSBuildScriptAnalysis.tests.ps1— full instruction coverage, no message ever asserted43 tests, 55 of 55 commands covered. The assertions are 8 bare
Should -Throwand 12 bareShould -Not -Throw, with zero-ExpectedMessageand zero-Becauseanywhere in the file.Test-PSBuildScriptAnalysishas fivethrowsites drawing on three localized messages, and no test can tell them apart.Measured by swapping two of the three messages in the module's localized data —
ScriptAnalyzerErrorsandScriptAnalyzerIssues— so theErrorthreshold now reports "issues" and theAnythreshold now reports "errors":That matters because the message is what a consumer sees. The three are not interchangeable:
ScriptAnalyzerErrorsmeans "your build has error-severity findings",ScriptAnalyzerWarningscovers the Warning and Information thresholds, andScriptAnalyzerIssuesmeans "any diagnostic at all was returned, at theAnythreshold you asked for". A consumer reading "One or more ScriptAnalyzer errors were found!" after configuringSeverityThreshold = 'Any'would go looking for errors that are not there.Honourable exception, and the model to copy: the three tests in
Analyzer rule crash retryuseShould -Invoke -CommandName 'Invoke-ScriptAnalyzer' -Times N -Exactlyto pin the retry count at 2, 3, and 1 respectively. Those are the strongest tests in the suite — they assert a specific observable count rather than the absence of an exception, and they would fail on exactly the regression they were written for.Analyzes the supplied path recursivelyis the same shape with a-ParameterFilter.Fix shape: the file already imports the module, so the
$LocalizedDatavalues can be read and used asShould -Throw -ExpectedMessage. Adding the expected message to the eight throwing tests is a mechanical change and would make all five sites distinguishable.3.
tests/Build-PSBuildModule.tests.ps1— two "Emits no warning" assertions that are structurally unfalsifiableBoth assert
$script:buildWarning | Should -BeNullOrEmptyafter a build with no-Compile. EveryWrite-WarninginBuild-PSBuildModule— theExport-ModuleMemberwarning at line 174 andNoScriptsToCompileat line 214 — is insideif ($Compile.IsPresent), which begins at line 140 and closes into theelseat line 253. A non-compile build cannot emit either one, so neither assertion can fail no matter what happens to the function.The two sibling assertions in compile contexts are fine:
Emits no warning for a root module that does not call Export-ModuleMember(line 381) andEmits no warning for a mention inside a block comment(line 582) both run with-Compile, where the warning is reachable, and the second one specifically pins that a comment mentioningExport-ModuleMemberis not reported as a call.All four arrived in #205, merged earlier today alongside the
Export-ModuleMemberwarning itself. This is not a criticism of that change — it is what happens when a "does not warn" assertion is copied into a context where warning was never possible.Fix shape: delete the two non-compile assertions, or give them something falsifiable to assert. The genuinely interesting claim for the non-compile path is the one the comment at line 535 makes — "nothing is wrong with the loader; compilation is what breaks it" — and that is already asserted by
Builds a module that exports its public functionsimmediately above it.4.
tests/Build-PSBuildHelp.tests.ps1— the HelpInfoUri guard test would pass with the guard deleteddeclines when the manifest declares no HelpInfoUri(line 250) stripsHelpInfoUrifrom a scenario manifest, callsBuild-PSBuildUpdatableHelp, and asserts:The
nouriscenario never hasBuild-PSBuildMarkdownrun against it, andtests/fixtures/PSBuildTestFixturecontains nodocsdirectory, soDocsPathdoes not exist.Get-PSBuildHelpLocalefinds nothing,$cabinetWorkis empty, and the function returns atif (-not $cabinetWork) { return }— which is after theHelpInfoUriguard but still beforeOutputPathis created.Measured, running the same shape twice with only the manifest differing:
Both assertions hold identically either way. The test also never asserts the
HelpInfoUriRequiredwarning, which is the guard's only observable effect and the thing that tells a consumer what to fix.Fix shape: give the scenario a real docs tree — generate markdown for it the way the
cabcontext does — so that removing the guard would actually produce a cabinet, and assert on the warning as well as on the absent output.5.
tests/Help.tests.ps1— the link-validation block currently generates no testsNo command in the module has a
.LINKsection. Measured across all twelve exported commands: the link count is0for every one, so this block expands to zero tests on every run, and the guard makes that invisible — a suite reporting hundreds of passing help tests contains no link check at all. If a.LINKis ever added and later removed, its coverage disappears the same way.This is the mildest item in the list, and the guard itself is not wrong — its comment explains that it preserves Pester 5's empty
-ForEachbehaviour on Pester 6, which is a real problem it solves correctly.Explicitly not claimed: the neighbouring
if ($commandParameters)guard is not a hole.$commandParametersis derived from the command's own parameter metadata rather than from its help, so the block skips only a command that genuinely has no parameters, and the parameter-help assertions inside it do go red when help is missing — measured, twelve failures onExpected a value, but got $null or emptyat line 167 when help is unavailable. An earlier reading of this file claimed that total help loss passes green; it does not, andHelp is not auto-generated,Has description, andHas example codeall fail in that case.Fix shape: either add
.LINKsections to the public commands so the block does something, or drop the block and record that the module does not use.LINK. A one-line note either way is enough.Why these five, and not a general "improve the tests" issue
Each of the five is a test that reports success while being unable to report failure, which is worse than no test: it occupies the slot where real coverage would go, and it tells a reader the behaviour is checked. Items 2 and 3 are the ones with live consequences — a wrong analyzer message reaches consumers directly, and item 3 sits in a file that is otherwise among the strongest in the suite. Item 5 is a note rather than a defect, and is included so the list is complete rather than because it is urgent.
Related: #17 (closed today as too open-ended — this list is scoped to avoid the same fate), #205 (which added item 3's contexts and much of the surrounding good coverage), #96 (which added the analyzer test file), #149 (which added the help test file).