Skip to content

Community Markdown receives reliable pull request validation - #12

Open
Marius Storhaug (MariusStorhaug) wants to merge 16 commits into
mainfrom
pull-request-validation
Open

Marius Storhaug (MariusStorhaug) wants to merge 16 commits into
mainfrom
pull-request-validation

Conversation

@MariusStorhaug

@MariusStorhaug Marius Storhaug (MariusStorhaug) commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Contributors to MSXOrg/.github receive a named, read-only check for applicable source changes, including Markdown-only updates and PRs retargeted to main. PR descriptions use canonical guidance only; this validator does not require, copy, generate, or compare a PR template. Ordinary repositories need no validator installation or copied policy files.

Fixed: Changed source files receive complete, source-bound validation

Correct reported Markdown violations even if a file contains a markdownlint disable/configuration directive. Every selected .md file is read literally and linted with unchanged shared rules and noInlineConfig: true from an isolated temporary configuration. Inline directives, local ignores/configuration, and glob-like filenames cannot hide selected Markdown; valid directive examples still pass. Explicit opened, synchronize, reopened, and edited activities cover updates and retargeting to main without a new commit.

Use PowerShell 7.4+, Git, and Node.js/npm in this source repository. Fetch the intended base and commit the content being checked. The native runner discovers .github/tests without a Path override, and CI exercises that default. Pinned tooling is restored when needed:

pwsh -NoProfile -File .github\scripts\Invoke-PesterSuite.ps1 -RequiredVersion 6.1.0
pwsh -NoProfile -File .github\scripts\Test-OrganizationDefaults.ps1 -BaseRevision origin/main

Intentional template absence/deletion is valid. Removed paths come from the committed diff with case-sensitive identity and are not counted as linted content. Empty/unsupported-only inputs and zero-file Markdown scans fail. Test discovery cannot pass when no test passes, including skipped-only suites; summary and exit use the same success condition. The obsolete scaffold fixture/comparison is removed, not replaced.

Absence is not record-quality proof. Actual titles/descriptions still need canonical PR Format's boundary, ordered results, complete consumer actions, scoped evidence, and release decisions only on applicable release-managed routes. Other defaults and product-integration templates are distinct. #8 owns central removal at 990bfcbfd1275f80d58b3a9c51781f3873e22876; this PR does not edit its paths, forms, consumer copies, hooks, release contracts, or live governance.

Technical details

Verified head: a6c4484fa197f44bf45e244e17480cb40deafd3e, merge base 0c67251812ea306543f3a14ab473ae357cfb76ac. Run 41248565958 completed successfully at this exact head. Its job/check completed actual-head/source validation, literal Markdown lint, all 44 tests, and the remaining PowerShell/YAML/Actions/security routes. The earlier cancelled run is not used as validation evidence.

Exact check/source: Organization defaults validation, GitHub Actions integration/app 15368, from .github/workflows/Organization-Defaults-Validation.yml. Default-deny permissions, only contents read, actual-head verification, and disabled persisted credentials/language-status writes/PR-summary comments are preserved. Literal Markdown precedes other include-only Super-Linter routes. No publication, tag/label mutation, rules/access change, merge, or auto-merge was performed.

Case reproduced before correction Observed defect Verified correction
Malformed Markdown behind whole-file disable Released hosted parser exits 0 although plain malformed text exits 1 Literal shared-rule/no-inline check rejects it; corrected content passes
No-template source/deletion inputs Old copied-scaffold lookup fails Anonymous controller passes and distinguishes deletion from linted content
Edited retarget to main Default activities exclude edited Explicit activity coverage passes; no production retarget experiment is claimed
Mixed linter flags Actual hosted initialization fails Include-only regression and subsequent hosted routes pass
PowerShell wildcard path plus unchanged match Path invocation analyzes the wrong file; literal changed content produces its warning Surviving wildcard-bearing PowerShell inputs fail before counting lint inputs
Native runner default Root test-directory lookup fails .github/tests default, anonymous/full default invocation, and CI default execution pass
Case-only rename Deleted path is hidden Case-sensitive committed deletion/live lists report one of each
Skipped-only discovery Exit 0 with no passed regressions Zero-passed-test guard exits 1 with explicit diagnostic
Skipped-only summary Failure exit still writes success heading One Pester success decision controls summary/exit; regression verifies both

Every listed defect was demonstrated before correction. All 44 local and hosted tests pass in two suites, with zero failed/skipped/not-run tests. Source/head/containment, nonempty checks, literal paths, valid examples, links, staged/unstaged consistency, and control/backslash/finder-pruned guards remain intact.

Actual checked input scope Result
Existing applicable changed files 13
Changed Markdown actually linted 1: .github/VALIDATION.md
Checked deletions in this PR's base-to-head diff 0; #8 owns central removal and the removed fixture has no net base-to-head file
Shared configuration sources verified 4
Required template/scaffold files None

Deletion-only anonymous inputs report one checked deletion and zero linted Markdown, verify sources, and skip only inapplicable other-language lint. This is not an empty-input or Markdown pass. The isolated skipped-only suite is rejected and its summary is failure, not executed-regression proof.

#8 compatibility: Actual head 990bfcbfd1275f80d58b3a9c51781f3873e22876 versus base 0c67251812ea306543f3a14ab473ae357cfb76ac changes only two guides and central deletion. Both changed contents pass this literal/no-inline checker. The complete inventory has 42 files and zero PR templates. This is local source compatibility, not a hosted #8 check, record-quality approval, or merged-deletion claim; its checkout was untouched.

Exact local command Result
pwsh -NoProfile -File .github\scripts\Invoke-PesterSuite.ps1 -RequiredVersion 6.1.0 Passed without Path override: 44 tests in 2 suites
pwsh -NoProfile -File .github\scripts\Test-OrganizationDefaults.ps1 -BaseRevision origin/main -ExpectedHead (git rev-parse HEAD) Passed at this head: 13 inputs, 1 linted Markdown, 0 deletions, 4 configs
Invoke-ScriptAnalyzer -Path .github -Recurse -Settings .github\linters\.powershell-psscriptanalyzer.psd1 Passed with 1.25.0: zero diagnostics
Session-restored actionlint 1.7.12 on .github\workflows\Organization-Defaults-Validation.yml Passed
yamllint -c .github\linters\.yaml-lint.yml .github\workflows\Organization-Defaults-Validation.yml Passed
zizmor --offline --no-progress -c .github\linters\zizmor.yml .github\workflows\Organization-Defaults-Validation.yml Passed with 1.25.2
git --no-pager diff --check origin/main...HEAD Passed

Metadata pins configuration Git blobs from MSXOrg/docs 36da9154aa55d2052345fb2f28b812b29daf8bc9 and Super-Linter 4ce20838b8ab83717e78138c5b3a1407148e0918, not a scaffold fingerprint. The shared Pester helper has the documented source-local default and unified real-success adaptations. External SHA pins and permissions are unchanged.

Ownership: Repository Standard owns no-file policy; PR Format owns record quality. This ready, unmerged source is authoring context, not a runtime dependency. Absence does not certify coverage, actions, markers, sections, or evidence.

Complete review disposition: The exact-head review completed with no new inline findings; all nine threads are resolved and pagination completed. The entire summary was read. Its remaining concern cited a superseded cancelled CI run; the newer completed actual-head run above independently resolves that evidence gap. No zero-inline count was used to dismiss code or summary concerns. Every actionable finding in this work was reproduced and corrected.

Surface Standards/domain evidence Result
Controller/module/runner/tests PowerShell, errors/messages, deterministic testing, source/head/containment, literal content/committed identity, shared success verdict Aligned and verified locally/hosted
Workflow GitHub Actions/YAML, activities, native defaults, least privilege, immutable released actions Aligned; complete actual-head result verified
Metadata/guide Immutable sources, file-policy versus record ownership, Markdown/Natural Language Aligned
Source-only hooks #13 Separate follow-up, not consumer policy-copy enforcement

Plan/convergence: Seven ordinary commits complete the user-directed no-template revision, demonstrated Markdown/retarget fixes, and tightly coupled full-review corrections. #11 closes through this PR; #8/#10 and the hook follow-up remain separate. No forms, consumer copies, release signals, or live check requirements are changed.

Operational state: The repository owner externally marked this PR ready at 22:38:19Z; this session did not change readiness. Independent native read-back confirms Agent merge maintenance enabled, review/CI/conflict actions enabled, merge_pr=false, and GitHub autoMergeRequest=null. No merge or enqueue was performed. Inherited main rules still name no required status checks: requiring exactly Organization defaults validation from GitHub Actions/app 15368 remains a separately unapproved governance action. The visible ready state does not waive that governance boundary. This source route delivers public community files, not versioned artifacts, so no release-decision block or release label applies.

Related references

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The changed-file filter can select unrelated Markdown and skip filenames containing tabs or form feeds, while the documented non-Windows command cannot load its configuration.

3 open issues
What changed in this PR

Adds read-only pull request validation for organization community defaults, addressing #11 without changing the inherited template or README. Fallback-format delivery and local pre-commit hooks remain separate work.

Changes:

  • Adds an exact-head workflow for changed-input validation, Pester tests, and selected linters.
  • Validates changed fallback scaffolds against a pinned source fingerprint, with malformed-content and file-selection regression tests.
  • Documents local validation and adds shared lint configurations and Dependabot updates.
File Description
.github/​workflows/​Organization-Defaults-Validation.yml Runs the read-only validation job for PRs targeting main.
.github/​validation.psd1 Records the immutable scaffold source and fingerprint.
.github/​VALIDATION.md Documents validation commands, scope, and troubleshooting.
.github/​tests/​PullRequestValidation.Tests.ps1 Tests scaffold rejection and committed-file selection.
.github/​tests/​fixtures/​pull_request_template.md Supplies the isolated complete source scaffold.
.github/​scripts/​Test-OrganizationDefaults.ps1 Coordinates validation and emits the changed-file lint filter.
.github/​scripts/​PullRequestValidation.psm1 Implements scaffold checks and changed-input discovery.
.github/​scripts/​Invoke-PesterSuite.ps1 Restores pinned Pester and reports suite results.
.github/​linters/​zizmor.yml Configures workflow security auditing.
.github/​linters/​.yaml-lint.yml Supplies shared YAML lint rules.
.github/​linters/​.powershell-psscriptanalyzer.psd1 Supplies shared PowerShell analysis rules.
.github/​linters/​.markdown-lint.yml Supplies shared Markdown lint rules.
.github/​dependabot.yml Schedules weekly GitHub Actions dependency updates.

Review effort: Balanced

Comment thread .github/VALIDATION.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical validation gaps omit supported template locations, allow symlinks to bypass linting and scaffold checks, and can report success for malformed committed content by validating uncommitted working-tree changes.

3 open issues
3 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Path handling and linter discovery can silently skip selected files with backslashes or files in pruned directories, allowing validation to pass without checking all applicable changes.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Reject backslashes in input paths to avoid skipping Markdown files

.github/​scripts/​PullRequestValidation.psm1:196

On Linux, Join-Path treats a literal backslash in a filename as a directory separator, so a committed file such as changed\file.md can be silently omitted from MarkdownFiles and the linter filter. When another ordinary applicable file changes, validation can therefore pass without checking that Markdown file. Reject backslash-containing input paths before this selection and add a filename regression.

Medium severity Ensure linter-pruned files are not counted as validated inputs

.github/​workflows/​Organization-Defaults-Validation.yml:53

The pinned Super-Linter finder prunes directories such as node_modules, .venv, and __pycache__, but Get-ValidationInput still counts supported files there as applicable inputs. A malformed node_modules/broken.md alongside a valid ordinary Markdown change is therefore reported as an input but never linted, allowing an incomplete check to pass. Use discovery that includes every selected file, or reject excluded inputs explicitly, and add a regression for this case.

3 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because unpublished findings identify coverage gaps in pull-request validation triggers and Markdown linting.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a critical validation bypass can let changed Markdown pass without being linted, alongside missing checks when PRs are retargeted.

No open issues 🎉

Review effort: Balanced

@MariusStorhaug Marius Storhaug (MariusStorhaug) changed the title Organization fallback changes receive source-bound validation Community Markdown receives reliable pull request validation Oct 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Conflicting Super-Linter validation flags cause initialization failures before checks run, and wildcard-bearing PowerShell paths can cause an unchanged file to be analyzed instead of the selected file.

2 open issues

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because an unpublished finding identifies a default test-path failure that CI bypasses by supplying an explicit path.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because case-only renames can bypass deletion validation on Ubuntu, and this moderate finding was omitted from the published review.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed to assess a success-gate gap that allows skipped-only Pester runs to exit successfully without executing regression tests.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Skipped-only runs receive a success heading before the zero-passed-test guard exits with failure, so the summary verdict must use the same pass conditions as the exit decision.

1 open issue

Review effort: Balanced

Comment thread .github/scripts/Invoke-PesterSuite.ps1 Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cited CI run was cancelled before other-language lint completed, so a completed hosted result for the actual head is still needed before landing the validation gate.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run pull request validation for organization fallback changes

2 participants