Surface checks for generated SDKs (design + runnable demo) - #11186
Draft
iscai-msft wants to merge 9 commits into
Draft
Surface checks for generated SDKs (design + runnable demo)#11186iscai-msft wants to merge 9 commits into
iscai-msft wants to merge 9 commits into
Conversation
microsoft-github-policy-service
Bot
added
the
emitter:client:python
commit: |
Contributor
|
❌ There is undocummented changes. Run The following packages have changes but are not documented.
Show changes |
|
You can try these changes here
|
microsoft-github-policy-service
Bot
added
the
spector
Member
There was a problem hiding this comment.
are those files generated, do we want them commited"?
Member
Author
There was a problem hiding this comment.
this is just for the demo, going to remove them after I do my demo
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
microsoft-github-policy-service
Bot
added
the
lib:http-specs
Add the @surfaceDoc decorator to spector for describing expected properties of generated SDK surfaces (naming, access, hierarchy, paging, etc.) that wire tests cannot observe. Includes: - @surfaceDoc decorator definition and implementation in spector - Surface checks manifest generation (Markdown table format) - Shared language-agnostic verify-surface-checks runner - @surfaceDoc annotations on http-specs: pageable, special-words, encoded-name/json, single-discriminator, not-discriminated - Python client-criteria support (verifiers.json, context.md, verify runner) - AI orchestrator prompt for residual checks Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
iscai-msft
force-pushed
the
python/spectorClientCriteria
branch
from
July 23, 2026 17:04
ca7456e to
b1adedd
Compare
The @surfaceDoc decorator uses SpectorStateKeys.SurfaceDoc to store state, but the key was never declared in the state map, which would cause a runtime error when the decorator is used. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
…riteria-clean # Conflicts: # packages/http-specs/package.json # packages/http-specs/specs/payload/pageable/main.tsp # packages/http-specs/specs/special-words/main.tsp # packages/spector/generated-defs/TypeSpec.Spector.ts # packages/spector/package.json
Convert @surfaceDoc from extern dec to auto dec:
- Use positional params (category, subject, expected, doc?)
- Add tspconfig.yaml with kind:project and auto-decorators feature
- tspd gen-extern-signature now generates getSurfaceDoc/setSurfaceDoc accessors
- Remove surfaceDoc from TypeSpecSpectorDecorators and decorator map
- Update all spec call sites from keyword #{} syntax to positional
- Clean up tests: remove autoDecOptions workaround
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
Change @surfaceDoc to hybrid call style:
@surfaceDoc(Subject, #{ category: "...", expected: "...", doc: "..." })
subject stays positional (type reference), rest uses keyword object literal.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
…tring
@surfaceDoc now uses fully keyword syntax:
@surfaceDoc(#{ category: "naming", expected: "W", subject: "Widget" })
- subject is optional string, defaults to target's name when omitted
- Removed SurfaceSubject type (no longer holds Type references)
- Removed kind/origin from SurfaceDetails (no longer derivable)
- Simplified buildSurfaceDetails, removed getSurfaceParent/getSurfaceKind
- Location now derived from target (decorated element) not subject
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
…rd surfaceDoc
- Remove `casing` from Rule interface and `byKind` modifier from resolveVar
- Simplify verifiers.json: naming uses {expected} directly, remove client-location
- Update verify-surface-checks tests to use new details shape (no kind/name)
- Update README.md with new decorator syntax and placeholder table
- Update context.md to remove kind-based casing references
- Regenerate surface-checks.md from current annotations (6 checks)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
Add optional `kind` field to the @surfaceDoc decorator (e.g. 'property', 'model', 'enum') so each language verifier can apply per-kind conventions without encoding language-specific info in the spec. - Add `kind?: string` to surfaceDoc check model in main.tsp - Thread kind through StoredSurfaceDoc, SurfaceDoc, SurfaceDetails - Re-add `casing` map to Rule interface and `byKind` modifier to resolveVar - Update Python verifiers.json with per-kind casing map - Add kind='property' to encoded-name naming spec annotation - Add byKind recasing test (property → snake_case) - Update README.md and context.md - Regenerate surface-checks.md (now shows kind in details) E2E verified: spec says expected='defaultName' + kind='property', Python verifier recases to 'default_name' via byKind, finds it in generated code. Wrong names correctly fail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
Move check-surface.prompt.md to .github/prompts/ (it's language-agnostic,
parameterized via ${input:emitter}).
Move design README.md to packages/spector/docs/surface-checks.md (it
documents the @surfaceDoc system which lives in spector).
Python emitter now only has the 2 files a language needs:
- eng/scripts/client-criteria/context.md (emitter facts + concepts)
- eng/scripts/client-criteria/verifiers.json (declarative routing)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54fb668c-64c3-41e5-8933-c4300c884974
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surface checks for generated SDKs (design + runnable demo)
Status: draft / RFC. This adds the design and a working proof-of-concept for
surface checks — validating properties of a generated SDK that the wire
tests can't observe (naming, access/visibility, operation location, inheritance
reshaping, …), described once in the spec and checked per emitter.
Today these are checked by hand, once per language. The idea: annotate a spec
element with a new spector decorator
@surfaceDoc("…"), precompute a sharedchecks manifest, and let each emitter supply a
context.md(what each conceptlooks like in its output) + optional
verifiers.json(routing table). A sharedverify.pydecides everything with a verifier deterministically; only the fuzzyresidual goes to an AI orchestrator prompt.
Key property: whether a check is deterministic or AI is decided by whether the
language authored a verifier for it — not by the scenario. Add a verifier and the
same check is promoted from AI to deterministic; the spec assertion never changes.
What's in this PR (all under
packages/http-client-python/eng/scripts/client-criteria/)verify.py— deterministic runner (--batchmode + per-check routines).verifiers.json/verifiers.promoted.json— routing tables (the.promotedvariant adds the
hierarchyverifier to demonstrate promotion).demo-checks.json— self-contained manifest with three examples.context.md,README.md— emitter facts + design write-up..github/prompts/check-client-surface.prompt.md— AI orchestrator prompt (design).Runnable demo
cd packages/http-client-python python eng/scripts/client-criteria/verify.py --batch \ --checks eng/scripts/client-criteria/demo-checks.json \ --verifiers eng/scripts/client-criteria/verifiers.json \ --generated-root tests/generated --flavor azure --language python2 of 3 checks (client naming, client location) resolve deterministically; the
hierarchy check routes to AI. Rerun with
verifiers.promoted.jsonand all threeare deterministic — no change to the spec assertion.
Full design write-up: https://gist.github.com/iscai-msft/8be1b3ecf016bfd3bfbf3a7f6857d78c
Not yet built (follow-ups)
@surfaceDocdecorator + precompute (listSurfaceDocs,generate-surface-checks.ts) — currently design-only./check-surfaceorchestrator prompt into CI.Co-authored-by: Copilot