Skip to content

fix(cli): align integration guidance with available output - #5041

Open
mnriem wants to merge 4 commits into
github:mainfrom
mnriem:mnriem-artifact-free-cli-guidance
Open

mnriem wants to merge 4 commits into
github:mainfrom
mnriem:mnriem-artifact-free-cli-guidance

Conversation

@mnriem

@mnriem mnriem commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Description

Make shared CLI guidance describe an integration's actual output and executable requirements.

  • When list_command_templates() is empty, keep project-directory guidance and recommend specify integration list followed by specify integration switch <key>. Do not recommend unavailable agent commands or enhancement commands.
  • Report non-CLI integrations as having no CLI requirement, rather than inferring IDE-only usage, in both built-in and installed-external checks and in integration list.
  • Preserve ordinary command and skill guidance and document the behavior in the reorganized integration CLI reference.

This addresses the shared CLI concern reported in #5030 (comment). The implementation is capability-based, without integration-key exceptions, and the empty-inventory regression uses a test-only override of the existing generic integration. It does not change integration setup, event ownership, command rendering, or core templates.

Latest main is merged, including the none integration and the integration-documentation reorganization. The documentation conflict was resolved by preserving the guidance in docs/integrations/cli.md, not restoring the deleted reference.

Testing

  • Tested locally with uv run specify --help
  • Ran existing tests with uv sync && uv run pytest
  • Tested with a sample project (if applicable)

Exact automated commands and outcomes:

  • uv sync --extra test — passed; dependencies installed into this worktree's own virtual environment.
  • uv run specify --help — passed on the final merged branch.
  • Before the fix, LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/specify_cli/test_command_init.py tests/specify_cli/test_command_check.py tests/specify_cli/integrations/test_command_list.py -q — 5 failed, 25 passed. The failures reproduced missing integration-selection guidance for new-directory and --here initialization, IDE-only labels in built-in and external checks, and the integration-list label.
  • After the fix, LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/specify_cli/test_command_init.py tests/specify_cli/test_command_check.py tests/specify_cli/integrations/test_command_list.py tests/specify_cli/integrations/test_installed_adapters.py -q — 702 passed.
  • After merging latest main, LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/specify_cli/test_command_init.py tests/specify_cli/test_command_check.py tests/specify_cli/integrations/test_command_list.py tests/integrations/test_integration_none.py -q — 37 passed.
  • git diff --check and git diff --cached --check — passed.

Sample-project coverage invokes the real CLI in temporary projects, verifies the empty-template path generates no generic command directory, and checks that normal command and Copilot skill guidance still names the artifacts actually generated. Negative coverage verifies unavailable recommendations and IDE-only labels are absent, and integrations without required executables are not probed.

The DocFX preview built successfully with three warnings (parent-directory resource/content globs and an existing design-document link). The full repository test suite was not run. No manual agent slash-command execution was performed; this change affects CLI presentation rather than slash-command behavior.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: GitHub Copilot using GPT-6.1 Sol (gpt-6.1-sol) in autonomous mode on behalf of @mnriem authored the implementation, tests, documentation, merge-conflict resolution, and this PR description, and ran the reported automated checks. No personal human testing or line-by-line review is claimed. The contributor explicitly authorized this submission after being informed of the open-PR prioritization policy.

mnriem and others added 2 commits October 20, 2026 13:10
Show integration selection and switching guidance when an integration has no command templates, and report CLI requirements without inferring IDE-only usage. Preserve ordinary command and skill next steps.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

# Conflicts:
#	docs/reference/integrations.md
Copilot AI max review requested due to automatic review settings October 20, 2026 20:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The localized presentation changes have focused regression coverage, with only non-blocking wording refinements remaining.

1 open issue
What changed in this PR

Aligns Specify CLI guidance with integrations’ actual output and executable requirements.

Changes:

  • Recommend integration selection when no command templates exist.
  • Remove IDE-only assumptions from CLI requirement labels.
  • Add regression coverage and update the integration reference.

Reported test results were considered; tests were not rerun here.

File Description
tests/​specify_cli/​test_command_init.py Covers empty-output, command, and skill guidance.
tests/​specify_cli/​test_command_check.py Checks neutral labels and avoids unnecessary tool probes.
tests/​specify_cli/​integrations/​test_installed_adapters.py Updates external-adapter output expectations.
tests/​specify_cli/​integrations/​test_command_list.py Verifies CLI requirement labels.
src/​specify_cli/​integrations/​command_list.py Removes IDE classification from requirement labels.
src/​specify_cli/​command_init.py Adds guidance for empty template inventories.
src/​specify_cli/​command_check.py Reports integrations without CLI requirements neutrally.
docs/​integrations/​cli.md Documents requirement labels and initialization guidance.

Review effort: Max

Comment thread docs/integrations/cli.md Outdated
Clarify that integration list reports config.requires_cli while external adapter descriptor tools are validated separately during installation and specify check.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings October 20, 2026 21:08
@mnriem

mnriem commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Clarified the documentation in 7497edf: CLI Required reflects the agent CLI flag, while external adapter requires.tools dependencies are validated separately during installation and specify check, including when the column says no. Runtime behavior is unchanged. DocFX rebuilt successfully with the same three existing warnings; git diff --check passed. No executable tests were rerun for this documentation-only correction.

Authored on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol (gpt-6.1-sol), autonomous mode; documentation correction, automated documentation build, and this review-round summary.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The built-in check test’s no-probe assertion targets an entry removed before execution, leaving that assertion ineffective.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Negative test uses unregistered integration and never exercises target

tests/​specify_cli/​test_command_check.py:42

The injected artifact-free entry is removed before check() iterates over integrations. For this fresh project root, load_installed_integrations() clears and rebuilds AGENT_CONFIG from INTEGRATION_REGISTRY (installer.py:578-602, 820-849), where this fake integration is absent.

The not-probed assertion therefore passes without exercising its target; the output assertions only cover other built-ins. Use the registered none integration and assert it is not passed to check_tool, so this negative case actually checks the behavior required by CONTRIBUTING.md:227-247.

1 resolved since last review

Review effort: Max

Use the registered none integration for the no-probe assertion and require its own no-CLI output, so registry refresh cannot make the negative case vacuous.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings October 20, 2026 21:50
@mnriem

mnriem commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed the ineffective negative test in 694d461. The built-in case now uses the registered none integration, asserts that its own None (shared scaffolding only) (no CLI required) line is present, and verifies that none is never passed to check_tool. It no longer injects an entry that registry refresh removes. Runtime behavior is unchanged.

Evidence: adding a target-specific output assertion to the old fake-entry test reproduced the missing-target failure. LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/specify_cli/test_command_check.py tests/integrations/test_integration_none.py -q passed all 10 tests. An isolated in-memory mutation setting NoneIntegration.config["requires_cli"] to true made the corrected test fail specifically at the no-probe assertion, confirming it detects the prohibited behavior. git diff --check passed.

Authored on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol (gpt-6.1-sol), autonomous mode; test correction, automated validation, and this review-round summary.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused presentation changes have regression coverage, with only a nonblocking wording issue remaining.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Clarify wording for switching targets and command template inventory

src/​specify_cli/​command_init.py:1065

Optional: use "Switch this project to it" here and "switch to it" in docs/integrations/cli.md:540. For an uninstalled target, switch removes the current default (docs/integrations/cli.md:217–220), whereas the repository uses "add" for side-by-side install (docs/integrations/index.md:24–42). "Add" can imply that the current integration stays installed. In the documentation paragraph, "has no command templates" also describes the inventory check more accurately than "generates no command templates".

Review effort: Max

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants