Skip to content

copilot: discover and switch between all authorized models - #1218

Merged
xi-db merged 11 commits into
mainfrom
xi-db/stack/copilot-model-discovery
Oct 22, 2026
Merged

xi-db merged 11 commits into
mainfrom
xi-db/stack/copilot-model-discovery

Conversation

@xi-db

@xi-db xi-db commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


What did you change, and why?

Change: ug copilot now registers every Claude and GPT model the user can use on the workspace and lets them switch between them inside one Copilot session.

  • Per-session registry. Each launch writes a Copilot providers.json (COPILOT_PROVIDERS_CONFIG) to ~/.copilot/ug-providers-<id>.json (0600), deleted when the session ends. Each session holds a lock on a sibling .lock file, and registries whose lock nobody holds (a killed session) are swept at the next launch and by ug revert. The user's own ~/.copilot/providers.json is never touched.
    • Providers: dbx-anthropic (Anthropic Messages), dbx-responses (Responses, GPT-5+) and dbx-completions (Chat Completions, older GPT). All of them point at the existing loopback auth proxy with the session secret; only providers that have models are written.
    • Claude entries carry Copilot's catalog modelId, so Copilot sends Claude request settings, plus prompt/output limits. Without explicit limits Copilot assumes a 128K prompt for registry models.
    • The launch env sets COPILOT_PROVIDERS_CONFIG, COPILOT_MODEL=<provider>/<id> and COPILOT_OFFLINE=true, and drops inherited COPILOT_PROVIDER* settings. The single-provider env (COPILOT_PROVIDER_TYPE/BASE_URL/WIRE_API/MODEL_ID) and the COPILOT_PROVIDER_WIRE_MODEL override are gone.
  • Discovery.
    • The catalog is classified from the UC model-services listing, which now keeps each model's supported_api_types. When that listing is empty, it falls back to the gateway listings. No extra requests.
    • Names decide the wire: claude-* (including bare databricks-claude-*, GitHub Copilot does not work with Anthropic models #108) -> Anthropic; gpt-N -> Responses for N >= 5, else Completions. Advertised API types only exclude.
    • Excluded, with a reason: Gemini (the gateway rejects Copilot's stream_options), open-source models (not validated with Copilot yet), non-chat models, claude-opus-4-1 (the gateway 400s Copilot's thinking settings).
    • The catalog is saved per workspace. A failed listing keeps the last good one and warns; a listing with nothing usable is an error.
  • Model selection.
    • --model/-m accept a short name, system.ai.* or Copilot's provider/id. They're rewritten to provider/id, because Copilot only matches that form; Copilot has no -m, so it becomes --model.
    • An unknown or unsupported model fails before Copilot starts and lists the available models.
    • The start model is: --model, then the budget recommendation or managed default (now passed through LaunchOptions.launch_model; it was ignored for Copilot before), then the managed list, then the newest Sonnet/Opus/Haiku, then GPT.
  • Managed config. The admin's model_services list for Copilot now reaches state through the existing static-models seam and replaces discovery. Entries Copilot can't serve are skipped with a warning, and the wizard's Copilot families drop Gemini/OSS.
  • Visibility. ug status lists Copilot's catalog and default, and ug doctor reports the model counts.
  • Other changes.
    • Copilot CLI 1.0.91 or newer is required: older versions ignore the registry.
    • ug copilot -p keeps stdout for Copilot by moving ug's banners to stderr.
    • The shared launch_token helper is reused.

Why: Copilot's env-var BYOK supports one provider per session, so /model couldn't reach Databricks models across families, and switching between Claude and GPT meant relaunching. Copilot also only saw the newest model per Claude family plus the Codex catalog, which leaked gpt-oss. Copilot doesn't discover gateway models itself the way Claude Code and Codex do, so ug builds the catalog.

BEFORE:

  • ug copilot pins one model and wire per launch, and /model can't switch to another Databricks model family.
  • Older Claude versions and databricks-claude-* ids don't reach Copilot or take the OpenAI path.
  • A managed Copilot model list and the budget launch model are ignored.

AFTER:

  • /model lists all authorized Claude and GPT models (on dogfood: 14 Claude + 18 GPT) and switches across providers mid-session, carrying tool history.
  • --model accepts friendly names and fails fast with the list.
  • Managed lists and the budget model apply.

How do you know it works?

Testing:

  • Unit:
    • classifier, catalog order and precedence, keep-last-good vs definitive empty, resolver and argv rewrite, launch-model order, Claude limits table and fallback;
    • registry content (pinned provider type/wireApi/baseUrl), file mode and cleanup, sweep of dead sessions, env, version gate;
    • managed static list and unservable warnings, status/doctor rows, -p stdout, the listing's API types;
    • tests/test_agent_copilot.py runs the real proxy with a stubbed Copilot process.
    • The full unit suite passes except 8 tests in TestInstallToolBinary / test_e2e_user_agent, which also fail on main in this environment. just lint is clean.
  • Integration (new tests/integration/test_ug_copilot_models.py, non-blocking Copilot lane):
    • one setup runs a Claude task, then a GPT task selected as dbx-responses/<model>;
    • an older Claude from the full catalog;
    • an unknown model exits 1 before Copilot starts and lists the models;
    • a TUI journey answers folder trust, completes a Claude task, runs /model dbx-responses/<model>, completes a GPT task and /exits.
    • Model evidence comes from Copilot's own session-state/*/events.jsonl (assistant.message.model, session.start.selectedModel, session.model_change.newModel). The existing headless journeys now also assert the answering model and that no registry is left.
  • Live (Copilot 1.0.98, ug from this branch, dogfood):
    • claude-haiku-4-5, dbx-responses/system.ai.gpt-5-4-nano and claude-sonnet-4-5 headless tasks completed with clean stdout;
    • system.ai.gemini-3-flash failed before launch with the list;
    • in the TUI, Haiku answered, /model dbx-responses/system.ai.gpt-5-4-nano printed "Model changed ... for this session", and GPT answered;
    • no registry was left.
    • The new integration journeys run in CI for the first time on this PR.

Not in this PR: open-source models (next on this stack), Model Provider Service models in the registry, GPT token limits, mixing GitHub-seat models into /model (depends on the seat-based-auth decision).

Stacked on #1217.

Test infra deflakes (Copilot lanes): on Windows, a dedicated pwsh install directory is added to the isolated test PATH, because Copilot's shell tool tries pwsh first and Windows PowerShell alone can miss its 40s startup probe, which left the revoke step with no shell. The Copilot GPT journeys also get the fixture's full path (FileTask.prompt_with_path), because gpt-5-4-nano sometimes drops with spaces when it rebuilds the path.

This pull request and its description were written by Isaac.

Comment thread src/ucode/agents/copilot.py
Comment thread src/ucode/cli.py
Comment thread tests/test_agent_copilot.py Outdated
@xi-db
xi-db force-pushed the xi-db/stack/copilot-model-discovery branch from 70b8f0e to 5b488aa Compare October 21, 2026 12:27
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None confirmed from the frozen diff and available context.

Security vulnerabilities

No new vulnerability confirmed by static inspection. Inherited Databricks credentials predate this change; removing them without preserving MCP authentication could break existing launches.

Non-blocking notes

  • src/ucode/doctor.py:190: can doctor resolve managed state like status does? with managed model_services, it counts discovered models rather than the catalog launch registers?
  • src/ucode/agents/copilot.py:142: are dotted model names supported? _model_tail turns gpt-6.1-sol into 1-sol; preserve version punctuation and restore classification cases if supported?
  • tests/integration/test_ug_copilot_models.py:35: can this journey absorb the repeated headless Claude/GPT tasks, retaining the smoke target, provider-ref check, and cleanup assertions?
  • tests/integration/utils/agents/copilot.py:11: can the existing evidence unit suite cover valid, unrelated, and malformed Copilot records so transcript parsing cannot silently weaken live assertions?
  • tests/test_cli.py:6573: can Copilot extend the actual stdout-routing scenario, and the headless test parse raw stdout? the predicate and tolerant answer parser do not prove clean JSON output?
  • tests/test_agent_copilot.py:841: can TestManagedKeys.test_excludes_credentials be removed? the preceding exact ownership-set assertion already protects this contract?

Approach

The approach is sound: reuse the loopback proxy, shared token selector, cross-platform locks, and existing evidence adapters. Keep model and argument matrices local; reserve live journeys for native provider behavior, authentication, migration, and same-session TUI switching.

Summary

Diff size (computed): 40 files; +2811 / -544 lines (3355 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 26 1821 330 64.1%
Documentation 1 20 6 0.8%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 13 970 208 35.1%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

The intended outcome is discovering compatible authorized Claude/GPT models and switching between them within one Copilot session. The diff adds classification, managed catalogs, launch selection, private session registries, proxy authentication, cleanup, diagnostics, documentation, and supporting CI coverage.

Most implementation breadth follows that stateful workflow. Shared authentication extraction also explains the Codex changes. Concrete reductions are available in repeated headless journeys and duplicate setup, rather than removing distinct provider or lifecycle boundaries.

The frozen snapshot’s checksum matches the supplied value. Both trusted review policies were read in full. Review was static: no PR code, tests, lint, or live gateway calls were executed. The declared head was unavailable locally, so findings use the frozen diff’s new-file coordinates.

Tests

Test-by-test assessment

Rows group named tests only where the assessment and recommendation match. Parameterized rows include all their parameters. Removed cases are identified explicitly.

Test / case Behavior protected Layer Needed? Action / rationale
tests/AGENTS.md:95 — integration scope paragraph Documents Copilot journeys and the advisory lane Documentation keep Relevant suite documentation; reviewed as PR content, not trusted policy.
tests/README.md:435 — Copilot coverage rows Maps behavior to coverage and limitations Documentation keep Supports understanding and maintaining the new coverage.
tests/integration/README.md:206 — Copilot journeys, runner selection, suite inventory and CI lane Explains reproduction and live-test boundaries Documentation keep Necessary operational context; near-natural-expiry coverage remains in the existing cache tests.
tests/integration/conftest.py:22 — pytest_collection_modifyitems Runs explicitly selected agent cases Collection support keep Extends the existing selector.
tests/integration/pytest.ini:15 — copilot marker Registers Copilot test selection Collection support keep Required companion to collection and CI selection.
tests/integration/test_ug_copilot_headless.py:16 — test_ug_copilot_headless_completes_file_task Native Claude task, selected model, credential-free disk artifacts and cleanup Live process/transport keep Useful smoke entry point; strengthen raw-stdout validation as noted above.
tests/integration/test_ug_copilot_headless.py:55 — test_ug_copilot_headless_gpt_responses_task Native Responses-wire task and cleanup Live process/transport consolidate Preserve this wire boundary within the one-setup/provider-ref journey.
tests/integration/test_ug_copilot_lifecycle.py:94 — test_ug_copilot_refreshes_a_revoked_token_mid_session Real bearer-command refresh following revocation Live authentication/process keep Establishes the real issuer and refresh boundary beyond hermetic retries.
tests/integration/test_ug_copilot_lifecycle.py:134 — test_ug_copilot_reports_expired_sign_in Native refresh-failure presentation and file-only diagnostics Live authentication/process keep Distinct failure journey with meaningful terminal-versus-log assertions.
tests/integration/test_ug_copilot_lifecycle.py:174 — test_ug_configure_copilot_repeat_and_revert Repeated setup, usable launch, repeated revert and preservation Live CLI lifecycle keep Covers the public lifecycle; its empty MCP input provides narrower preservation evidence than arbitrary populated user configuration.
tests/integration/test_ug_copilot_lifecycle.py:221 — test_ug_copilot_upgrade_scrubs_token_saved_by_older_ug Migration from actual historical-release output Cross-version process keep Historical installation establishes provenance that hand-built fixtures cannot.
tests/integration/test_ug_copilot_lifecycle.py:26 — _configure; tests/integration/test_ug_copilot_models.py:22 — _configure Executes public configuration with explicit inputs Scenario support consolidate Consolidating repeated journeys can also remove duplicate setup mechanics.
tests/integration/test_ug_copilot_lifecycle.py:39 — _pat_sign_in, nested step, _minted_count Bearer-command setup and mint/revocation evidence Authentication support keep Specific mechanics required by the authentication journeys.
tests/integration/test_ug_copilot_lifecycle.py:60 — _delete_minted Removes test-created PATs Authentication support keep Necessary teardown; cleanup attempts are not evidence that every deletion succeeded.
tests/integration/test_ug_copilot_lifecycle.py:71 — _install_token_saving_ug, TOKEN_SAVING_UG Reproduces historical configuration behavior Cross-version support keep Justified by the migration boundary rather than ordinary launch coverage.
tests/integration/test_ug_copilot_models.py:35 — test_ug_copilot_one_setup_runs_claude_then_gpt_by_provider_ref One configuration supports sequential provider-ref launches Live process/transport consolidate Combine repeated headless task coverage here while retaining both wires and cleanup assertions.
tests/integration/test_ug_copilot_models.py:78 — test_ug_copilot_runs_an_older_claude_from_the_full_catalog Registration extends beyond newest family defaults Live native-provider keep Distinct full-catalog regression using the older Claude model.
tests/integration/test_ug_copilot_models.py — test_ug_copilot_tui_switches_from_claude_to_gpt_with_model_command Same-session cross-wire switching, trust and normal exit Live TUI/process keep Separate headless invocations cannot establish in-session switching.
tests/integration/utils/agents/copilot.py:11 — is_child_session, _messages, assistant_answers, completed_task_models, selected_models Interprets native transcript records and model identity Evidence support keep Appropriate adapter boundary; add focused offline parser coverage as noted above.
tests/integration/utils/constants.py:10 — COPILOT_OLDER_CLAUDE_MODEL, COPILOT_HEADLESS_FLAGS Supplies stable, explicit native CLI inputs Scenario support keep Shared inputs avoid divergent journey configuration.
tests/integration/utils/evidence.py:8 — _AGENT_HELPERS Routes Copilot transcripts to their adapter Evidence support keep Reuses existing dispatch instead of introducing parallel infrastructure.
tests/integration/utils/evidence.py:135 — assert_no_bearer_on_disk Detects supplied bearer values and obsolete credential keys on disk Evidence support keep Useful scoped assertion; does not prove a credential-free child environment or detect unknown newly minted values.
tests/integration/utils/evidence.py:150 — copilot_selected_models, assert_no_session_registry_left Observes model selection and completed-session cleanup Evidence support keep Small reusable assertions; selection metadata is not independent gateway-routing evidence.
tests/integration/utils/evidence.py:204 — FileTask.assert_headless_answer Copilot branch Checks exit, completed turn and assistant-held unknown value Evidence support keep Extends established task evidence; tolerant parsing must not substitute for strict stdout checks.
tests/integration/utils/pat_bearer.py:16 — databricks, main modes mint, revoke, sign-out, cleanup Drives real short-lived token lifecycle Authentication/process support keep Required for the real refresh journeys; token lifecycle is distinct from mocked refresh behavior.
tests/integration/utils/terminal.py:212 — TerminalProcess.__exit__ Collects native Copilot transcript evidence Terminal support keep Fits the existing artifact mechanism.
tests/integration/utils/terminal.py:355 — AgentTerminal.boot Copilot trust/title branches Handles visible trust and interactive readiness Terminal support keep Required to reach the real TUI boundary.
tests/test_agent_codex.py:984 — TestCodexLaunch._patch, test_sets_oauth_token, test_provider_discovery_uses_custom_oauth_token, nested custom_token, test_launches_legacy_config_after_configure, test_requires_populated_ucode_profile Preserves launch, authentication and legacy-profile behavior Unit/component keep Patch-target changes follow shared token extraction without weakening existing assertions.
tests/test_agent_codex.py:2260 — TestOtelTokenProvider.test_token_provider_uses_default_profile, test_token_provider_uses_custom_oauth_when_configured Preserves principal selection for telemetry Unit keep Existing authentication-source coverage remains sufficient for the extraction.
tests/test_agent_copilot.py:21 — catalog, isolate_copilot_config_paths Supplies explicit catalog inputs and isolated filesystem paths Unit support keep Small helpers support deterministic tests without live setup.
tests/test_agent_copilot.py:53 — TestClassifyCopilotModel.test_names_decide_the_wire Classifies Claude, Responses and Completions models Unit keep Model matrices belong here; dotted spelling support needs the clarification noted above.
tests/test_agent_copilot.py:67 — TestClassifyCopilotModel.test_excluded_models_say_why Rejects unsupported families, non-chat models and incompatible aliases Unit keep Parameters protect distinct exclusion reasons.
tests/test_agent_copilot.py:71 — TestClassifyCopilotModel.test_advertised_routes_only_exclude Applies advertised API metadata as exclusions Unit keep Distinct route-metadata contract.
tests/test_agent_copilot.py:81 — TestBuildCopilotCatalog.test_orders_claude_by_family_newest_first_then_gpt, test_uses_the_listing_routes_and_drops_duplicates Orders, filters and deduplicates registry entries Unit keep Lowest reliable layer for deterministic catalog behavior.
tests/test_agent_copilot.py:117 — TestCatalogFromListings.test_prefers_the_model_services_listing, test_falls_back_to_the_gateway_listings, test_no_answer_keeps_the_saved_catalog, test_an_answer_with_nothing_usable_is_an_empty_catalog Distinguishes authoritative discovery, fallback, failure and empty usable results Unit keep These are materially different discovery outcomes.
tests/test_agent_copilot.py:136 — TestCopilotCatalog.test_managed_list_wins_and_skips_what_copilot_cannot_use, test_an_unusable_managed_list_falls_back_to_discovery, test_saved_catalog, test_an_empty_saved_catalog_does_not_revive_older_lists, test_state_from_before_the_catalog_uses_the_claude_and_gpt_lists Handles managed precedence, persistence and legacy-state transitions Unit keep Explicit-empty and absent-catalog states require separate assertions.
tests/test_agent_copilot.py:191 — TestResolveCopilotModel.test_accepts_ids_short_names_and_refs, test_accepts_the_databricks_prefix, test_prefers_the_system_ai_id, test_unknown_model_lists_what_is_available Resolves accepted spellings and reports unsupported choices Unit keep Alias parameters and precedence/error cases fit this layer.
tests/test_agent_copilot.py:216 — TestLaunchModel.test_requested_model_wins, test_managed_default_then_list, test_defaults_to_sonnet_then_opus_then_gpt, test_an_unavailable_requested_model_warns_and_falls_through, test_default_model_is_quiet Applies launch/default precedence and fallback diagnostics Unit keep Distinguishes explicit selection, managed selection and automatic defaults.
tests/test_agent_copilot.py:255 — TestRewriteModelArgs.test_rewrites_to_the_registry_ref Rewrites separate, equals, short and repeated model flags Unit keep Argument spelling belongs in a local matrix.
tests/test_agent_copilot.py:262 — TestRewriteModelArgs.test_leaves_arguments_after_the_separator; :266 — test_unknown_model_raises Preserves the separator boundary and validates selected models Unit keep Protects two externally meaningful argument contracts.
tests/test_agent_copilot.py:283 — TestCopilotCatalogModelId.test_maps_claude_ids, test_other_models_have_no_catalog_id Supplies native Claude settings IDs only where applicable Unit keep Covers mapping branches and meaningful negative inputs.
tests/test_agent_copilot.py:304 — TestCopilotModelLimits.test_claude_limits, test_gpt_keeps_copilots_defaults Supplies Claude limits and preserves native GPT defaults Unit keep Parameters exercise distinct limit-table and fallback behavior.
tests/test_agent_copilot.py:312 — TestRenderProvidersConfig.test_one_provider_per_wire_in_use_on_the_proxy, test_completions_provider, test_models_carry_claude_settings_and_limits Builds provider URLs, credentials, model IDs and limits Unit keep Exact registry shape is a valuable local contract.
tests/test_agent_copilot.py:355 — TestBuildRuntimeEnv.test_points_copilot_at_the_registry, test_drops_the_users_own_provider_settings Sets registry/offline selection while excluding inherited provider overrides Unit keep Provider parameters cover ownership boundaries without live launch matrices.
tests/test_agent_copilot.py:380 — TestSessionRegistry.test_writes_a_private_registry_removed_at_exit, test_removed_when_the_session_fails, test_sweep_keeps_live_sessions_and_removes_dead_ones Protects file permissions, cleanup and active/dead/user-file separation Filesystem component keep Real files and locks are appropriate; coverage is local rather than a separate process-held-lock journey.
tests/test_agent_copilot.py:536 — TestWriteToolConfig.test_records_the_default_model_and_drops_stale_provider_settings, test_scrubs_a_token_an_earlier_version_wrote, test_does_not_back_up_its_own_file_on_reconfigure Preserves configuration ownership through migration and repeated setup Filesystem component keep Distinct persistence and migration regressions.
tests/test_agent_copilot.py:580 — TestScrubStoredCredentials.test_drops_a_token_bearing_backup_so_revert_cannot_restore_it, test_keeps_a_backup_without_credentials, test_leaves_a_credential_free_env_file_untouched Prevents credential restoration while preserving unrelated files Filesystem component keep Negative preservation assertions complement the scrubbing case.
tests/test_agent_copilot.py:611 — TestProxyHeaders.test_identifies_ucode_and_copilot, test_adds_request_tags, test_managed_headers_win_by_name_but_never_replace_authorization Applies attribution and managed-header ownership Unit keep Header composition is distinct from HTTP forwarding behavior.
tests/test_agent_copilot.py:645 — TestTokenProvider.test_uses_the_cli_profile, test_uses_custom_oauth_when_configured Selects the configured principal and refresh behavior Unit keep Focused adapter checks complement existing shared authentication tests.
tests/test_agent_copilot.py:682 — _ExitedProcess.__init__, _ExitedProcess.wait, _stub_launch Observes child launch and deterministic lifecycle Component support consolidate Retain lifecycle observations; wrong-secret requests in general launch setup duplicate the dedicated proxy suite.
tests/test_agent_copilot.py:725 — TestLaunch.test_copilot_gets_the_registry_and_never_the_token Observes registry credentials at launch Component keep Protects explicit token injection; does not establish isolation from credentials already inherited by the parent.
tests/test_agent_copilot.py:745 — TestLaunch.test_explicit_model_is_rewritten_to_the_registry_ref, test_starts_on_the_requested_launch_model Wires explicit and managed/budget model selection into launch Component keep Separate entry paths justify both checks.
tests/test_agent_copilot.py:768 — TestLaunch.test_unknown_model_fails_before_starting_copilot, test_no_model_is_a_launch_error Prevents launch side effects for invalid or empty selections Component keep Different failure triggers protect the pre-launch boundary.
tests/test_agent_copilot.py:778 — TestLaunch.test_warns_about_managed_models_copilot_cannot_use Surfaces skipped managed entries Component keep Establishes launch-level warning wiring beyond classifier results.
tests/test_agent_copilot.py:788 — TestLaunch.test_launch_persists_no_state_and_no_token Keeps transient launch state out of persisted configuration Component keep Distinct state-ownership assertion.
tests/test_agent_copilot.py:800 — TestLaunch.test_ctrl_c_keeps_waiting_for_copilot Preserves proxy lifetime through interrupts until child exit Component keep Deterministic lifecycle fault belongs below live tests.
tests/test_agent_copilot.py:809 — TestLaunch.test_proxy_and_registry_stop_when_copilot_cannot_start Cleans listener and registry after spawn failure Socket/filesystem component keep Exercises a material failure path with observable cleanup.
tests/test_agent_copilot.py:838 — TestManagedKeys.test_only_the_model_and_offline_mode Pins the exact managed ownership set Unit keep Strong exact-set assertion.
tests/test_agent_copilot.py:841 — TestManagedKeys.test_excludes_credentials Excludes credentials from managed ownership Unit remove Fully implied by the preceding exact-set assertion.
tests/test_agent_copilot.py:856 — TestMinimumVersion.test_requires_1_0_91 Rejects old versions and handles supported/unknown versions Unit keep Protects rollout behavior without installed-version matrices.
tests/test_agent_copilot.py — removed TestRenderEnvOverlay.test_sets_provider_base_url, test_claude_uses_anthropic_provider_with_catalog_model_id, test_sets_provider_type, test_sets_model, test_sets_offline_true Former single-provider configuration Unit consolidate Surviving contracts move to registry-rendering and runtime-environment tests.
tests/test_agent_copilot.py — removed TestRenderEnvOverlay.test_sets_bearer_token; TestBuildRuntimeEnv.test_sets_oauth_token_for_mcp Former explicit raw-token injection Unit remove The proxy/registry design intentionally removes these injections.
tests/test_agent_copilot.py — removed TestRenderEnvOverlay.test_claude_wire_model_override_selects_anthropic_provider, test_selects_wire_api_from_override_and_keeps_selected_model; TestBuildRuntimeEnv.test_inherited_wire_model_overrides_route_without_being_cleared Former inherited wire-model override behavior Unit remove Inherited provider overrides are deliberately excluded; dotted-version compatibility remains a separate question.
tests/test_agent_copilot.py — removed TestBuildRuntimeEnv.test_inherits_path, test_overrides_copilot_vars Environment inheritance and UG override precedence Unit consolidate New registry-environment assertions replace these contracts.
tests/test_agent_copilot.py — removed TestDefaultModel.test_prefers_claude_sonnet, test_falls_back_to_opus, test_falls_back_to_haiku, test_falls_back_to_codex_when_no_claude, test_returns_none_when_no_models, test_ignores_gemini_models Default selection and fallback behavior Unit consolidate Catalog and launch tests replace most assertions; the standalone Haiku-only assertion is no longer explicit.
tests/test_agent_copilot.py — removed TestCopilotCatalogModelId.test_non_system_claude_ids_return_none Former system-only model mapping Unit consolidate Bare and Databricks-prefixed Claude IDs are intentionally accepted now; non-Claude negatives remain.
tests/test_agent_copilot.py — removed TestModelUsesResponsesApi.test_numeric_gpt_majors_and_gateway_aliases_use_responses, test_pre_gpt5_non_gpt_and_unsupported_alias_shapes_use_completions Former boolean wire classification Unit consolidate The classifier replaces this API; dotted-version inputs were lost and require the support clarification above.
tests/test_agent_copilot.py — removed TestWriteToolConfig.test_manages_wire_api_and_preserves_wire_model_and_user_values, test_claude_drops_stale_wire_api_from_env_file Stale-provider cleanup and user-value preservation Filesystem component consolidate Replacement configuration tests retain cleanup and unrelated-value assertions.
tests/test_agent_copilot.py — removed TestLaunch.test_launch_model_precedence_preserves_argv Former explicit, pinned and default launch precedence Component consolidate New launch-model, argument-rewrite and precedence tests cover the replacement flow.
tests/test_agent_copilot.py — removed TestLaunch.test_token_refresh_keeps_launch_model Former periodic token-file refresh Component move layer Refresh now belongs to cache, proxy-retry and real authentication journeys; repeated token-file writes are obsolete.
tests/test_agent_copilot.py — removed TestManagedKeys.test_includes_required_vars Former managed environment ownership Unit remove Replaced by exact ownership equality under the new registry design.
tests/test_agent_copilot.py — removed TestManagedModels.test_managed_models_win_over_the_shared_discovery_lists, test_falls_back_to_the_shared_lists_without_a_managed_config, test_copilot_default_model_wins_over_allowlist Managed/default/discovery precedence Unit consolidate Static-catalog, legacy-fallback, launch-default and managed-state translation cases cover the supported replacement paths.
tests/test_agents_init.py:289 — TestCheckGatewayEndpoint.test_copilot_available_with_claude, test_copilot_available_with_codex Determines agent availability from compatible models Unit keep Realistic model IDs are necessary after compatibility validation.
tests/test_agents_init.py:1091 — test_fable_does_not_displace_existing_default Preserves existing defaults for Copilot and Pi Unit keep Both parameters retain the existing regression assertion.
tests/test_cli.py:3438 — TestOtelProxyCommand.test_reuses_cli_proxy_and_auth_for_the_configured_workspace Preserves CLI telemetry authentication CLI component keep Patch-target update follows shared token extraction.
tests/test_cli.py:3477 — TestCopilotLaunchModel.test_managed_default_is_the_launch_model, test_a_managed_default_still_discovers_the_catalog Passes managed selection without suppressing discovery CLI component keep Protects orchestration not established by isolated agent tests.
tests/test_cli.py:5701 — TestConfigureSharedStateCopilotCatalog._configure, test_saves_the_classified_catalog, test_failed_listing_keeps_the_saved_catalog Persists classified discovery and retains cached results on failure CLI component keep Tests shared-state wiring and failure recovery.
tests/test_cli.py:6573 — TestChildStdoutLaunch.test_copilot_prompt_mode_owns_stdout Identifies headless stdout ownership Unit consolidate Extend the existing actual output-routing scenario rather than relying only on predicate parameters.
tests/test_codex_smart_routing_v2.py:92 — TestLaunchCodex.test_codex_launch_bypasses_routing_for_other_shapes Preserves routing bypass behavior Component keep Only the authentication patch target changes; existing shape assertions remain meaningful.
tests/test_databricks.py:710 — TestDiscoverModelServices.test_entries_keep_the_advertised_api_types Retains typed route metadata through discovery/cache Unit keep Covers the new entry contract; existing pagination/filtering coverage remains applicable.
tests/test_databricks.py:730 — TestDiscoverModelServices.test_failed_listing_is_not_cached Allows retry following transient discovery failure Unit keep Distinct cache failure behavior.
tests/test_doctor.py:147 — TestCopilotModelsCheck.test_skipped_when_copilot_is_not_configured, test_counts_models_per_provider, test_warns_when_nothing_is_usable Reports applicability, provider counts and empty catalogs Unit keep Appropriate diagnostic cases; they do not cover the managed-overlay discrepancy noted above.
tests/test_e2e.py:1086 — TestCopilotLaunch.test_launch_copilot_per_model Exercises real model/provider compatibility Existing live component keep Broader model smoke coverage; results depend on the filtered catalog, and an empty loop alone establishes no compatibility.
tests/test_gateway_proxy_integration.py:49 — _FakeGateway.__init__, _FakeGateway.start, nested Handler._serve Supplies scripted upstream response sequences Hermetic socket support keep Extends the existing fake server rather than adding another framework.
tests/test_gateway_proxy_integration.py:276 — _running_gateway_auth_proxy, _SESSION_SECRET Starts and tears down the real proxy with explicit fault inputs Hermetic socket support keep Appropriate reusable mechanics for HTTP-boundary tests.
tests/test_gateway_proxy_integration.py:307 — TestGatewayAuthProxyEndToEnd.test_swaps_session_secret_for_fresh_token_and_adds_headers Replaces authentication and forwards headers, body and path Hermetic socket keep Exercises actual transport rather than dictionary construction alone.
tests/test_gateway_proxy_integration.py:332 — TestGatewayAuthProxyEndToEnd.test_accepts_secret_in_x_api_key_and_does_not_forward_it Accepts and strips Anthropic-style authentication Hermetic socket keep Distinct credential-header path.
tests/test_gateway_proxy_integration.py:348 — TestGatewayAuthProxyEndToEnd.test_rejects_missing_or_wrong_secret_without_forwarding Rejects missing, wrong and BASIC credentials before upstream access Hermetic socket keep Parameters protect the authentication boundary.
tests/test_gateway_proxy_integration.py:371 — TestGatewayAuthProxyEndToEnd.test_refuses_paths_outside_model_routes Rejects workspace APIs, traversal, nested encoding and telemetry paths Hermetic socket keep Security matrix belongs at this local HTTP boundary.
tests/test_gateway_proxy_integration.py:383 — TestGatewayAuthProxyEndToEnd.test_expired_token_is_refreshed_and_request_retried Retries upstream authentication failure using a refreshed principal Hermetic socket keep Complements existing cache tests; does not itself simulate natural expiry.
tests/test_gateway_proxy_integration.py:418 — TestGatewayAuthProxyEndToEnd.test_failed_refresh_returns_actionable_401 Handles both error dialects without retrying a failed mint or leaking diagnostics Hermetic socket keep Parameters exercise materially different response shapes.
tests/test_integration_contract.py:272 — SKIPPED_ON_FORK_PRS, consumed by test_integration_jobs_that_run_on_fork_pull_requests_need_no_credentials Keeps credentialed Copilot jobs excluded from fork runs Static contract keep Extends the existing workflow-security contract.
tests/test_integration_contract.py:579 — test_live_integration_cases_belong_to_exactly_one_ci_agent Assigns every live case exactly one CI owner Static contract keep Necessary inventory extension with preserved assertion strength.
tests/test_integration_runner.py:34 — test_copilot_headless_target_and_exact_version Registers the exact smoke target and pinned CLI version Unit keep Protects runner configuration without installing the CLI.
tests/test_integration_runner.py:48 — test_copilot_version_output_matches_the_pinned_version Normalizes Copilot update hints without changing Codex output Unit keep Distinct CLI-version compatibility boundary.
tests/test_managed_resolve.py:687 — TestManagedUnservableModels._copilot, test_copilot_list_with_only_gemini_and_oss_is_unservable, test_copilot_list_with_a_claude_model_is_servable, test_copilot_model_list_reaches_state_as_its_static_models Handles unusable/mixed lists and managed-to-static-state translation Unit keep Protects warning policy and ownership translation.
tests/test_managed_setup.py:335 — TestModelOptions.test_multi_provider_agents_see_everything Preserves broader OpenCode/Pi choices Unit keep Removing Copilot from this assertion matches its narrower compatibility policy.
tests/test_managed_setup.py:343 — TestModelOptions.test_copilot_sees_only_claude_and_gpt Limits admin-facing Copilot choices to supported families Unit keep Separate configuration contract from runtime classification.

Existing cache coverage in tests/test_gateway_proxy.py already addresses near-expiry forcing, single-flight refresh and refresher survival; another credentialed expiry matrix is unnecessary. Existing discovery pagination/filtering and Codex authentication tests also remain relevant.

The concrete coverage gaps are direct tests of the new Copilot evidence parser and actual clean-stdout routing. The managed doctor discrepancy needs a focused managed-overlay regression case when corrected. None of these missing tests alone establishes an additional correctness bug.

For validation, use a disposable head checkout without credentials first:
UV_OFFLINE=1 uv run pytest tests/test_agent_copilot.py tests/test_cli.py tests/test_doctor.py tests/test_gateway_proxy_integration.py.

In a separately authorized live environment, configure once, complete a Claude task, switch to GPT with /model in the same session, complete another task, exit normally, then revert twice. Confirm user configuration survives and session registries are removed. No such execution is claimed here.

Scope

No clearly unrelated change was established. Discovery, proxy authentication, registry lifecycle, managed selection, diagnostics, migration, documentation and CI support the stated Copilot workflow. Codex test adjustments follow the shared authentication extraction. The truncated description limits conclusions about omitted context; it does not justify treating inherited stack changes as unrelated.


Automated review by Polly · workflow run

@xi-db
xi-db force-pushed the xi-db/stack/copilot-model-discovery branch from 5b488aa to 253b9c8 Compare October 21, 2026 12:44
@xi-db
xi-db force-pushed the xi-db/stack/copilot-model-discovery branch from 060eea1 to 89e47e1 Compare October 21, 2026 16:04
zhulinchng pushed a commit to zhulinchng/ucode that referenced this pull request Oct 21, 2026
…atabricks#1217)

## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/unity-gateway/pull/1217/files) to
review incremental changes.
-
[**stack/copilot-auth-proxy**](databricks#1217)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1217/files)] ←
_this PR_
-
[stack/copilot-model-discovery](databricks#1218)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1218/files/629a83492996f5e7e641695f2f9d54b01d77bfbb..060eea1216d8ab1235712ec0748b427a8ed699c6)]
-
[stack/copilot-provider-services](databricks#1220)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1220/files/060eea1216d8ab1235712ec0748b427a8ed699c6..44f4efa48fd129d5f774649f927bceb5d751c0c2)]

---------
## What did you change, and why?

**Change:** `ug copilot` now runs Copilot CLI against a per-session
loopback proxy (`gateway_proxy.start_gateway_auth_proxy`) instead of
handing it a Databricks token.

- Copilot gets
`COPILOT_PROVIDER_BASE_URL=http://127.0.0.1:<port>/ai-gateway/...` and a
random per-session secret as `COPILOT_PROVIDER_BEARER_TOKEN`.
- The proxy accepts only that secret (Bearer or `x-api-key`) and only on
`/ai-gateway/{anthropic,mlflow/v1,openai/v1}/` (decoded until stable;
`.`/`..` segments rejected).
- It swaps the secret for a fresh Databricks token from the CLI profile
or custom OAuth, using `TokenCache` (refresh near expiry, force refresh
plus one retry on 401/403), and adds `ucode/<v> copilot/<v>` User-Agent,
request tags, and managed `copilot_http_headers`.
- If refresh fails, Copilot gets a dialect-shaped 401 ("Databricks
sign-in expired. Run `databricks auth login --host <ws>`, then restart
`ug copilot`."). The error goes to `~/.ucode/logs/copilot-proxy.log`,
not over the TUI.
- Configure and launch persist no credential. Tokens written by earlier
versions are scrubbed from `~/.copilot/ucode.env` and from a UG-produced
backup, including before `ug revert`, so revert cannot restore one.
- Removes the Copilot token-refresher thread and its SIGINT forwarding;
the parent keeps waiting on Ctrl-C so the proxy outlives Copilot.
- Drops inherited `COPILOT_PROVIDER_API_KEY` / `_API_KEY_COMMAND` /
`_TRANSPORT` so they can't bypass the proxy.
- Adds a non-blocking `copilot` integration lane (Linux + Windows, `live
and copilot`, outside `cujs`) mirroring OpenCode's, and amends the
`tests/AGENTS.md` scope line to allow Copilot headless tests.

**Why:** Copilot reads its provider credential once at startup.

- **Expiry:** the old refresher rewrote `ucode.env`, which Copilot never
re-reads, so sessions failed once the OAuth token expired (~1h).
- **Plaintext tokens:** the token sat in plaintext in `ucode.env`, its
backup, and the child env.
- **Shared-state writes:** the refresher also called `save_state` from a
background thread, which could reset the active workspace during a
concurrent `ug configure`.

BEFORE: a `ug copilot` session stops working after token expiry until
restarted; `~/.copilot/ucode.env` and
`~/.ucode/copilot-ucode-env.backup` contain the Databricks token; Ctrl-C
on Windows raises `ValueError` from `send_signal(SIGINT)`.

AFTER: requests carry a freshly minted token for the life of the
session; no file or Copilot env var holds a Databricks credential;
Ctrl-C is delivered to Copilot by the terminal and UG tears the proxy
down after Copilot exits.

## How do you know it works?

**Testing:**
-
`tests/test_gateway_proxy_integration.py::TestGatewayAuthProxyEndToEnd`
(real sockets):
  - secret swap and added headers;
  - `x-api-key` accepted and not forwarded;
  - a wrong or missing secret gets a 401 and nothing is forwarded;
  - path refusals (`../`, `%2e%2e`, `%252e%252e`, otel);
  - a 401 from the gateway triggers a forced refresh and one retry;
- the actionable 401 is shaped for each dialect, with nothing on stderr.
- `tests/test_agent_copilot.py`:
  - no credential in the env overlay or on disk;
  - scrubbing legacy tokens and UG-produced backups;
  - token provider for both the CLI profile and custom OAuth;
  - header precedence;
  - launch through the real proxy with a stubbed child process.
- `tests/test_e2e.py::TestCopilotLaunch` now runs through the real proxy
(and patches `ucode.agents.copilot.save_state`, which the old patch
target missed).
- New `tests/integration/test_ug_copilot_headless.py`: a Claude
(Anthropic) task and a GPT (Responses) task, each asserting no
`DATABRICKS_BEARER` on disk under `~/.copilot` / `~/.ucode`. **Not run
yet**: needs the e2e workspace and Copilot 1.0.91. This draft is so the
lane can run in CI.
- Locally on Linux: 807 passed across the touched suites; `just lint`
(ruff check/format, ty) passes. The remaining unit failures in
`TestInstallToolBinary` / `test_e2e_user_agent` also fail on a clean
`main` checkout here.

**Open questions for reviewers:**
- `tests/AGENTS.md` scope change needs CODEOWNERS sign-off.
- Revocation returns an actionable 401 rather than retrying with a
browser login, because UG's login path writes to the terminal Copilot
owns. A browser-login retry is a follow-up.
- A user `~/.copilot/providers.json` can still override UG's provider;
guard left for a follow-up.
- Still to check live: how Copilot renders the proxy's 401 body; Windows
firewall behavior on the loopback bind.

Not in this PR: binary resolver (databricks#970), version floor / wizard /
`forceLoginOrgs` doctor, `--workspace`, `-p` stdout banner, MPS, OTEL,
OSS models.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Rohit Agrawal <10210921+rohita5l@users.noreply.github.com>

@masonc08 masonc08 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stamping as theres no breaking changes. would be interested in discussing followups below

Comment thread src/ucode/managed_setup.py
Comment thread src/ucode/agents/copilot.py Outdated
Comment thread src/ucode/agents/copilot.py
Comment thread src/ucode/agents/copilot.py
@xi-db
xi-db force-pushed the xi-db/stack/copilot-model-discovery branch from c5f22a4 to 4dcc345 Compare October 21, 2026 21:29
@xi-db xi-db added ug-review Run the automated UG review and removed ug-review Run the automated UG review labels Oct 21, 2026
@xi-db
xi-db enabled auto-merge October 21, 2026 21:46
Comment thread src/ucode/agents/copilot.py
Comment thread src/ucode/agents/copilot.py Outdated
Comment thread src/ucode/agents/copilot.py
Comment thread src/ucode/cli.py
Comment thread src/ucode/cli.py Outdated
Comment thread src/ucode/cli.py Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

  1. [P1] Partial discovery can make Copilot unavailable — src/ucode/agents/copilot.py:262

    can we retain missing-family gateway fallbacks? a Gemini-only UC result discards successfully discovered Claude/GPT models, persists an empty catalog, and prevents launch.

  2. [P2] Previously supported GPT-4o models are rejected — src/ucode/agents/copilot.py:114

    can we classify gpt-4o and gpt-4o-mini as Completions? the numeric regex now excludes both; the previous implementation and a deleted regression case supported them.

  3. [P2] Concurrent cleanup can fail on Windows — src/ucode/agents/copilot.py:536

    can finalization hold the registry mutex? a concurrent sweep can open the released lock before unlink, causing Windows PermissionError after an otherwise successful session.

Security vulnerabilities

No confirmed security vulnerability found. Registry creation, credential separation, environment isolation, and cleanup were inspected statically; Windows permissions were not runtime-verified.

Non-blocking notes

  • Managed-default compatibility — src/ucode/agents/copilot.py:367: should a valid managed default outside model_services be rejected during configuration or included? it previously won; launch now warns and substitutes another model.
  • Repeated live tasks — tests/integration/test_ug_copilot_headless.py:16: can both headless cases fold into test_ug_copilot_one_setup_runs_claude_then_gpt_by_provider_ref, retaining credential, identity, and cleanup assertions?
  • Rejection-test layer — tests/integration/test_ug_copilot_models.py:78: can test_ug_copilot_unknown_model_lists_available_models become a CLI component test? controlled discovery can prove rejection without live authentication.
  • Lost assertions — tests/test_agent_copilot.py:235: can the default matrix restore Haiku-only selection, and the writer test restore the assertion that configuring never requests a token?
  • Completion evidence — tests/test_e2e.py:1086: can test_launch_copilot_per_model require an answering assistant message? exit zero plus nonempty output can pass without successful inference.

Approach

The per-session registry and existing proxy are a sound fit. Reuse the shared discovery path’s per-family fallback, preserve UC exclusions for matching IDs, and use the existing registry mutex for finalization.

Summary

Diff size (computed): 28 files; +1667 / -482 lines (2149 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 18 996 328 61.6%
Documentation 1 16 9 1.2%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 9 655 145 37.2%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

The change replaces single-provider Copilot configuration with a session registry, adds authorized-model discovery and selection, carries managed defaults into launch, and exposes the catalog through status and doctor.

The diff includes implementation, replacement unit coverage, live journeys, evidence helpers, runner support, and documentation. There are no dependency or generated-file changes. Most of the size follows from replacing the previous configuration contract and covering its lifecycle. No separate implementation-only refactor clearly needs removal.

Tests

Test-by-test assessment

This is a static assessment, not a passing test report. The snapshot checksum and both trusted policy files were verified. No PR code or tests were executed, and no outbound requests were made. Native Copilot behavior and the Windows cleanup interleaving remain runtime-unverified. The trusted checkout lacks tests/_helpers/README.md; existing test guides and neighboring helpers were inspected instead.

Tooling limitation: the OpenCode, Cursor, Hermes, Pi, and Antigravity CLIs were unavailable.

“Removed” identifies deleted coverage. Grouped rows name tests that protect the same contract or share the same disposition.

Test / case Behavior protected Layer Needed? Action / rationale
scripts/run_integration.py:738 — Windows PATH setup; no test case Copilot discovers the installed PowerShell runtime Runner support keep Supports the native Windows process boundary; setup alone is not execution evidence.
tests/AGENTS.md:96 — documentation only Integration scope includes model switching Test documentation keep Documents the new journey; changed instructions were not used as review policy.
tests/README.md:464 — coverage documentation Discoverability of headless and model-switch coverage Documentation keep Coverage inventory belongs beside the suite.
tests/integration/README.md:238 — coverage and helper documentation Reproducible integration selection and evidence boundaries Documentation keep Explains how to exercise the new native-client journeys.
tests/integration/test_ug_copilot_headless.py:16 — test_ug_copilot_headless_completes_file_task Claude task completion, model identity, credential exclusion, cleanup Live CLI consolidate Transfer these assertions into the one-setup Claude/GPT journey.
tests/integration/test_ug_copilot_headless.py:55 — test_ug_copilot_headless_gpt_responses_task GPT task completion and Responses selection Live CLI consolidate Duplicates the GPT task boundary in the one-setup journey; retain credential assertions there.
tests/integration/test_ug_copilot_lifecycle.py:180 — test_ug_configure_copilot_repeat_and_revert Repeated configure/revert preserves user configuration Live lifecycle keep Protects a distinct stateful ownership journey.
tests/integration/test_ug_copilot_lifecycle.py:227 — test_ug_copilot_upgrade_scrubs_token_saved_by_older_ug Upgrade removes previously persisted credentials Live upgrade keep The older-installation boundary differs from fresh configuration.
tests/integration/test_ug_copilot_models.py:34 — test_ug_copilot_one_setup_runs_claude_then_gpt_by_provider_ref One configuration exposes both provider families Live CLI consolidate Use as the combined headless journey, retaining all identity, credential, and cleanup assertions.
tests/integration/test_ug_copilot_models.py:78 — test_ug_copilot_unknown_model_lists_available_models Unknown selection lists alternatives without starting Copilot Live CLI move layer A CLI component test with controlled discovery can establish this rejection contract.
tests/integration/test_ug_copilot_models.py:101 — test_ug_copilot_tui_switches_from_claude_to_gpt_with_model_command Native in-session switching across protocols Live TUI keep Headless relaunches cannot establish interactive /model behavior.
tests/integration/utils/agents/copilot.py:1 — _messages, assistant_answers, completed_task_models, selected_models, is_child_session Native transcript interpretation Evidence support keep Reuses the evidence adapter boundary and has focused parser coverage.
tests/integration/utils/evidence.py:150 — copilot_selected_models, assert_no_session_registry_left, FileTask.__init__, prompt_with_path Shared model evidence, cleanup checks, explicit task paths Test support keep Extends existing task/evidence primitives rather than introducing parallel setup.
tests/integration/utils/terminal.py:232 — AgentTerminal.__exit__, AgentTerminal.boot Transcript capture, visible trust handling, prompt readiness PTY support keep Required for the distinct interactive process boundary.
tests/test_agent_copilot.py:56 — TestClassifyCopilotModel.test_names_decide_the_wire Model names choose the correct wire protocol Unit keep Add GPT-4o and GPT-4o-mini regression inputs for blocking issue 2.
tests/test_agent_copilot.py:70 — TestClassifyCopilotModel.test_excluded_models_say_why Unsupported models receive actionable exclusion reasons Unit keep Tests policy without making unsupported live requests.
tests/test_agent_copilot.py:74 — TestClassifyCopilotModel.test_advertised_routes_only_exclude API metadata excludes incompatible routes Unit keep Protects metadata policy independently of name classification.
tests/test_agent_copilot.py:84 — TestBuildCopilotCatalog.test_orders_claude_by_family_newest_first_then_gpt Stable catalog and default ordering Unit keep Ordering affects both model presentation and launch selection.
tests/test_agent_copilot.py:106 — TestBuildCopilotCatalog.test_uses_the_listing_routes_and_drops_duplicates Metadata filtering and ID deduplication Unit keep Exercises catalog construction beyond individual classification.
tests/test_agent_copilot.py:120 — TestCatalogFromListings.test_prefers_the_model_services_listing Discovery-source precedence Unit keep Correct the expectation for blocking issue 1: retain missing-family fallbacks without reviving same-ID UC exclusions.
tests/test_agent_copilot.py:124 — TestCatalogFromListings.test_falls_back_to_the_gateway_listings Gateway-only discovery remains usable Unit keep Covers installations without a UC answer; partial answers also need coverage for issue 1.
tests/test_agent_copilot.py:129 — TestCatalogFromListings.test_no_answer_keeps_the_saved_catalog Failed discovery preserves saved inventory Unit keep Protects transient-failure recovery.
tests/test_agent_copilot.py:132 — TestCatalogFromListings.test_an_answer_with_nothing_usable_is_an_empty_catalog Unsupported-only results differ from failed discovery Unit keep Protects the empty-result policy, but does not cover successful fallback alongside partial UC results.
tests/test_agent_copilot.py:139 — TestCopilotCatalog.test_managed_list_wins_and_skips_what_copilot_cannot_use Managed precedence and partial filtering Unit keep Exercises a partially usable managed list.
tests/test_agent_copilot.py:153 — TestCopilotCatalog.test_an_unusable_managed_list_falls_back_to_discovery Entirely unsupported managed lists degrade safely Unit keep Covers the whole-list fallback separately from partial filtering.
tests/test_agent_copilot.py:162 — TestCopilotCatalog.test_saved_catalog Catalog persistence shape Unit keep Checks IDs and timestamps through serialization/readback.
tests/test_agent_copilot.py:169 — TestCopilotCatalog.test_an_empty_saved_catalog_does_not_revive_older_lists Empty state suppresses obsolete lists Unit keep Distinguishes an explicitly empty catalog from an absent catalog.
tests/test_agent_copilot.py:173 — TestCopilotCatalog.test_state_from_before_the_catalog_uses_the_claude_and_gpt_lists Older saved state remains usable Unit keep Direct upgrade-compatibility boundary.
tests/test_agent_copilot.py:194 — TestResolveCopilotModel.test_accepts_ids_short_names_and_refs, TestResolveCopilotModel.test_accepts_the_databricks_prefix Accepted aliases normalize to registry references Unit keep Covers different input/catalog shapes at the resolver boundary.
tests/test_agent_copilot.py:202 — TestResolveCopilotModel.test_prefers_the_system_ai_id Ambiguous aliases resolve deterministically Unit keep Tie-breaking is separate from ordinary alias acceptance.
tests/test_agent_copilot.py:206 — TestResolveCopilotModel.test_unknown_model_lists_what_is_available Unknown selection reports reasons and alternatives Unit keep Asserts meaningful error contents.
tests/test_agent_copilot.py:219 — TestLaunchModel.test_requested_model_wins Requested launch model outranks managed defaults Unit keep Direct precedence invariant.
tests/test_agent_copilot.py:223 — TestLaunchModel.test_managed_default_then_list Managed defaults and lists supply initial selection Unit keep Protects administrator-driven startup.
tests/test_agent_copilot.py:235 — TestLaunchModel.test_defaults_to_sonnet_then_opus_then_gpt Unmanaged family preference Unit keep Restore the removed Haiku-only input in this matrix.
tests/test_agent_copilot.py:238 — TestLaunchModel.test_an_unavailable_requested_model_warns_and_falls_through Unavailable recommended models degrade with a warning Unit keep Distinct from strict explicit-argument rejection.
tests/test_agent_copilot.py:244 — TestLaunchModel.test_default_model_is_quiet Inspection avoids launch warnings and handles no catalog Unit keep Protects quiet status/default queries.
tests/test_agent_copilot.py:258 — TestRewriteModelArgs.test_rewrites_to_the_registry_ref Argument forms become native provider references Unit keep Keeps the argument matrix below the live-client layer.
tests/test_agent_copilot.py:265 — TestRewriteModelArgs.test_leaves_arguments_after_the_separator -- preserves literal downstream arguments Unit keep Explicit parsing-boundary coverage.
tests/test_agent_copilot.py:269 — TestRewriteModelArgs.test_unknown_model_raises Resolver failure propagates through rewriting Unit keep Small boundary assertion; launch coverage separately proves no process starts.
tests/test_agent_copilot.py:286 — TestCopilotCatalogModelId.test_maps_claude_ids — modified matrix Claude IDs receive native request settings Unit keep Covers versions, suffixes, prefixes, and Haiku mapping under the broader contract.
tests/test_agent_copilot.py:289 — TestCopilotCatalogModelId.test_other_models_have_no_catalog_id GPT avoids Claude-specific settings Unit keep Cross-family negative boundary.
tests/test_agent_copilot.py:307 — TestCopilotModelLimits.test_claude_limits Explicit Claude limits and future-version fallback Unit keep Nontrivial policy belongs at this layer.
tests/test_agent_copilot.py:310 — TestCopilotModelLimits.test_gpt_keeps_copilots_defaults GPT does not inherit Claude limits Unit keep Distinct negative policy boundary.
tests/test_agent_copilot.py:315 — TestRenderProvidersConfig.test_one_provider_per_wire_in_use_on_the_proxy, TestRenderProvidersConfig.test_completions_provider Provider schema, wire routing, omission of unused providers Unit keep Covers the distinct native provider schemas without live inference.
tests/test_agent_copilot.py:340 — TestRenderProvidersConfig.test_models_carry_claude_settings_and_limits Helper results reach native registry fields Unit keep Helper tests alone cannot catch missing rendered fields.
tests/test_agent_copilot.py:358 — TestBuildRuntimeEnv.test_points_copilot_at_the_registry Registry path, initial reference, offline mode, PATH Unit keep Protects the replacement process-environment contract.
tests/test_agent_copilot.py:376 — TestBuildRuntimeEnv.test_drops_the_users_own_provider_settings Inherited BYOK settings cannot override the registry Unit keep Checks environment isolation with representative inherited keys.
tests/test_agent_copilot.py:383 — TestSessionRegistry.test_writes_a_private_registry_removed_at_exit Real registry contents, permissions, normal cleanup Filesystem component keep Uses actual file operations with platform-qualified permission assertions.
tests/test_agent_copilot.py:396 — TestSessionRegistry.test_removed_when_the_session_fails Exceptional session exit removes the registry Filesystem component keep Distinct failure-exit path.
tests/test_agent_copilot.py:404 — TestSessionRegistry.test_sweep_keeps_live_sessions_and_removes_dead_ones Sweeping preserves live sessions and user-owned files Locking component keep Extend with coordinated exit/sweep contention for blocking issue 3; current coverage misses that interleaving.
tests/test_agent_copilot.py:539 — TestWriteToolConfig.test_records_the_default_model_and_drops_stale_provider_settings Config migration and user-data preservation Filesystem component keep Restore the removed assertion that configuration never calls the token provider.
tests/test_agent_copilot.py:725 — TestLaunch.test_copilot_gets_the_registry_and_never_the_token Launch composes registry, proxy credentials, isolation, cleanup Launch/socket component keep Real proxy plus substituted child establishes composition, not native inference.
tests/test_agent_copilot.py:745 — TestLaunch.test_explicit_model_is_rewritten_to_the_registry_ref Launch invokes rewriting and explicit selection wins Component keep Protects adapter wiring beyond the pure resolver.
tests/test_agent_copilot.py:759 — TestLaunch.test_starts_on_the_requested_launch_model Shared launch options reach Copilot Component keep Covers the new launch-option transport boundary.
tests/test_agent_copilot.py:768 — TestLaunch.test_unknown_model_fails_before_starting_copilot Invalid selection starts neither process nor registry Component keep Establishes side-effect-free rejection.
tests/test_agent_copilot.py:778 — TestLaunch.test_warns_about_managed_models_copilot_cannot_use Launch emits managed-exclusion warnings Component keep Tests observable warning delivery.
tests/test_agent_copilot.py:788 — TestLaunch.test_launch_persists_no_state_and_no_token — modified Launch avoids persistent state/config writes Component keep Ownership invariant differs from temporary-file cleanup.
tests/test_agent_copilot.py:800 — TestLaunch.test_ctrl_c_keeps_waiting_for_copilot — modified Interrupt handling preserves waiting and exit status Component keep Retains existing signal-handling coverage with new setup.
tests/test_agent_copilot.py:809 — TestLaunch.test_proxy_and_registry_stop_when_copilot_cannot_start; replaces test_proxy_stops_when_copilot_cannot_start Spawn failure closes both resources Component/socket keep Extends the existing failure scenario with registry cleanup.
tests/test_agent_copilot.py:838 — TestManagedKeys.test_only_the_model_and_offline_mode Exact persistent ownership allowlist Unit keep Stronger than separate membership-only assertions.
tests/test_agent_copilot.py:852 — TestMinimumVersion.test_requires_1_0_91 Old, minimum, newer, and unknown version handling Unit keep Boundary matrix belongs in the version checker.
tests/test_agent_copilot.py:1 — removed TestRenderEnvOverlay.test_sets_provider_base_url, test_claude_uses_anthropic_provider_with_catalog_model_id, test_sets_provider_type, test_sets_model, test_carries_no_credential, test_sets_offline_true Former single-provider overlay contents Unit remove The renderer is gone; replacement registry, mapping, environment, and credential tests cover its surviving contracts.
tests/test_agent_copilot.py:1 — removed TestRenderEnvOverlay.test_claude_wire_model_override_selects_anthropic_provider, test_selects_wire_api_from_override_and_keeps_selected_model; TestBuildRuntimeEnv.test_inherited_wire_model_overrides_route_without_being_cleared Former ambient wire override precedence Unit remove Intentionally superseded by registry isolation; restoring these expectations would contradict the new contract.
tests/test_agent_copilot.py:1 — removed TestBuildRuntimeEnv.test_inherits_path, test_points_copilot_at_the_session_proxy, test_claude_goes_through_the_proxy_anthropic_route, test_sets_no_oauth_token, test_drops_inherited_settings_that_bypass_the_proxy Former provider environment, PATH, and credential handling Unit consolidate Surviving invariants move into registry environment/rendering and launch tests.
tests/test_agent_copilot.py:1 — removed TestDefaultModel.test_prefers_claude_sonnet, test_falls_back_to_opus, test_falls_back_to_haiku, test_falls_back_to_codex_when_no_claude, test_returns_none_when_no_models, test_ignores_gemini_models Default preference and absent-model handling Unit consolidate New catalog/selection tests replace old fixtures; restore the Haiku-only input noted above.
tests/test_agent_copilot.py:1 — removed TestCopilotCatalogModelId.test_non_system_claude_ids_return_none Former restriction to system Claude IDs Unit remove The restriction is deliberately broadened to support additional Claude ID forms.
tests/test_agent_copilot.py:1 — removed TestModelUsesResponsesApi.test_numeric_gpt_majors_and_gateway_aliases_use_responses, test_pre_gpt5_non_gpt_and_unsupported_alias_shapes_use_completions GPT routing and name boundaries Unit consolidate Classification replaces this API, but the deleted GPT-4o Completions input exposes blocking issue 2.
tests/test_agent_copilot.py:1 — removed TestWriteToolConfig.test_manages_wire_api_and_preserves_wire_model_and_user_values, test_claude_drops_stale_wire_api_from_env_file, test_needs_no_token_and_writes_none Persistent ownership, legacy cleanup, configure-time token avoidance Component consolidate Replacement writer coverage handles ownership/cleanup; restore the no-token-provider assertion there.
tests/test_agent_copilot.py:1 — removed TestLaunch.test_launch_model_precedence_preserves_argv Selection precedence and argument preservation Component consolidate Replacement selection, rewrite, CLI, and launch-option tests cover the new API.
tests/test_agent_copilot.py:1 — removed TestLaunch.test_copilot_gets_the_proxy_and_a_session_secret_not_the_token Child receives session credentials, not workspace credentials Launch/socket component consolidate Replaced by the registry-based launch credential test.
tests/test_agent_copilot.py:1 — removed TestManagedKeys.test_includes_required_vars, test_excludes_credentials Persistent key ownership Unit consolidate The replacement exact-list assertion covers both contracts.
tests/test_agent_copilot.py:1 — removed TestManagedModels.test_managed_models_win_over_the_shared_discovery_lists, test_falls_back_to_the_shared_lists_without_a_managed_config Managed and legacy discovery precedence Unit consolidate Re-expressed through static-model, catalog-precedence, and older-state tests.
tests/test_agent_copilot.py:1 — removed TestManagedModels.test_copilot_default_model_wins_over_allowlist Managed default outside the list previously wins Unit uncertain Confirm the compatibility contract in the non-blocking note before accepting this coverage loss.
tests/test_agents_init.py:290 — TestCheckGatewayEndpoint.test_copilot_available_with_claude, test_copilot_available_with_codex Availability recognizes supported model families Component keep Realistic IDs are required now that availability classifies models.
tests/test_agents_init.py:1092 — test_fable_does_not_displace_existing_default — modified matrix Adding Fable preserves the GPT default Unit keep Updated fixture preserves the existing assertion under stricter classification.
tests/test_cli.py:3477 — TestCopilotLaunchModel.test_managed_default_is_the_launch_model, test_a_managed_default_still_discovers_the_catalog Managed initial selection and discovery policy reach launch CLI component keep Protects two independently observable launch inputs.
tests/test_cli.py:5844 — TestConfigureSharedStateCopilotCatalog.test_saves_the_classified_catalog Shared configure persists filtered catalog state Component keep Add partial-UC/successful-fallback coverage for blocking issue 1 without mocking catalog construction.
tests/test_cli.py:5855 — TestConfigureSharedStateCopilotCatalog.test_failed_listing_keeps_the_saved_catalog Same-workspace failure retains inventory and warns Component keep Distinct stateful recovery path.
tests/test_cli.py:5864 — TestConfigureSharedStateCopilotCatalog.test_failed_listing_after_a_workspace_switch_drops_the_other_workspace_catalog, test_failed_listing_after_a_workspace_switch_keeps_that_workspace_catalog Workspace transitions use only destination inventory Component keep Both absent and existing destination state protect against cross-workspace reuse.
tests/test_cli.py:6606 — TestChildStdoutLaunch.test_status_goes_to_stderr_and_arguments_are_preserved — added Copilot inputs Headless diagnostics preserve child stdout CLI component keep Asserts observable stream ownership and forwarding.
tests/test_cli.py:6655 — TestChildStdoutLaunch.test_early_launch_failure_keeps_stdout_clean — added Copilot input Bootstrap failure also preserves stdout CLI component keep Failure precedes native launch and needs a separate assertion.
tests/test_cli.py:6692 — TestChildStdoutLaunch.test_other_launches_keep_status_on_stdout — added Copilot inputs Interactive/separator-delimited launches retain normal output CLI component keep Protects the negative parsing boundary.
tests/test_cli.py:6724 — TestChildStdoutLaunch.test_copilot_prompt_mode_owns_stdout Prompt-flag classification Unit keep Cheap predicate matrix complements observable CLI stream assertions.
tests/test_databricks.py:710 — TestDiscoverModelServices.test_entries_keep_the_advertised_api_types Typed metadata survives parsing and caching Component keep Covers the new metadata contract beyond existing ID-list tests.
tests/test_databricks.py:730 — TestDiscoverModelServices.test_failed_listing_is_not_cached Failed discovery remains retryable Component keep Adds cache-state coverage beyond HTTP retry behavior.
tests/test_doctor.py:147 — TestCopilotModelsCheck.test_skipped_when_copilot_is_not_configured Irrelevant diagnostics are omitted Unit keep Optional-check boundary.
tests/test_doctor.py:151 — TestCopilotModelsCheck.test_counts_models_per_provider Doctor reports effective provider counts Unit keep User-facing summary without a live client.
tests/test_doctor.py:163 — TestCopilotModelsCheck.test_counts_the_managed_list_launch_registers Diagnostics apply managed-list precedence Component keep Exercises resolution plus counting.
tests/test_doctor.py:183 — TestCopilotModelsCheck.test_warns_when_nothing_is_usable Empty catalog produces remediation Unit keep Distinct diagnostic failure branch.
tests/test_e2e.py:1086 — TestCopilotLaunch.test_launch_copilot_per_model — modified Native request compatibility across selected models Live agent/gateway keep Distinct model/request boundary; strengthen answer evidence as noted above. Legacy model selection is not full-catalog discovery coverage.
tests/test_integration_evidence.py:158 — test_copilot_model_identity_uses_only_answering_assistant_messages Echoes and non-answer messages cannot prove identity Unit keep Adversarial inputs protect evidence validity.
tests/test_integration_evidence.py:178 — test_copilot_model_identity_rejects_an_answer_without_a_model Missing identity fails explicitly Unit keep Malformed-evidence boundary.
tests/test_integration_evidence.py:184 — test_copilot_selected_models_follow_start_then_switches Native selection events preserve order Unit keep Protects the parser underlying in-session switching evidence.
tests/test_managed_resolve.py:690 — TestManagedUnservableModels.test_copilot_list_with_only_gemini_and_oss_is_unservable Entire unsupported managed lists trigger fallback Unit keep Tests policy at its owning layer.
tests/test_managed_resolve.py:694 — TestManagedUnservableModels.test_copilot_list_with_a_claude_model_is_servable Mixed lists are not wholly rejected Unit keep Negative boundary for whole-list fallback.
tests/test_managed_resolve.py:698 — TestManagedUnservableModels.test_copilot_model_list_reaches_state_as_its_static_models; _copilot helper Managed list reaches the adapter’s state key Component/support keep Covers the new translation; helper supplies explicit inputs only.
tests/test_managed_setup.py:335 — TestModelOptions.test_multi_provider_agents_see_everything — Copilot removed from matrix Other agents retain broader model options Unit keep Preserves unaffected-agent behavior while removing an obsolete Copilot expectation.
tests/test_managed_setup.py:343 — TestModelOptions.test_copilot_sees_only_claude_and_gpt Managed setup exposes compatible families Unit keep Protects the narrowed setup surface.

The demonstrated gaps are partial UC discovery, GPT-4o classification, and concurrent Windows cleanup. Existing proxy socket and unchanged authentication-recovery coverage suffice for token refresh; no additional live authentication journey is requested.

After fixes, run focused checks in a credential-isolated environment with dependencies already installed:

uv run --offline pytest tests/test_agent_copilot.py tests/test_cli.py tests/test_databricks.py tests/test_doctor.py tests/test_integration_evidence.py tests/test_managed_resolve.py tests/test_managed_setup.py

For manual verification, use Copilot 1.0.91 or newer in a disposable home. Configure once, complete a Claude file task, switch with /model dbx-responses/<authorized-gpt-id>, complete another task, and exit. On Windows, overlap one session’s exit with another launch; both should finish normally without leftover session registries.

Scope

The inspected changes support discovery, selection, session isolation, configuration compatibility, diagnostics, or verification of those behaviors. No clearly unrelated changes were found. Whether a managed default outside model_services should remain supported is an unresolved compatibility question, not evidence of independent scope.


Automated review by Polly · workflow run

@xi-db
xi-db added this pull request to the merge queue Oct 22, 2026
@xi-db
xi-db removed this pull request from the merge queue due to a manual request Oct 22, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
xi-db and others added 10 commits October 22, 2026 06:48
….json

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The e2e workspace lists claude-sonnet-4-5 but the CI principal gets 403 on it;
claude-haiku-4-5 is already older than its family's newest there.

Co-authored-by: Isaac <no-reply@databricks.com>
- A workspace switch no longer carries the source workspace's Copilot
  catalog when the destination's listing fails.
- ug doctor counts the managed model list that launch registers.
- Dotted model names keep their version punctuation when classified.
- Copilot evidence parsing and -p stdout routing get unit coverage.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…tion journeys

Copilot's shell tool tries pwsh first; with only Windows PowerShell on the test
PATH its 40s startup probe can time out, leaving Copilot without a shell, so the
revoke/sign-out step never runs. Expose pwsh to Copilot lanes on Windows.

gpt-5-4-nano sometimes drops the path segment after the space in
'project with spaces' when it rebuilds the file's absolute path. Give the GPT
journeys the full path.

Co-authored-by: Isaac <no-reply@databricks.com>
…s test PATH

Co-authored-by: Isaac <no-reply@databricks.com>
… and discovery

- A managed model_config.models list picks the start model again, as before.
- An unservable managed entry or a repeated candidate no longer warns twice.
- COPILOT_PROVIDER_WIRE_MODEL still sets the start model's served name.
- A failed Copilot session-file sweep warns instead of stopping ug revert.
- Copilot skips discovery when a provider, parent schema or managed list names
  its models.
- Drop the unreachable copilot status fallback.

Co-authored-by: Isaac <no-reply@databricks.com>
…ns under the mutex

- A UC listing without Claude or GPT models no longer discards the models the
  gateway listings found for that family, so a Gemini-only listing doesn't
  leave Copilot with an empty catalog.
- gpt-4o and gpt-4o-mini classify as Completions models again.
- Session cleanup holds the registries mutex, so a concurrent sweep can't reopen
  the lock file before it is deleted (a PermissionError on Windows).

Co-authored-by: Isaac <no-reply@databricks.com>
@xi-db
xi-db force-pushed the xi-db/stack/copilot-model-discovery branch from d27ae5f to fe24b4f Compare October 22, 2026 06:50
@xi-db
xi-db added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit 6d3c5bd Oct 22, 2026
55 checks passed
@xi-db
xi-db deleted the xi-db/stack/copilot-model-discovery branch October 22, 2026 07:33
yaoharry pushed a commit to yaoharry/unity-gateway that referenced this pull request Oct 24, 2026
…#1220)

## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/unity-gateway/pull/1220/files) to
review incremental changes.
-
[stack/copilot-auth-proxy](databricks#1217)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1217/files)]
[MERGED]
-
[stack/copilot-model-discovery](databricks#1218)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1218/files)]
[MERGED]
-
[**stack/copilot-provider-services**](databricks#1220)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1220/files)] ←
_this PR_
-
[stack/copilot-model-location](databricks#1256)
[[Files
changed](https://github.com/databricks/unity-gateway/pull/1256/files/a6c28cbc82fe5d36f47fdbf8f06744d4a2019427..f8ffea64fa222d47f61b3e53375bca312ebc780a)]

---------
## What did you change, and why?

**Change:** `ug copilot --provider <catalog>.<schema>.<name>` runs
Copilot through a Unity Catalog Model Provider Service, with that
service's models in `/model`. Claude Code and Codex already support
provider services; Copilot refused all of them ("supported: none").

- **Routing.** The session registry from databricks#1218 gets the service's
models. Every registry provider carries `headers:
{"Databricks-Model-Provider-Service": <service>}`, which Copilot sends
on every request, and the loopback proxy forwards it unchanged. The
proxy removes `reasoning_effort` from Chat Completions requests: Copilot
sends it, and OpenAI and Azure reject it for the models on that wire.
  - Claude models -> `dbx-anthropic` on `/ai-gateway/anthropic`.
- Other chat models -> `dbx-responses` (GPT 5+ and o-series, which
reject Copilot's `temperature: 0` on Chat Completions) or
`dbx-completions`, both on `/ai-gateway/openai/v1`, the gateway's native
OpenAI route, which serves provider services untranslated.
Databricks-hosted models keep `/ai-gateway/mlflow/v1`.
- **Provider types.**
- Supported: `anthropic`, `amazon_bedrock` and `bedrock_mantle` (Claude
targets only), `openai`, `azure_openai`.
- Refused with an actionable error: subscription-relay Anthropic
services (Claude Code only), Bedrock services without Claude targets,
other provider types. `microsoft_foundry` is left for a follow-up once
there is a test fixture for it.
- **Models.**
- Provider ids are classified whole (`gpt-5.5`, `gpt-4o`,
`us.anthropic.claude-sonnet-4-6`). The `system.ai` name logic took the
text after the last dot, which dropped `gpt-5.5` and `gpt-4o`.
- Dated, versioned and region-less Claude ids
(`claude-sonnet-4-5-20250929`, `...-20250929-v1:0`,
`anthropic.claude-opus-4-8`) map to Copilot's catalog id
(`claude-sonnet-4.5`), so Copilot sends the right Claude settings.
Provider Claude models whose catalog limits exceed 200K are capped at
200K, because a provider may not serve the 1M window.
- Declared targets are used as is. A service that allows every model
registers the gateway's listing for it (Anthropic `/v1/models`, Codex
`/v1/models`) plus `--model` (plain id or `dbx-<wire>/<id>`) or the
managed default. Without either, the session starts on Copilot's default
pick from the listing, as `ug codex` and `ug claude` do; the listing is
a fixed catalog, not what the provider account serves, so that pick can
fail. An empty or failed listing asks for `--model`, and a failed
listing is reported with its cause.
- Targets whose declared APIs (`native_api_types`) include no chat API,
such as embedding deployments, are skipped with a warning. The declared
APIs only exclude; the wire still comes from the model name. GPT and
o-series versions sort newest first.
- **Selection.** `ug configure` offers matching services for Copilot
under External Models, and a managed config's `model_provider_service`
for Copilot applies, the same way as for Claude Code and Codex. Under a
provider the managed Databricks model list is ignored.
- **Workspace.** `ug copilot --workspace <url>` targets that workspace
for the launch and sets it up first if needed, as `ug claude` and `ug
codex` do. Before, the flag was passed through to Copilot.

## How do you know it works?

**Testing:**
- **Unit:**
- provider classifier (wire per provider type, non-chat and non-Claude
exclusions), catalog order, Claude catalog ids for dated/Bedrock ids;
- provider catalog: declared targets, allow-all listing for
Anthropic/Bedrock and OpenAI, `--model` on allow-all, empty listing, no
usable model, skipped-model warning, invalid service;
- registry rendering with the header, native OpenAI base URL and 200K
Claude limits;
- launch through the real proxy with a stubbed Copilot process: registry
content, start model, a model the service doesn't serve fails before
Copilot starts;
- `resolve_provider_service` / `service_usable_for_tool` for Copilot,
and `ug copilot --provider` reaching the launch.
- The full unit suite passes except tests that also fail on the base in
this environment (`TestInstallToolBinary`, `test_e2e_user_agent`, three
`test_smart_router` cases). `just lint` is clean.
- **Integration (new `tests/integration/test_ug_copilot_providers.py`,
non-blocking Copilot lane):** file tasks through the existing Anthropic
MPS fixture (no `--model`), the OpenAI and Azure OpenAI MPS fixtures on
`gpt-5-nano`, and the Bedrock allow-all fixture with a named model, with
the answering model checked from Copilot's own session events; the
relayed fixture fails before Copilot starts. These run in CI for the
first time on this PR.
- **Live (`ug` from this branch, one provider service per row unless
noted):** One headless `ug copilot --provider <service> [--model <id>]
-p "Reply with exactly: OK"` per model, plus each service once without
`--model` to check the start model. The model shown is the one Copilot's
session log says answered.

| Provider type | Model | Copilot wire | Result |
|---|---|---|---|
| OpenAI | `gpt-5.4-nano`, `gpt-5.6-luna`, default -> `gpt-5.6-luna` |
Responses | ✅ OK |
| OpenAI (allow-all) | `gpt-5-nano`, default -> `gpt-6.1-sol` (from the
gateway listing) | Responses | ✅ OK |
| OpenAI (allow-all) | `gpt-4o-mini` | Chat Completions | ✅ OK |
| Azure OpenAI | `gpt-5.5`, `gpt-5.6-luna`, default start model |
Responses | ✅ OK |
| Azure OpenAI | `gpt-4.1-mini`, `gpt-4o-mini` | Chat Completions | ✅ OK
|
| Anthropic | `claude-haiku-4-5`, `claude-sonnet-5`, default ->
`claude-sonnet-5` | Anthropic | ✅ OK |
| Anthropic (allow-all) | `claude-haiku-4-5`, `claude-sonnet-4-5`,
default -> `claude-sonnet-5-5` (from the gateway listing) | Anthropic |
✅ OK |
| Amazon Bedrock (with non-Claude targets) |
`us.anthropic.claude-haiku-4-5-20251001-v1:0`,
`us.anthropic.claude-sonnet-4-6`, default ->
`us.anthropic.claude-sonnet-4-6` | Anthropic | ✅ OK; the service's
non-Claude targets (`grok-4.6`, `gpt-oss`) are left out of `/model`,
with a warning |
| Amazon Bedrock (service credential, allow-all; 3 services) |
`us.anthropic.claude-haiku-4-5-20251001-v1:0` | Anthropic | ✅ OK |
| Relayed Anthropic subscription | - | - | ✅ refused before launch:
"relays a Claude subscription, which only Claude Code can use" |

**Not in this PR:** `--model-location` for Copilot (next), Microsoft
Foundry services (needs a fixture), mixing Databricks and provider
models in one session.

Stacked on databricks#1218.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ug-review Run the automated UG review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants