Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,9 @@
<PackageVersion Include="Shouldly" Version="4.3.0" />
<PackageVersion Include="Microsoft.Testing.Extensions.CodeCoverage" Version="18.12.0" />
<PackageVersion Include="Microsoft.Testing.Extensions.TrxReport" Version="2.5.1" />
<!-- Source guards: the tests parse agent sources with Roslyn (syntax only) — e.g. no agent may hard-code the
provider or model of its LLM call (AgentRegistryGuardTests). -->
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="5.9.0" />

<!-- Aspire orchestration + service defaults (.NET 10). HELD at 13.5.3: Aspire 13.6.x builds but the
AppHost fails at startup on this stack (DCP object creation times out after 60s in
Expand Down
26 changes: 17 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ Light and dark are first-class (top bar → appearance toggle, or System → App
| Concept | Lives in | Notes |
|---|---|---|
| **Module** | `IModule` (SharedKernel) | One assembly = one module. Optional `IEndpointModule` (HTTP), `IInitializableModule` (startup hook), `ISettingsContributor` (settings it owns). |
| **Agent** | `Modules.Pipeline.Agents` | A specialist — Requirement, Coding, Testing, QA, Orchestrator (+ IssueWork, Decomposer). Talks to models only through `ILlmClientFactory`. |
| **Agent** | `Modules.Pipeline.Agents` | One registry (`AgentRoles`): the Board agents (Discovery, Coding (tickets), Reviewer, Testing (tickets), Autopilot lead, Retrospective) and the 5-agent Workflow pipeline (Orchestrator, Requirement, Coding, Testing, QA). Each role key names its profile, prompt and Insights row. Talks to models only through `ILlmClientFactory`. |
| **LLM client** | `Domain.Llm.ILlmClient` | `SendAsync(LlmRequest, ct) → LlmResponse`, registered as a keyed service under its provider name. |
| **Setting** | `Domain.Configuration.SettingDescriptor` | Key, kind, scope, default, validation, UI label. Resolved lock → tenant → platform → seed → default ([ADR-0011](docs/adr/0011-ui-first-configuration.md)). |
| **Tool** | `Domain.Tools.ITool` | Invoked by models via the gateway, always through `IToolGateway` (policy → invoke → evidence). |
Expand Down Expand Up @@ -295,12 +295,18 @@ so a key saved in Settings is used by the next call.

| Agent | Provider | Model |
|---|---|---|
| Orchestrator | Claude | `claude-haiku-4-5` |
| Requirement | Claude | `claude-sonnet-4-6` |
| Coding | AzureOpenAI | `gpt-4.1` |
| Testing | AzureOpenAI | `gpt-4o-mini` |
| QA | Claude | `claude-haiku-4-5` |
| IssueWork / Decomposer | Claude | `claude-sonnet-4-6` |
| Board: Discovery, Coding (tickets) | Claude | *Computer's default* (on our servers: Claude's configured model) |
| Board: Reviewer, Testing (tickets), Autopilot lead, Retrospective | RemoteAgent (a paired computer, 0 server tokens) | *Computer's default* (the member's CLI model) |
| Workflow pipeline: Orchestrator | Claude | `claude-haiku-4-5` |
| Workflow pipeline: Requirement | Claude | `claude-sonnet-4-6` |
| Workflow pipeline: Coding | AzureOpenAI | `gpt-4.1` |
| Workflow pipeline: Testing | AzureOpenAI | `gpt-4o-mini` |
| Workflow pipeline: QA | Claude | `claude-haiku-4-5` |

Every agent — Board and pipeline — reads its profile and its prompt (Settings → Prompts) per call, passes the budget
gate unless it really runs on a member's computer, and appears in Insights under the same name. A Board agent chooses
no model until someone picks one: a paired computer's CLI then runs its own default, our servers the provider's
configured model; a picked model reaches the CLI as `claude --model` / `codex -m`. See [CONFIG.md](docs/CONFIG.md#settings--agent-profiles).

The model picker lists every priced model for the chosen provider with its list price (e.g.
`claude-haiku-5-5 · $0.1 / $0.5` per 1M tokens), plus *Custom…* for Azure deployment aliases.
Expand Down Expand Up @@ -397,8 +403,10 @@ The most used routes:
### Add a pipeline agent

Use the bundled `agent-scaffold` skill (`/agent-scaffold {Name}`) — it generates the contract,
implementation, DI registration and xUnit stubs. Give it an agent profile in `AgentsOptions` so it
appears in Settings → Agent profiles.
implementation, DI registration and xUnit stubs. Register its role in `AgentRoles` (key, label, Board or
Workflow pipeline) and give it a seed in `AgentsOptions` (property named after the key) so it appears in Settings →
Agent profiles; resolve its profile per call (`BoardAgent.ProfileAsync` / `IAgentProfiles`) — `AgentRegistryGuardTests`
fails the build on a hard-coded provider or model.

### Add a setting

Expand Down
60 changes: 46 additions & 14 deletions docs/CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,20 +98,52 @@ exists on the options type but is not honoured yet.

### Settings → Agent profiles

Per role — `Orchestrator`, `Requirement`, `Coding`, `Testing`, `Qa`, `IssueWork`, `Decomposer` — keys
`Agents:<Role>:Provider` / `Model` / `Temperature` / `MaxTokens`, tenant scope. The model field lists
priced models for the chosen provider (plus *Custom…* for deployment aliases); `Temperature` is hidden
for models that reject sampling params; `MaxTokens` ≤ 128 000.

| Role | Provider | Model | Temp | Max output |
|---|---|---|---|---|
| Orchestrator | Anthropic | `claude-haiku-4-5` | 0.3 | 2000 |
| Requirement | Anthropic | `claude-sonnet-4-6` | 0.1 | 2000 |
| Coding | AzureOpenAI | `gpt-4.1` | 0.2 | 4000 |
| Testing | AzureOpenAI | `gpt-4o-mini` | 0.2 | 3000 |
| Qa | Anthropic | `claude-haiku-4-5` | 0.1 | 1500 |
| IssueWork | Anthropic | `claude-sonnet-4-6` | 0.2 | 32768 |
| Decomposer | Anthropic | `claude-sonnet-4-6` | 0.2 | 4000 |
One agent registry (`AgentRoles`): every agent AgentOS.NET runs has one stable **role key**, used for its
profile (`Agents:<Key>:Provider` / `Model` / `Temperature` / `MaxTokens`, tenant scope), its prompt in
**Settings → Prompts** (`prompt/<Key>/system`), the `AgentRole` of its LLM calls (what **Insights → By agent**
groups and names by) and the label of its stored runs. The page shows two tables — the **Board agents** and the
legacy 5-agent **Workflow pipeline** — so a change to one never silently touches the other. Every profile is read
per call (no restart). The model field lists priced models for the chosen provider (plus *Custom…* for deployment
aliases); `Temperature` is hidden for models that reject sampling params; `MaxTokens` ≤ 128 000.

| Shown as | Key | Provider | Model | Temp | Max output |
|---|---|---|---|---|---|
| Discovery (New idea: requirement + ticket split) | `Discovery` | Anthropic | *Computer's default* | 0.2 | 4000 |
| Coding (tickets) | `IssueWork` | Anthropic | *Computer's default* | 0.2 | 32768 |
| Reviewer | `Reviewer` | RemoteAgent | *Computer's default* | 0 | 32768 |
| Testing (tickets) | `Acceptance` | RemoteAgent | *Computer's default* | 0 | 32768 |
| Autopilot lead | `AutopilotOrchestrator` | RemoteAgent | *Computer's default* | 0 | 4096 |
| Retrospective | `Retrospective` | RemoteAgent | *Computer's default* | 0 | 4096 |
| Orchestrator (Workflow pipeline) | `Orchestrator` | Anthropic | `claude-haiku-4-5` | 0.3 | 2000 |
| Requirement (Workflow pipeline) | `Requirement` | Anthropic | `claude-sonnet-4-6` | 0.1 | 2000 |
| Coding (Workflow pipeline) | `Coding` | AzureOpenAI | `gpt-4.1` | 0.2 | 4000 |
| Testing (Workflow pipeline) | `Testing` | AzureOpenAI | `gpt-4o-mini` | 0.2 | 3000 |
| QA (Workflow pipeline) | `Qa` | Anthropic | `claude-haiku-4-5` | 0.1 | 1500 |

- **Computer's default (Board agents).** A Board agent chooses **no model** unless someone picks one. On a member's
paired computer (`RemoteAgent`) the request carries no model and the runner passes no flag, so the CLI runs its
own default — the member's choice, usually newer than any id AgentOS.NET would pin. On our servers the LLM gateway
fills in the provider's configured model (`Llm:Claude:Model`, the Azure default deployment, the vendor's / the
OpenAI-compatible server's model from **Settings → Model keys**) before the request is validated. A model picked
in the profile is always passed: `claude --model <id>`, `codex -m <id>` (only when that CLI can serve it). The
Workflow-pipeline agents always name a model. Reviewer and Testing (tickets) need a checkout, so keep them on a
paired computer unless the chosen provider can reach one.
- **Budget gate.** Every Board agent passes the tenant's enforced monthly cap before its call, decided from the
*effective* provider (after `Llm:ForceProvider`); only a call that really runs on a member's computer (0 server
tokens) is exempt.
- **Discovery** covers both calls of New idea: the requirement step uses the Discovery profile (and is filed under
Discovery in Insights) with the Requirement agent's prompt and spec contract; the ticket split uses the Discovery
prompt. It replaces the former `Decomposer` keys, which **keep working**: while a Discovery field has no stored
value, the tenant's (then the platform's, then the appsettings) `Agents:Decomposer:<Field>` applies, and the
Decomposer prompt override applies while no Discovery override is saved. Agent profiles marks the row *Saved as
"Decomposer" — still applies*; saving a field stores it under Discovery, and **Reset** (profile or prompt) clears
the tenant's Decomposer values too.
- **Prompts.** Board prompts are templates: `{owner}`, `{repo}`, `{branch}`, `{defaultBranch}` (Reviewer, Testing
(tickets)) and `{issue}`, `{branchStep}`, `{commitType}`… (Coding (tickets), which has one prompt for a member's
computer and one — `IssueWork.Server` — for the server runner) are filled on every run. The untrusted-text rule is
appended by code and cannot be overridden; workspace rules follow it.
- **Practices.** Coding, Reviewer, Testing (tickets) and the Autopilot lead read the project's practices (Board →
Knowledge) before they work.

`Pipeline:MaxIterations` (QA-loop cap when a story sets no `NMax`) — default 3, range 1–10.
An unpriced model is charged at the fail-closed $15/$75 per 1M rate until added to `CostCalculator`.
Expand Down
19 changes: 14 additions & 5 deletions src/AgentOs.Domain/Llm/LlmRequest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,20 @@ public sealed record LlmRequest(
/// Validates required values. Throws <see cref="System.ArgumentException"/> if invalid.
/// </summary>
public void Validate()
{
ValidateAllowingNoModel();
if (string.IsNullOrWhiteSpace(Model))
{
throw new System.ArgumentException("Model must not be null or whitespace.", nameof(Model));
}
}

/// <summary>
/// Validates required values but accepts an empty <see cref="Model"/>: a provider that chooses the model itself (a
/// member's paired CLI runs its own default) accepts "no model chosen". Server providers resolve an empty model to
/// their configured one and then call <see cref="Validate"/>.
/// </summary>
public void ValidateAllowingNoModel()
{
if (SystemPrompt is null)
{
Expand All @@ -88,11 +102,6 @@ public void Validate()
throw new System.ArgumentException("UserPrompt must not be null or whitespace.", nameof(UserPrompt));
}

if (string.IsNullOrWhiteSpace(Model))
{
throw new System.ArgumentException("Model must not be null or whitespace.", nameof(Model));
}

if (Temperature is < 0.0 or > 2.0)
{
throw new System.ArgumentException("Temperature must be in [0, 2].", nameof(Temperature));
Expand Down
17 changes: 17 additions & 0 deletions src/AgentOs.Domain/Workspaces/IBoardRepos.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// The repositories a board works on, as "owner/repo" keys — what project memory (lessons, practices) is keyed by. Lets
// an agent that sees a whole board (the Autopilot lead) read the practices of its projects without depending on the
// Workspaces module. Tenant-explicit: background runs have no request context.

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;

namespace AgentOs.Domain.Workspaces;

/// <summary>Lists a board's repositories.</summary>
public interface IBoardRepos
{
/// <summary>The <c>owner/repo</c> keys of board <paramref name="boardId"/> in <paramref name="tenantId"/> (empty when unknown).</summary>
Task<IReadOnlyList<string>> RepoKeysAsync(string tenantId, Guid boardId, CancellationToken ct = default);
}
6 changes: 5 additions & 1 deletion src/AgentOs.Modules.Llm/ChatClientLlmClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,24 @@ namespace AgentOs.Modules.Llm;
public sealed class ChatClientLlmClient : ILlmClient
{
private readonly IChatClient _chat;
private readonly string? _defaultModel;

/// <inheritdoc />
public string Provider { get; }

public ChatClientLlmClient(IChatClient chat, string provider)
/// <summary>Create. <paramref name="defaultModel"/> is used when a request chose no model.</summary>
public ChatClientLlmClient(IChatClient chat, string provider, string? defaultModel = null)
{
_chat = chat ?? throw new ArgumentNullException(nameof(chat));
Provider = string.IsNullOrWhiteSpace(provider) ? "ChatClient" : provider;
_defaultModel = defaultModel;
}

/// <inheritdoc />
public async Task<LlmResponse> SendAsync(LlmRequest request, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(request);
request = ProviderDefaultModel.Apply(request, _defaultModel);
request.Validate();

var messages = new List<ChatMessage>(2);
Expand Down
14 changes: 10 additions & 4 deletions src/AgentOs.Modules.Llm/LlmModule.cs
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,9 @@ public void AddServices(IServiceCollection services, IConfiguration configuratio
tenantProvider: sp,
// Classify SDK errors so the key pool fails over on transient (5xx/timeout) + auth errors,
// not only 429.
classifyError: SdkChatClients.Classify);
classifyError: SdkChatClients.Classify,
// An agent profile left on "no model chosen" runs Claude's configured model.
defaultModel: () => opts.Value.Claude.Model);
});
services.AddKeyedSingleton<ILlmClient>("AzureOpenAI", (sp, _) =>
{
Expand Down Expand Up @@ -107,7 +109,9 @@ string DefaultDeployment() => settings is null
// Fold the tenant's resolved endpoint into the client cache key (read per call from the same
// override source the factory uses) so two tenants sharing a key but not an endpoint never
// share a client bound to the wrong Azure resource.
cacheKeyDiscriminator: () => !string.IsNullOrWhiteSpace(ov.AzureEndpoint) ? ov.AzureEndpoint! : opts.Value.AzureOpenAi.Endpoint);
cacheKeyDiscriminator: () => !string.IsNullOrWhiteSpace(ov.AzureEndpoint) ? ov.AzureEndpoint! : opts.Value.AzureOpenAi.Endpoint,
// An agent profile left on "no model chosen" runs the tenant's default deployment.
defaultModel: DefaultDeployment);
});

// BYOK vendors (OpenAI, Gemini, xAI): fixed endpoints, tenant-scoped keys read live per call, the profile's
Expand All @@ -132,7 +136,8 @@ string DefaultDeployment() => settings is null
toolInvocationLog: sp.GetService<AgentOs.Domain.Tools.IToolInvocationLog>(),
tenantProvider: sp,
classifyError: SdkChatClients.Classify,
modelResolver: requested => vendor.ResolveModel(requested, Read(vendor.ModelSetting)));
modelResolver: requested => vendor.ResolveModel(requested, Read(vendor.ModelSetting)),
defaultModel: () => vendor.ResolveModel(null, Read(vendor.ModelSetting)));
});
}

Expand Down Expand Up @@ -160,7 +165,8 @@ string Endpoint() => Read(LlmSettings.CompatEndpoint)
classifyError: SdkChatClients.Classify,
cacheKeyDiscriminator: () => Read(LlmSettings.CompatEndpoint) ?? string.Empty,
modelResolver: requested => OpenAiCompatibleModels.Resolve(requested, Read(LlmSettings.CompatModel)),
costOverride: (_, usage) => OpenAiCompatibleModels.Cost(usage, Read(LlmSettings.CompatInputPrice), Read(LlmSettings.CompatOutputPrice)));
costOverride: (_, usage) => OpenAiCompatibleModels.Cost(usage, Read(LlmSettings.CompatInputPrice), Read(LlmSettings.CompatOutputPrice)),
defaultModel: () => Read(LlmSettings.CompatModel));
});

services.AddSingleton<ILlmClientFactory, LlmClientFactory>();
Expand Down
6 changes: 6 additions & 0 deletions src/AgentOs.Modules.Llm/MafChatClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ _settings is null
public async Task<LlmResponse> SendAsync(LlmRequest request, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(request);
// No model chosen in the agent's profile → the configured Azure deployment (live setting, else the option).
if (string.IsNullOrWhiteSpace(request.Model))
{
request = ProviderDefaultModel.Apply(request,
(_settings is null ? null : LiveSettings.Read(_settings, LlmSettings.AzureModel, AmbientIdentity.Current?.TenantId)) ?? _options.Model);
}
request.Validate();

// Resolve credentials with the SAME tenant-override-first precedence as the AzureOpenAI pool — a tenant
Expand Down
9 changes: 8 additions & 1 deletion src/AgentOs.Modules.Llm/PooledChatLlmClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,10 @@ public PooledChatLlmClient(
Func<Exception, LlmErrorKind>? classifyError = null,
Func<string>? cacheKeyDiscriminator = null,
Func<string, string>? modelResolver = null,
Func<string, TokenUsage, decimal?>? costOverride = null)
Func<string, TokenUsage, decimal?>? costOverride = null,
Func<string?>? defaultModel = null)
{
_defaultModel = defaultModel;
Provider = string.IsNullOrWhiteSpace(provider) ? throw new ArgumentException("provider required", nameof(provider)) : provider;
_clientFactory = clientFactory ?? throw new ArgumentNullException(nameof(clientFactory));
_keyProvider = keyProvider ?? throw new ArgumentNullException(nameof(keyProvider));
Expand Down Expand Up @@ -97,6 +99,9 @@ public PooledChatLlmClient(
private readonly Func<string, string>? _modelResolver;
private readonly Func<string, TokenUsage, decimal?>? _costOverride;

// The provider's configured model (read live per call), used when the agent's profile chose none.
private readonly Func<string?>? _defaultModel;

// The tenant for tool tagging / telemetry when there is no AmbientIdentity on the call. Prefer a
// directly-supplied ITenantContext (tests); otherwise resolve the request-scoped one inside a fresh
// scope off the root provider (safe from a singleton — IHttpContextAccessor is itself a singleton, so
Expand All @@ -113,6 +118,8 @@ public PooledChatLlmClient(
public async Task<LlmResponse> SendAsync(LlmRequest request, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(request);
// No model chosen in the agent's profile → this provider's configured model (before validation).
request = ProviderDefaultModel.Apply(request, string.IsNullOrWhiteSpace(request.Model) ? _defaultModel?.Invoke() : null);
request.Validate();

var keys = await _keyProvider(cancellationToken).ConfigureAwait(false);
Expand Down
22 changes: 22 additions & 0 deletions src/AgentOs.Modules.Llm/ProviderDefaultModel.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// "No model chosen" in an agent profile (the Board agents' default) means: let whoever serves the call pick. A member's
// paired CLI runs its own default model; a server provider runs the model configured for it (Settings → Model keys /
// Llm:<Provider>:Model). Each server client resolves the empty model here, BEFORE LlmRequest.Validate, so a profile
// left on its default never throws on our servers.

using AgentOs.Domain.Llm;

namespace AgentOs.Modules.Llm;

/// <summary>Fills an empty request model with the serving provider's configured model.</summary>
public static class ProviderDefaultModel
{
/// <summary><paramref name="request"/> unchanged when it names a model (or no default is known); otherwise the same
/// request with <see cref="LlmRequest.Model"/> = <paramref name="providerDefault"/>.</summary>
public static LlmRequest Apply(LlmRequest request, string? providerDefault)
{
System.ArgumentNullException.ThrowIfNull(request);
return string.IsNullOrWhiteSpace(request.Model) && !string.IsNullOrWhiteSpace(providerDefault)
? request with { Model = providerDefault.Trim() }
: request;
}
}
Loading
Loading