Skip to content

PROVIDER_API.md

Latest commit

 

History

History
339 lines (282 loc) · 19.5 KB

PROVIDER_API.md

File metadata and controls

339 lines (282 loc) · 19.5 KB
Raw

Provider API

A provider teaches lean-coder to talk to a model backend other than the bundled Ollama - a hosted API, an OAuth subscription, anything. It is its own plugin type (NOT a lean-tool): a .py file with a module-level PROVIDER dict, dropped in the providers/ dir and toggled with /provider enable|disable. A provider never ships to a /connect remote and exposes no model tool; it wires in through a native registry, so it never has to monkey-patch core or mutate cfg.

Ollama is the bundled default (provider = "ollama"); more providers ship bundled too (anthropic_api, gemini, groq, openai, openrouter, kilo, ovh, plus the local servers llamacpp and mlx), disabled until you enable and key them. Config autosaves - no manual /save step.

To start your own, copy the annotated template at examples/providers/example.py.

Registering

A file in the providers dir (~/.config/leancoder/providers/, or the repo's providers/) declares a module-level PROVIDER dict and an optional, helpers-only setup(lc, cfg). The ProviderManager reads the dict and registers it for you - the file does not call register_provider itself.

# ~/.config/leancoder/providers/cloud.py
PROVIDER = {                       # module-level; the ProviderManager reads + registers it
    "name":           "cloud",
    "description":    "my hosted backend",         # shown in the /provider menu
    "make_client":    lambda cfg: ApiClient(cfg),  # required
    "list_models":    lambda: [...ids...],         # required; safe unauthed - cached list or [], never throws
    "context_window": lambda model: 200000,        # required
    # --- everything below is optional and absent-safe ---
    "capabilities":   lambda model: {              # declare settable knobs (see below)
        "thinking": ["off", "adaptive", "max"],    # [] / absent = unsupported
        "effort":   ["low", "med", "high"],
    },
    "available":      lambda: _Auth().is_logged_in(),  # provider offered only when True (default True)
    "autostart":      lambda: _Auth().is_active(),     # activate on launch when True (default False)
    "on_activate":    lambda agent, cfg: ...,          # called when switched to
    "on_deactivate":  lambda agent, cfg: ...,          # called when switched away
    "usage":          lambda agent, cfg: {...},        # quota meters; numbers, core renders (below)
    "status_line":    lambda agent, cfg: "raw string", # raw escape hatch if usage's model doesn't fit
    "login":          lambda agent, cfg: True,         # enables /provider login (returns truthy on success)
    "clear":          lambda agent, cfg: None,         # enables /provider clear (wipes credentials)
    "detail":         lambda agent, cfg: "full view",  # full account/billing view for /usage
    "tag":            "cloud",                         # short label in the unified /model row (default: name)
    "warm_models":    lambda: ["m1"],                  # loaded/instant models (green dot in /model, /usage)
    "model_status":   lambda model: None,              # install/availability hint in /model (e.g. "pull")
    "endpoints":      lambda: ["http://host:11434"],   # multi-endpoint failover (Ollama uses this)
    "restore_settings": lambda agent, cfg: ...,        # re-sync provider-HELD settings after a session load (below)
    "worker_inherit": ("account",),                    # captured settings keys a dispatched worker inherits (default ("account",); secret-looking keys never pass)
    "location":       lambda cfg: "box (http://...)",  # status/info label for the active backend
    "prepare":        lambda agent, cfg: None,         # pre-activation transport setup (below)
    "on_conn_error":  lambda agent, cfg, err: False,   # send-failure recovery; True = retry the turn
    "system_addendum": lambda agent, cfg: "",          # extra system-prompt text for the active model
    "login_files":    "auth*.json",                    # its login files in the config dir: /push + /pull list them under it
}

def setup(lc, cfg):               # OPTIONAL, helpers-only - do NOT register_provider here
    _lc.update(lc)                # the provider API (see "What lc contains" below)
    lc["register_command"]("/cloud_push", _push)   # bespoke provider commands are still fine

What lc contains

setup(lc, cfg) receives the provider API: a read-only mapping of exactly these core names, and nothing else (core's internals are not reachable, and a provider can't write into core). Look a name up as lc["dim"]; an unknown name raises KeyError. Keep your own state (e.g. the cfg) in your module.

Group Names
Display green red yellow dim bold blue glyph GLYPH Spinner THINK_FRAMES MarkdownStream
Streaming consume_openai_stream upstream_error_of stream_tiered open_tiered StreamStall parse_text_tool_calls messages_tokens encode_image_block
Config + limits CONFIG_DIR save_config autosave_config sampling_for hosted_sampling RateLedger log_system_activity
Commands + prompts register_command register_provider pick_one pick_model_menu model_available handle_model_command handle_sync_command ask menu_index visible_wait TurnAbort TTY
Hosts resolve_host host_label valid_host norm_host
Live state active_composer() - the input composer owning the terminal, or None

The bundled providers use nothing outside this list (a test enforces it). Need something that isn't here? Ask for it to be added rather than reaching into core.

Opt-in hooks are absent-safe: implement only what you want; omitting a hook costs nothing (core degrades gracefully). A hosted single-endpoint API returns [] from warm_models() and omits endpoints(); the bundled Ollama provider implements both. There is no on_error hook - retry/fallback lives inside the client (client.chat()), with core's cross-provider next-model fallback as the backstop.

Transport hooks (all optional; the bundled Ollama provider implements each):

  • location(cfg) -> str: short label for the active backend shown in status/info output. Ollama returns name (url) when the host has a machine alias, else the bare URL. Exceptions are swallowed and the label is omitted.
  • prepare(agent, cfg): run by core BEFORE activate_provider() when switching to the provider. Use it for transport setup: Ollama probes its priority host pool, picks the first live host into cfg.host, then adopts that host's last-used model. Return value is ignored.
  • on_conn_error(agent, cfg, err) -> bool: last-chance recovery when a send fails at the transport level. Return True if you fixed things (e.g. Ollama fails over to the next live host in its pool) and core retries the turn; False and core prints the original error.
  • system_addendum(agent, cfg) -> str: extra text appended to the system prompt for the active model (only when the model supports tools). Ollama returns short tool-calling guidance for small models (by parameter size) and "" otherwise. Keep it deterministic per model so prompt caching stays valid.

First-class onboarding: login + recognizable auth errors

Two small conventions give your provider the same smooth OBE as the bundled ones:

  1. Implement login (and clear). Core wires login to /provider login <name>, and /provider login <name> will even enable a bundled-but-disabled provider on the spot, so a new user goes straight from key to running. Read the secret with getpass (never echo it), store it chmod 600 under lc["CONFIG_DIR"], and return truthy on success / False if the user cancelled.

  2. Raise a recognizable auth error from chat() when the key is missing or the API rejects it (HTTP 401/403). Core inspects the exception text and, if it looks like an auth failure, offers the login flow inline and retries the turn once - so a stale or absent key never dead-ends at a red error. Make the RuntimeError message contain one of: no API key, API key, 401, 403, or unauthorized. For example:

    def chat(self, messages, tools, should_abort=None):
    key = _api_key()
    if not key:
        raise RuntimeError("no API key - use /provider login")   # -> core offers login + retries
    ...
    except urllib.error.HTTPError as e:
        if e.code in (401, 403):
            raise RuntimeError(f"API {e.code} - key rejected; /provider login")

    Returning a normal assistant message on an auth failure would look like a real reply and strand the user - raise instead. See examples/providers/example.py for the full pattern.

Secrets go under lc["CONFIG_DIR"] (~/.config/leancoder), chmod 600 - never in config.toml (it autosaves and would round-trip your key back to disk). Resolve keys from an env var first, then a keyfile (see the bundled providers for the pattern).

The client (make_client(cfg) returns)

Same duck type as OllamaClient - the contract the agent loop already speaks:

  • chat(messages, tools, should_abort) -> (assistant_dict, prompt_eval|None, aborted_bool) (required). should_abort() returns True when the user hits Ctrl-C; poll it during streaming and terminate early (set aborted=True) if it fires.
  • uncapped_ctx = True (optional attr) opts out of the 32k local-VRAM OOM cap - a hosted client has no local memory limit, so it should trust its own window.
  • list_models() / running_models() - optional; core falls back to the spec's list_models and [].
  • Per-call observability attributes the client MAY set after each chat() (core reads them off the client; all optional, treated as 0 if absent):
    • last_out_tokens - output tokens for the call; feeds agent.session_out.
    • last_cache_read / last_cache_write - prompt-cache read/write tokens. A caching backend MUST set these. Core's context meter (and the auto-compaction zones, and the status row) compute the true fill as prompt_eval + last_cache_read + last_cache_write - with caching, prompt_eval is just the uncached input and alone under-reports real context size by ~10x, which misleads the meter and stops compaction firing. A non-caching backend leaves them 0 (the meter falls back to prompt_eval).

The client holds the cfg it was built with and reads, at send time:

  • cfg.active_model() - the model to send (the provider's, not Ollama's).
  • cfg.setting("thinking"), cfg.setting("effort"), cfg.setting("cache"), ... - any knob the provider declared, plus any free-form one the user set via /provider set. The client interprets these; core only stores them.

It does not mutate cfg.

Streaming: use the shared reader

If your backend streams OpenAI-style chunks (data: {...chat.completion.chunk...} SSE lines, ending in data: [DONE]), don't write your own loop. Call core's reader from chat() on the open response:

out = lc["consume_openai_stream"](client, resp, should_abort, tools,
                                  on_line=..., on_obj=..., on_done=...,
                                  text_fallback=False, split_think=False)
return out        # (assistant_dict, prompt_eval|None, aborted)

It renders as it goes (spinner, then reasoning dimmed, then the answer), never stores reasoning (reasoning_content / reasoning / thinking fields), joins streamed tool calls (string fragments or a whole parsed object), polls should_abort, and sets client.last_finish, last_truncated (finish length) and last_out_tokens.

Options:

  • text_fallback - when the model sent no native tool calls, parse text-encoded ones (<tool_call>{...}</tool_call> etc.) from the answer. On for local servers whose small models do this; off for a hosted API, so text that merely looks like a call stays text.
  • split_think - move inline <think>...</think> out of the answer into the dim reasoning line. On only where models send thinking as raw tags (local servers); off elsewhere, so a literal <think> in prose or code is kept.

Hooks, for what is specific to your backend (all optional):

  • on_line(line, obj, st) - every non-blank line before core handles it; obj is the parsed chunk, or None for a non-data: line / bad JSON. Return True to stop reading, or raise to fail the request. Pair it with lc["upstream_error_of"](obj) to catch an upstream error sent inside a 200 stream: raise (so your retry logic runs) when not st["printed"] and not st["tool_acc"], else warn and return True to keep the partial reply.
  • on_obj(obj, st) - every parsed chunk, after core: read extra fields (served model, cache counts, timings).
  • on_done(st) - once, after the stream: adjust st["prompt_total"] / st["out_tokens"] / st["finish"] (e.g. subtract cached tokens; mark a refusal).

st is a plain dict: finish, content_parts, prompt_total, out_tokens, aborted, printed, printed_think, chunks, tool_acc. The bundled openai, groq, openrouter, kilo, ovh, llamacpp and mlx providers all use it; the template examples/providers/example.py shows the minimal form.

Settings & capabilities

capabilities(model) returns {setting_key: [choices]}. thinking and effort are the well-known keys (they get the friendly /think / /effort commands); any other key you declare is a first-class knob too - core lists it under /provider set, validates it against its choices, persists it, and menus it. A knob with no declared choices is still settable free-form via /provider set <key> <value> and read with cfg.setting(key), so a provider can wire in something core has never heard of - no core change.

"capabilities": lambda model: {
    "thinking": ["off", "adaptive", "max"],
    "effort":   ["low", "med", "high"],
    "cache":    ["off", "on"],          # custom: core persists/sets, client reads
},

This is the extensibility seam: a provider adds new behaviour (prompt caching, a tool-calling dialect, a beta header) by declaring and reading a setting key.

Config & lifecycle

Provider state is core-owned and autosaved to config.toml:

  • provider - the active backend name (default "ollama").
  • [providers.<name>] table - per-provider model (and optionally thinking / effort). The bundled Ollama provider's model lives in [providers.ollama].model; top-level model is kept as a back-compat alias. Ollama's transport config (host / hosts / [machines] / num_ctx) stays top-level.
  • num_ctx is set from context_window(model) on activate / /model, and is never written to config.toml by a provider.

At launch, core activates a provider if cfg.provider names a registered + available() one, or some provider's autostart() returns true. Activation builds the client via make_client, swaps it in (re-running ctx detection), applies thinking / effort against capabilities, and calls on_activate. Switching away (or /provider off) calls on_deactivate and falls back to Ollama.

Session-scoped settings (restore_settings). A provider setting stored in cfg.provider_settings[name] (via cfg.set_setting) rides the session meta and is restored on /load like thinking / effort. If your provider reads that setting LIVE from cfg each turn, that's all you need. But if you cache it in your own module state (e.g. the Anthropic provider keeps the active account label in a module global, seeded once at setup), a load updating cfg won't reach that state - so core calls restore_settings(agent, cfg) after a load to let you re-sync from cfg and rebuild (re-auth, refetch usage). It MUST be session-scoped: do NOT persist to config.toml there, so two windows on one box can hold different accounts. Absent -> nothing.

Commands core provides

So each backend stops re-implementing them:

  • /provider list | enable <name> | disable <name> - manage provider plugins in the dir (no arg = toggle menu). Enabling registers it; disabling deactivates + drops it.
  • /provider [name] - list registered providers (available ones marked) and switch the active one. No arg = menu.
  • /provider login [name] / /provider clear [name] - run the provider's login / clear hooks. login does the auth flow and returns truthy on success (core then activates it); clear wipes credentials (core drops back to Ollama if it was active). A provider that omits a hook simply isn't offered it; the name is optional when exactly one provider implements the hook.
  • /model - shows the active provider's list_models().
  • /think / /effort - capability-aware pickers over the declared choices when the active provider declares them; a clear "not supported" otherwise. (Ollama keeps its boolean /think.)
  • /provider set [key [value]] - list the active provider's declared settings + values (no args), menu a declared key (key only), or set any key including a free-form one. (App-config knobs live in the separate /set.)
  • /usage - prints the active provider's detail(agent, cfg) -> str (its full account/billing view), or a core default (model, session tokens, context %, warm models) for a provider without detail.

Auth is the provider's business, but its command surface is core: /provider login|clear replace per-backend login commands. Only genuinely bespoke actions stay as the provider's own register_command.

The usage / status line

The per-response line under each turn is rendered entirely by core - the provider feeds numbers, so styling stays consistent across every backend and there's one place to change it. After the context meter, core calls usage(agent, cfg) on the active provider if defined:

def usage(agent, cfg):
    return {
        "meters": [                                   # rendered as "label pct% (resets)"
            {"label": "5h", "pct": 12, "resets_at": "2026-06-27T20:00:00Z"},
            {"label": "wk", "pct": 40, "resets_at": "2026-07-01T00:00:00Z"},
        ],
        "note": "",                                   # optional trailing text
    }

Core owns the colour ramp (green/orange/red by pct), the resets_at -> "20:00" / "01Jul" formatting, and FULL handling at >=100%. Returning None / {} keeps the plain context meter. The same output also renders as the tail of the condensed status row above the prompt - fill in usage and you get quota meters in both places for free. status_line(agent, cfg) -> str is a raw-string escape hatch for a backend whose display genuinely doesn't fit the meter model; if both are defined, usage wins.

Session token counter

Core accumulates agent.session_in / agent.session_out across the session (zeroed by /clear), so /usage shows a token count for any backend without the provider tracking it. Input comes from chat()'s returned prompt_eval; output from the optional client.last_out_tokens the client sets per call.

Sampling (local providers)

Local backends should not send a global temperature. Call _lc["sampling_for"](cfg, model, server_card, "<where the card came from>") and send only the returned send dict; pass the defaults your server already applies (e.g. llama.cpp /props, ollama /api/show PARAMETERs, generation_config.json) as server_card, or None if unknown. Store {"params": effective, "source": source, "sent": send} on client.last_sampling (and/or expose client.resolve_sampling()) so /usage and lean_coder.effective_sampling() can report it. Hosted providers send no sampling parameters at all. See docs/design/sampling-defaults.md.