Repository navigation
cli: talk to any TypeSafe-compatible server (e.g. Open-Jev) and run local models #36
Description
Today cli always builds DefaultTypeSafeClient.builder().apiKey(ApiKey.fromDefaultFile()) against api.typesafe.ai: no other endpoint, the key file is required, and nothing local.
Proposal
Any TypeSafe-compatible HTTP server
--endpoint <url>→Builder.endpoint(URI), e.g. a server running Open-Jev or the MLX Clef-flash server from the how-to (http://localhost:8000/v1/systemone).- The API key becomes optional when
--endpointis given: no~/.typesafe.apikeymeans noAuthorizationheader (the builder already supports that).
Local models (client-local)
- e.g.
--local laya <dir>/--local qwen <dir>/--local clef <dir>→LocalLayaTypeSafeClient.load(dir)etc.; the rest of the CLI (questions,--print,--verbose,--timing) is unchanged, since all of them areTypeSafeClients.
Open questions
- Jar size:
client-localbrings ONNX Runtime (~56 MB, native libraries for every platform) into thealluber-jar. Options: a separate classifier (e.g.all-local), or loadclient-localonly if it's on the classpath. - Flag shape:
--local <engine> <dir>, or--engine laya --model-dir <dir>? - Model id:
--modelis meaningless locally (the local clients accept onlyjev-latest/jev-preview/their own id); reject it there, or ignore it?
Depends on the module split (#35) landing first.
🤖 Generated with Claude Code
Activity
dfa1 commentedon Oct 8, 2026
--endpointlanded (33d74d1): points the CLI at any TypeSafe-compatible server, with~/.typesafe.apikeybecoming optional once it's set.Decided against
--localafter checking real numbers: ONNX Runtime alone is ~53MB (com.microsoft.onnxruntime:onnxruntime:1.30.0), vs. today'sall.jarat ~2.6MB — a ~21x jump for an opt-inall-localclassifier, with no way to trim it since the size is the runtime itself, not the models. That's a disproportionate cost for what the CLI is for (quick ad hoc checks without writing Java).docs/how-to.mdalready covers running local models by addingclient-localas a library dependency and writing a few lines of Java, which stays the right path for that use case.Closing as resolved via
--endpoint.cli: --endpoint for any TypeSafe-compatible server
dfa1 commentedon Oct 8, 2026
To be precise about what still covers the local-model use case: run the model as its own server (Open-Jev, the MLX Clef-flash server from the how-to, etc.) and point the CLI at it with
--endpoint— no need for the CLI itself to embed ONNX Runtime.