BaseRT

Fastest LLM inference on Apple Silicon — now on NVIDIA GB10 (DGX Spark) and AMD Strix Halo / Strix Point, too.

Pull a model from HuggingFace, chat with it, or serve an OpenAI-compatible API — all from one CLI.

---

BaseRT runs large language models locally on Apple Silicon (Metal), NVIDIA GB10 / DGX Spark (CUDA) and AMD Strix Halo / Strix Point (ROCm), accelerated by hand-written GPU kernels. This repository is the open ecosystem around the engine: the basert CLI (model hub + converter), the .base model format, the public C API, and language bindings for Python, Node, Rust, and Swift.

• One CLI for everything — basert pull, chat, serve, convert.
• OpenAI-compatible server — chat, completions, embeddings, transcription, tool calls, continuous batching, paged-KV, and prefix caching.
• Multimodal — text, vision, and audio on supported models.
• Its own .base format — affine quantization (Q2–Q8), optional AWQ calibration, and signed bundles.
• Bindings everywhere — Python, Node, Rust, and Swift over a stable C API.
• A drop-in coding-agent backend — see below.

Documentation: docs.basecompute.co (source in docs/).

Requirements

One of:

• Apple Silicon (M1 or later), macOS 14+.
• NVIDIA GB10 (DGX Spark), Linux / arm64 with a CUDA driver.
• AMD Strix Halo (Ryzen AI Max, gfx1151) or Strix Point (Ryzen AI 300, gfx1150), Linux / x86-64 on Ubuntu 24.04 or newer, with the ROCm 10 runtime from AMD's apt repository (below). The kernel must be one AMD supports for RDNA3.5: on Ubuntu 24.04 the HWE kernel 6.17.0-19.19~24.04.2 or later or the OEM kernel 6.14.0-1018 or later, Ubuntu 26.04's own kernel, and elsewhere Linux 6.18.4 or later (AMD's list).

On AMD, add AMD's ROCm 10 repository, then install the runtime with the BLAS kernels for your chip: amdrocm-blas10.0-gfx1151 on Strix Halo, amdrocm-blas10.0-gfx1150 on Strix Point. AMD serves the repository for Ubuntu 24.04 (ubuntu2404, as below) and 26.04 (ubuntu2604). The installer checks for all of this and prints the commands for your machine.

sudo curl -fsSL https://stable.repo.amd.com/rocm/gpg/packages.gpg -o /etc/apt/keyrings/amdrocm.asc
echo 'deb [arch=amd64 signed-by=/etc/apt/keyrings/amdrocm.asc] https://stable.repo.amd.com/rocm/core/packages/ubuntu2404 stable main' | sudo tee /etc/apt/sources.list.d/amdrocm.list
sudo apt update
sudo apt install amdrocm-blas10.0-gfx1151
sudo usermod -aG render "$USER"   # access to the GPU; log in again afterwards

Quickstart

1. Install

curl -LsSf https://basecompute.co/install.sh | sh

This installs the prebuilt engine for your platform and the basert CLI to ~/.basert and adds it to your PATH. Restart your shell afterward (or export PATH="$HOME/.basert:$PATH"). See Installation for building from source.

2. Pull a model

basert pull Qwen/Qwen3-4B    # download from HuggingFace + convert to .base
basert list                  # show installed models

Models are cached under ~/.cache/baseRT/models (override with $BASERT_MODELS_DIR). To pick a specific quant build, append :default-q8 to the id (or pass --variant default-q8).

3. Chat

basert chat Qwen/Qwen3-4B

4. Serve an OpenAI-compatible API

basert serve Qwen/Qwen3-4B --api-key "$(uuidgen)" --port 8080

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-4B","messages":[{"role":"user","content":"Hello!"}]}'

5. Or call it from your language

import baseRT

model = baseRT.Model("models/your-model.base")
print(model.generate_text("The capital of France is", max_tokens=64))

Node, Rust, and Swift bindings work the same way — see Bindings.

Use it as a coding agent

BaseRT runs as a local backend for the pi coding agent through the pi-basert extension, which auto-discovers whatever models your server is running.

# 1. Serve a model (a dense model is best for tool calling)
basert serve basecompute/gemma-4-E2B-it

# 2. Install the extension and launch pi
pi install git:github.com/basecompute/pi-basert
pi

Inside pi, run /model to pick a served model. See the pi-basert README for details.

The basert CLI

basert is a single front-end. Model-management commands run natively; runtime commands are forwarded to the matching basert-<cmd> engine binary.

Command	What it does
basert pull <id>	Download from HuggingFace (convert-on-pull) or the catalog
basert list	List installed models (--remote to include the catalog)
basert convert <src>	Convert a local GGUF / HF / MLX checkpoint to .base
basert chat <model>	Interactive chat
basert serve <model>	OpenAI-compatible HTTP server (--model repeats for multi-model)
basert complete <model>	One-shot completion (text / image / audio)
basert bench <model>	Throughput benchmark
basert inspect <model>	Dump a .base header + tensor inventory
basert sign / verify / keygen	ed25519 signing of .base bundles

Full flag reference: CLI reference.

What's in this repo

Path	Contents
base-convert/	The basert CLI (Rust): model hub, converter, launcher — plus the .base format and quantization specs and the generic quant profiles.
include/baseRT/	The stable public C API (baseRT.h, types.h).
bindings/	Python, Node, Rust, and Swift.
benchmarks/	Scripts and reference results.
docs/	The documentation site.

This repository — the CLI, .base format, public headers, and bindings — is licensed under Apache-2.0. The engine ships as a separate prebuilt binary under its own license; see Engine releases.

Links

• Documentation · Server API · .base format
• Security policy · Contributing

License

Apache-2.0 — see LICENSE and NOTICE. The prebuilt engine binary distributed via Releases is proprietary and ships under its own license.