[v1.0] Complete and enforce the comparative benchmark contract

[codeforester]

Goal

Make the benchmark delivered for #251 exercise the promised scenarios and compare all declared frameworks in CI.

Background

#251 required cold and warm invocation, JSON mode, diagnostics, nested commands, and a distinction between parser cost and lifecycle cost. Current scripts/benchmark_runtime.py measures only:

• a fresh interpreter import;
• a warm in-process no-op invocation.

The implementation has no JSON, diagnostics, nested-command, cold end-to-end invocation, or lifecycle-disabled baselines:

base-cli/scripts/benchmark_runtime.py

Lines 108 to 203 in 1df8bc4

	def _measure_import(iterations: int, framework: str = "base-cli") -> list[float]:
	package_root = Path(__file__).resolve().parents[1] / "lib" / "python"
	environment = dict(os.environ)
	existing_path = environment.get("PYTHONPATH")
	environment["PYTHONPATH"] = f"{package_root}{os.pathsep}{existing_path}" if existing_path else str(package_root)
	samples: list[float] = []
	for _ in range(iterations):
	started = time.perf_counter_ns()
	subprocess.run(
	[sys.executable, "-c", f"import {framework.replace('-', '_')}"],
	check=True,
	env=environment,
	stdout=subprocess.DEVNULL,
	stderr=subprocess.PIPE,
	)
	samples.append(_elapsed_ms(started))
	return samples
	def _measure_invocations(iterations: int, framework: str = "base-cli") -> list[float]:
	if framework == "click":
	return _measure_click_invocations(iterations)
	if framework == "typer":
	return _measure_typer_invocations(iterations)
	if framework == "cyclopts":
	return _measure_cyclopts_invocations(iterations)
	import base_cli
	from base_cli.testing import invoke
	app = base_cli.App(name="benchmark-runtime")
	@app.command()
	def main(ctx: base_cli.Context[Any, Any, Any]) -> None:
	del ctx
	samples: list[float] = []
	with tempfile.TemporaryDirectory(prefix="base-cli-benchmark-") as tmpdir:
	home = Path(tmpdir)
	for _ in range(iterations):
	started = time.perf_counter_ns()
	result = invoke(app, [], home=home)
	if result.exit_code != 0:
	raise RuntimeError(f"benchmark invocation failed: {result.output}")
	samples.append(_elapsed_ms(started))
	return samples
	def _measure_click_invocations(iterations: int) -> list[float]:
	import click
	from click.testing import CliRunner
	@click.command()
	def command() -> None:
	return None
	runner = CliRunner()
	return _measure_runner(iterations, lambda: runner.invoke(cast(Any, command), []).exit_code)
	def _measure_typer_invocations(iterations: int) -> list[float]:
	import typer
	from click.testing import CliRunner
	from typer.main import get_command
	app = typer.Typer()
	@app.command()
	def callback() -> None:
	return None
	command = get_command(app)
	runner = CliRunner()
	return _measure_runner(iterations, lambda: runner.invoke(cast(Any, command), []).exit_code)
	def _measure_cyclopts_invocations(iterations: int) -> list[float]:
	cyclopts = importlib.import_module("cyclopts")
	app = cyclopts.App()
	@app.default # type: ignore[untyped-decorator]
	def callback() -> None:
	return None
	return _measure_runner(iterations, lambda: cast(Any, app)([]))
	def _measure_runner(iterations: int, callback: Callable[[], Any]) -> list[float]:
	samples: list[float] = []
	for _ in range(iterations):
	started = time.perf_counter_ns()
	result = callback()
	if result not in (None, 0):
	raise RuntimeError(f"benchmark invocation failed with status {result!r}")
	samples.append(_elapsed_ms(started))
	return samples

Cyclopts is declared in the separate benchmark extra, while every workflow installs only [dev,typer,quality]. Current local and WSL output therefore reports cyclopts: unavailable and CI never exercises the four-framework comparison:

base-cli/pyproject.toml

Lines 76 to 86 in 1df8bc4

	quality = [
	"bandit>=1.7,<2",
	"coverage[toml]>=7.6,<8",
	"pip-audit>=2.7,<3",
	"pytest-cov>=5,<7",
	"ruff>=0.8,<1",
	]
	benchmark = [
	"cyclopts>=3,<4",
	]
	docs = [

base-cli/.github/workflows/tests.yml

Lines 36 to 39 in 1df8bc4

	- name: Install full validation dependencies
	run: python -m pip install ".[dev,typer,quality]"
	- name: Run authoritative validation gate
	run: ./tests/full_validate.sh

Only base-cli's very broad absolute budgets are enforced; comparative drift is informational.

Scope

• Implement the complete scenario matrix accepted in [v1.0] Benchmark lifecycle overhead against Click, Typer, and Cyclopts #251.
• Install the benchmark extra in the authoritative benchmark job.
• Separate interpreter/parser cost, base-cli lifecycle cost, persistence cost, and optional feature cost.
• Publish versioned machine-readable results with environment metadata.
• Define which absolute and relative regressions block CI.

Acceptance criteria

• CI measures Click, Typer, Cyclopts, and base-cli; an unavailable comparator fails the benchmark setup.
• Results include cold and warm no-op, nested command, JSON success/error, diagnostics/logging, and persistence-disabled/enabled cases.
• Equivalent command semantics are documented and reviewed.
• Thresholds detect a realistic regression and are calibrated per supported platform class.
• The generated/public dashboard distinguishes lifecycle value from parser overhead.

Validation

Add benchmark self-tests, run enough hosted samples to characterize variance, and retain a dated baseline artifact.

Non-goals

Do not use download counts or synthetic benchmark wins as adoption evidence.

Project fields

• Status: Backlog
• Priority: P2
• Area: CI
• Initiative: Adoption Polish
• Size: M

Ownership

• Assignee: @codeforester