Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ docker version
cargo +1.98.0 fetch --locked
cargo +1.98.0 build --locked -p control_plane -p data_plane
cargo +1.98.0 build --locked -p control_plane \
--example calibration_candidates --example inspect_physical_dag \
--example calibration_candidates --example compile_workload_artifact \
--example compile_clickhouse_workload
target/debug/data_plane --help
mkdir -p target/readme-evidence
Expand All @@ -278,7 +278,7 @@ mkdir -p target/readme-evidence
The executable is `target/debug/data_plane`, or `target/release/data_plane` if
you build with `--release`. Access to the pinned Git dependencies is required.

### 3. Export candidates and inspect the ordinary selected plan
### 3. Export candidates and select a deployable plan by complete cost

```bash
target/debug/examples/calibration_candidates \
Expand All @@ -287,10 +287,10 @@ target/debug/examples/calibration_candidates \
jq '.candidates[] | {candidate_index, unavailable_reason}' \
target/readme-evidence/candidates.json

target/debug/examples/inspect_physical_dag \
docs/examples/asapquery-compatibility-demo-snapshot.json \
target/debug/examples/compile_workload_artifact \
"$ASAPQUERY_PLANNING_SNAPSHOT" \
> target/readme-evidence/selected.json
jq '.purpose' target/readme-evidence/selected.json
jq '.cost_comparison' target/readme-evidence/selected.json
jq '.install_request.summary_catalog' target/readme-evidence/selected.json
jq '.install_request.precompute_plan | {materializations, executable_dags}' \
target/readme-evidence/selected.json
Expand All @@ -299,13 +299,13 @@ jq '.install_request.precompute_plan.schemas[] | {materialization, schema_id}' \
target/readme-evidence/selected.json
```

Expect `inspection_only`, catalog/plan objects and explicit candidate rejection
reasons where unsupported. One verified demo export contained five candidate
entries (one installable), five selected materializations and six query entries;
these are inspection evidence, not a permanent optimizer-count contract.
Materialization IDs are definitions, not physical SIDs. Demo costs are not
measurements. The [E2E walkthrough](docs/evaluation/e2e-physical-dag.md) explains
ERP evidence and the version-2 measured-cost workflow.
Candidate discovery accepts the checked-in unquoted templates. Deployment and
selected-plan inspection require `ASAPQUERY_PLANNING_SNAPSHOT` to point to a
snapshot with complete, valid workload cost evidence. Prepare that input using
the [cost evidence workflow](docs/examples/workload-cost-evidence.md).
There is one snapshot compiler: it compares complete executable alternatives,
including exact fallback. Materialization IDs are definitions, not physical SIDs.
For `--metricsql`, collect quotes for the MetricsQL frontend.

## Prometheus runbook

Expand Down Expand Up @@ -342,7 +342,7 @@ done
curl -fsS http://127.0.0.1:19090/-/healthy

target/debug/data_plane --profile asapquery \
--planning-snapshot docs/examples/asapquery-compatibility-demo-snapshot.json \
--planning-snapshot "$ASAPQUERY_PLANNING_SNAPSHOT" \
--prometheus-server http://127.0.0.1:19090 \
--forward-unsupported-queries --http-port 19091 \
--output-dir target/readme-evidence/prometheus/runtime \
Expand Down Expand Up @@ -409,6 +409,7 @@ kill "$(cat target/readme-evidence/prometheus/backend.pid)"
docker rm -f asap-readme-prometheus asap-readme-pushgateway
cargo +1.98.0 test --locked -p data_plane --test asapquery_compatibility_process_e2e \
collector_free_profile_serves_complete_matrix_and_falls_back_exactly -- --exact
export ASAPQUERY_PLANNING_SNAPSHOT=/absolute/path/priced-snapshot.json
./scripts/e2e.sh asapquery-demo
```

Expand Down Expand Up @@ -467,8 +468,8 @@ kill "$(cat target/readme-evidence/victoriametrics/backend.pid)"
To inspect supported MetricsQL planning independently:

```bash
target/debug/examples/inspect_physical_dag \
docs/examples/asapquery-compatibility-demo-snapshot.json --metricsql \
target/debug/examples/compile_workload_artifact \
"$ASAPQUERY_PLANNING_SNAPSHOT" --metricsql \
> target/readme-evidence/victoriametrics/selected.json
jq '.install_request.query_plan.entries' \
target/readme-evidence/victoriametrics/selected.json
Expand Down
2 changes: 1 addition & 1 deletion control_plane/docs/candidate-physical-explain.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Physical identity combines the logical/mask alternative with existing materializ

The read-only `/api/v1/physical-plan/cost-manifests` and MetricsQL equivalent retain their default manifest-array response. Add `"explain": true` to the existing request to receive `{ "manifests": [...], "alternatives": [...], "logical_selection": [...] }`. Failed alternatives remain alongside usable manifests. When none can bind or be completely priced, the error retains an `all_infeasible` report and every accumulated alternative rather than only a generic message.

Snapshot compilation exposes the same logical trace on `PhysicalPlan`; `inspect_physical_dag` prints it outside the install request. Compile-and-publish returns the trace without adding it to executable wire DTOs. SQL's existing selection trace gains the same semantic candidate/root identities.
Snapshot compilation exposes the same logical trace on `PhysicalPlan`; `compile_workload_artifact` prints it outside the install request. Compile-and-publish returns the trace without adding it to executable wire DTOs. SQL's existing selection trace gains the same semantic candidate/root identities.

These are bounded explanations: they cover the actual Planner search and the existing physical materialization/exact inventory, not every possible placement or resource-constrained cluster assignment. Missing numeric measurements remain missing. The next provider integration must occur before logical commitment and reuse Planner's provider/resource contracts.

Expand Down
23 changes: 15 additions & 8 deletions control_plane/examples/compile_workload_artifact.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,21 @@ use control_plane::physical::compiler::BackendLocalPlanningSnapshot;
use serde_json::json;

fn main() -> Result<(), Box<dyn std::error::Error>> {
let path = std::env::args()
.nth(1)
let mut args = std::env::args().skip(1);
let path = args
.next()
.ok_or("usage: compile_workload_artifact SNAPSHOT.json [--metricsql]")?;
let snapshot: BackendLocalPlanningSnapshot = serde_json::from_slice(&std::fs::read(path)?)?;
if snapshot.snapshot_version != 2 {
return Err(
"execution evaluation requires version 2 complete workload cost evidence".into(),
);
let metricsql = match args.next().as_deref() {
None => false,
Some("--metricsql") => true,
Some(_) => return Err("expected optional --metricsql".into()),
};
if args.next().is_some() {
return Err("unexpected arguments".into());
}
let snapshot: BackendLocalPlanningSnapshot = serde_json::from_slice(&std::fs::read(path)?)?;
let start = std::time::Instant::now();
let plan = if std::env::args().skip(2).any(|arg| arg == "--metricsql") {
let plan = if metricsql {
snapshot.compile_metricsql()?
} else {
snapshot.compile()?
Expand All @@ -29,6 +33,9 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
"planning_elapsed_ns": elapsed,
"envelope": plan.envelope,
"cost_comparison": comparison,
"logical_selection": plan.logical_selection,
"backend_revision": control_plane::physical::compiler::BACKEND_REVISION,
"planner_revision": control_plane::physical::compiler::PLANNER_REVISION,
"lifecycle_estimates": plan.lifecycle_estimates,
"install_request": {
"summary_catalog": plan.summary_catalog,
Expand Down
49 changes: 0 additions & 49 deletions control_plane/examples/inspect_physical_dag.rs

This file was deleted.

6 changes: 5 additions & 1 deletion control_plane/src/backend_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -601,7 +601,11 @@ mod tests {
"../../docs/examples/asapquery-planning-snapshot.json"
))
.unwrap();
let publication = snapshot.compile().unwrap().publication().unwrap();
let publication = crate::physical::compiler::tests::quoted_snapshot(snapshot, false)
.compile()
.unwrap()
.publication()
.unwrap();
let hits: StdArc<Mutex<Vec<serde_json::Value>>> = StdArc::new(Mutex::new(Vec::new()));
let route_hits = hits.clone();
let app = Router::new().route(
Expand Down
Loading
Loading