feat: adopt sketch-core runtime modules into existing sketches/ layout - #36
Merged
Conversation
This was referenced May 1, 2026
zzylol
force-pushed
the
refactor/adopt-sketch-core-modules
branch
from
May 1, 2026 19:18
63051a8 to
e473ccc
Compare
zzylol
changed the title
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 1, 2026
Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery
that referenced
this pull request
May 1, 2026
Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
refactor/adopt-sketch-core-modules
branch
4 times, most recently
from
May 1, 2026 20:34
8db5e7f to
943817e
Compare
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 1, 2026
Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery
that referenced
this pull request
May 1, 2026
Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
refactor/adopt-sketch-core-modules
branch
2 times, most recently
from
May 1, 2026 20:53
d5015ed to
0338b3b
Compare
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 1, 2026
Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery
that referenced
this pull request
May 1, 2026
Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
refactor/adopt-sketch-core-modules
branch
from
May 1, 2026 21:16
0338b3b to
d84ff15
Compare
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 1, 2026
Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery
that referenced
this pull request
May 1, 2026
Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 0.5 of the edge-framework design doc (#213) — retire the standalone `sketch-core` crate and consolidate ASAP's runtime sketches into asap_sketchlib's existing `src/sketches/` layout (no separate `asap/` namespace). For each sketch type that already had a home in `src/sketches/`, the wire-format-aligned variant (with public bucket-array fields, sparse multi-cell delta types, `apply_delta`, msgpack ser/de, and `from_raw` constructors — the surface needed for proto decode + delta-transmission reconstitution; paper §6.2 B3 / B4) is appended to the existing file under a clearly-marked section header. The existing in-process variants (`DDSketch`, `CountMin<S, Mode, H>`, `Count`, `HyperLogLog`, `KLL<T>`, `CMSHeap`) are unchanged. Single-file homes (existing → +appended wire-format types): - `src/sketches/ddsketch.rs` +DdSketch + DdSketchDelta - `src/sketches/countmin.rs` +CountMinSketch + CountMinDelta - `src/sketches/count.rs` +CountSketch + CountSketchDelta - `src/sketches/hll.rs` +HllSketch + HllSketchDelta + HllVariant - `src/sketches/kll.rs` +KllSketch + KllSketchData + KllBackend - `src/sketches/cms_heap.rs` +CountMinSketchWithHeap + CmsHeapItem New sibling files (no existing home — follow the one-file-per-sketch convention): - `src/sketches/hydra_kll.rs` (HydraKllSketch) - `src/sketches/set_aggregator.rs` (SetAggregator) - `src/sketches/delta_set_aggregator.rs` - `src/sketches/count_min_sketchlib.rs` (FFI helper) - `src/sketches/count_min_with_heap_sketchlib.rs` (FFI helper) - `src/sketches/kll_sketchlib.rs` (FFI helper) Top-level: - `src/asap_runtime.rs` — `ImplMode` (Legacy | Sketchlib) backend selector, used by the runtime CountMin / KLL / CountMinWithHeap dispatch. Naming-conflict resolutions: - `octo_delta::HllDelta` already exists (single-register, octo path), so the wire-format multi-register delta is renamed to `HllSketchDelta`. - `common::input::HeapItem` already exists, so the wire-format CMSHeap item is renamed to `CmsHeapItem`. Other notes: - `SketchInput` references in the wrappers (sketch-core was pinned against an older asap_sketchlib) renamed to current `DataInput`; KLL update path switched from `&DataInput::F64(v)` to `&v` to match the typed `KLL<T>` generic API. - `clap::ValueEnum` derive on `ImplMode` is gated behind a new `asap-cli` feature so non-CLI consumers don't pay for `clap`. The query engine enables this feature. - New deps added to Cargo.toml: `xxhash-rust` (xxh32), `dsrs` (datasketches-rs), `clap` (optional, behind `asap-cli`), `ctor` (dev-dep for the legacy-mode test initializer that previously lived in sketch-core's lib.rs). Tests: cargo test --lib → 376 passed, 0 failed (308 existing + 68 ported wire-format tests). No regressions. Followup: ASAPQuery-backend PR #73 and ASAPQuery PR #307 will switch consumer imports from `sketch_core::*` and the older `asap_sketchlib::asap::*` to the new `asap_sketchlib::sketches::*` paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
refactor/adopt-sketch-core-modules
branch
from
May 1, 2026 21:43
d84ff15 to
5f990ae
Compare
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 2, 2026
* refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery
that referenced
this pull request
May 2, 2026
* refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 2, 2026
* docs(todo): sync after 2026-05-01 query-path-closes-the-loop work (#72) Capture this session's merged backend work — #70 (tonic OTLP gRPC max_decoding_message_size bumped to 64 MiB) and #71 (store range_query_into overlap filter + engine closest-pane selection + Prometheus-adapter precompute_window annotation) — at the top of TODO.md so the runtime-warm-tier-actually-works claim is testable from the doc. Cited the matching collector-side PRs (ASAPCollector#210 + #211) in the companion-changes note so future readers can see both halves of the wire fix. Added one new entry under "Known reconciliation gap": `IngestState.sketch_snapshots` is RAM-only, so backend restarts break delta ingest until the agent restarts too. Same item is mirrored in the collector's PROGRESS.md follow-up list — fix on either side closes the gap. `_Last updated_` set to 2026-05-01. Docs only; no code changes. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: retire sketch-core mirror (#73) * refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: align CountSketchDelta consumer with sketchlib-go wire format Track the additive `hh_keys` field on `asap_sketchlib::CountSketchDelta` so the proto delta path constructs the type with all fields filled in. Sends an empty `hh_keys` for now: the vendored Rust proto bindings in `asap_otel_proto::sketchlib::v1` haven't been regenerated against the latest `.proto` (which carries `hh_keys` on the Go side). The TopK rebuild on the proto-delta path will fire once those bindings sync; the sketchlib-go-aligned semantics are already in place underneath. Bumps the asap_sketchlib git dep to `refactor/wire-format-align-go` (see asap_sketchlib PR #37). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
zzylol
added a commit
that referenced
this pull request
May 5, 2026
PR #36 silently reverted PR #37's hh_keys field on CountSketchDelta when its merge landed. Restore on top of the post-#39 module names (src/sketches/countsketch.rs). Mirrors sketchlib-go's Delta.HHKeys / apply-delta heavy-hitter rebuild path. Unblocks ASAPQuery-backend's PR-#74 consumer to pin asap_sketchlib to main, which in turn unblocks ASAPCollector Phase 3 step 3 (backend consumes asap-precompute-rs). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
to ProjectASAP/ASAPQuery-backend
that referenced
this pull request
May 5, 2026
* refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: align CountSketchDelta consumer with sketchlib-go wire format Track the additive `hh_keys` field on `asap_sketchlib::CountSketchDelta` so the proto delta path constructs the type with all fields filled in. Sends an empty `hh_keys` for now: the vendored Rust proto bindings in `asap_otel_proto::sketchlib::v1` haven't been regenerated against the latest `.proto` (which carries `hh_keys` on the Go side). The TopK rebuild on the proto-delta path will fire once those bindings sync; the sketchlib-go-aligned semantics are already in place underneath. Bumps the asap_sketchlib git dep to `refactor/wire-format-align-go` (see asap_sketchlib PR #37). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(asap_sketchlib): bump pin past PR #39 module renames Three asap_sketchlib modules were renamed in upstream PR #39: - sketches::countmin → sketches::countminsketch - sketches::count → sketches::countsketch - sketches::cms_heap → sketches::countminsketch_topk Backend consumers updated. Cargo.toml pin moved from refactor/wire-format-align-go branch to main (which now also has hh_keys restoration via PR #42 and DDSketch + KLL byte parity via #40 + #41). Unblocks ASAPCollector Phase 3 step 3 (backend consumes asap-precompute-rs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
pushed a commit
that referenced
this pull request
May 12, 2026
…dule and legacy msgpack shims (#46) * refactor: introduce wrapper + message_pack_format top-level modules Move PR #36–#45 wire-format-aligned variants out of src/sketches/ into a new src/wrapper/ module so the in-process algorithms and the sketchlib-go-aligned producer types live in separate trees. Add a src/message_pack_format.rs placeholder noting that asap_sketchlib (Rust) and sketchlib-go each maintain their own MessagePack representation, byte-compatible at the envelope level. Files moved: - src/wrapper/{ddsketch,countminsketch,countsketch,hll,kll,countminsketch_topk}.rs carry the wire-format types appended after the per-file banner in PR #36+. - src/wrapper/{hydra_kll,set_aggregator,delta_set_aggregator}.rs are whole-file moves (they were entirely wire-format). CountL2HH stays in src/sketches/countsketch_topk.rs — it's an in-process top-k algorithm used by sketch_framework, not a wire-format type. No semantic changes; cargo test --lib --all-features still 447 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: clean unused imports, apply fmt + clippy in wrappers Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(hashspec): drop broken intra-doc links in module-level docs The `[`CANONICAL_HASH_SEED_TABLE`]` and `[`CANONICAL_HASH_SEED`]` shortcut links in the module-level `//!` docs failed to resolve under `cargo doc --all-features`, breaking the Rust Docs CI workflow with `-D warnings`. Both items are still defined in this module; the doc text simply uses inline-code formatting now instead of intra-doc links so docgen succeeds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(message_pack_format): make module concrete with DTOs + codec trait Move all wire-format DTOs (WireFormat, CmsData, CountMinSketchWithHeapSerialized, HydraKllSketchData, KllSketchData, StringSet, DeltaResult) out of individual wrappers and into message_pack_format::dto as the single source of truth. Add a unified Error enum and a MessagePackCodec trait (static-dispatch, zero cost) that every wire-format-aligned wrapper implements. The legacy inherent serialize_msgpack / deserialize_msgpack methods are preserved with their original signatures as thin shims over the trait, so callers and FFI bindings need no changes. KllSketchData and DeltaResult remain pub via re-exports from their wrapper modules. Add tests/msgpack_compat.rs with round-trip and DTO-shape coverage for every type, plus #[ignore]-d slots reserved for cross-language golden-bytes fixtures from sketchlib-go. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(message_pack_format): drop broken intra-doc links in module docs `[`Error`]`, `[`MessagePackCodec`]`, `[`dto`]` shortcut links inside the module-level `//!` block failed to resolve under `cargo doc --all-features` even though the items are defined in the same module — likely a rustdoc quirk around `pub use` shadowing the shortcut-link namespace. Switch to inline-code formatting so docgen succeeds; the items are still discoverable via the usual rustdoc sidebar. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * update file structure: clear separation of wrapper and msgpack_pack_format * docs(message_pack_format): drop broken intra-doc links reintroduced in restructure The module restructure in 746e335 re-added `[`Error`]` / `[`MessagePackCodec`]` shortcut links in the `//!` block that fail to resolve under `cargo doc --all-features` (same rustdoc quirk fixed earlier in 64b6989). Switch back to inline-code formatting so docgen passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add wrapper and message_pack_format pages The docs predate the wrapper / message_pack_format split and didn't cover either module. Add dedicated pages for each and link them from index, project_overview, and library_map. Entry pages stay lean — details live on the dedicated pages so readers who don't need them aren't forced through them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(message_pack_format): split into native/ and portable/ Reorganize the ser/de module along an audience boundary so reviewers can tell at a glance whether a change touches the cross-language wire contract or only Rust-internal persistence. - portable/: cross-language wire format shared with sketchlib-go. The 9 existing per-algorithm files move here unchanged (only super:: imports rewritten to crate::message_pack_format::). Touching this directory is a protocol change. - native/: thin MessagePackCodec trait shims over the existing serialize_to_bytes / deserialize_from_bytes methods on the pure-Rust generic sketches in src/sketches/. Byte format is Rust-internal; Go never reads it. sketches/ and wrapper/ logic is unchanged. Existing public API methods (CountMin::serialize_to_bytes, CountMinSketch::serialize_msgpack, etc.) are preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(message_pack_format): unbreak CI and refresh docs for native/portable split Drop the outer `///` doc on `pub mod message_pack_format;` in lib.rs — it was merged with mod.rs's inner `//!` doc and forced intra-doc link resolution into the crate root, breaking `[`native`]`, `[`portable`]`, and `[`Error`]`. Also fix a stale `hydra_kll` path under `portable::` and pick up `cargo fmt` reorderings. Rewrite docs/message_pack_format.md to cover the new native/ vs portable/ layout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(wrapper): drop serialize_msgpack/deserialize_msgpack shims The wrapper sketch types each carried inherent serialize_msgpack / deserialize_msgpack methods that just forwarded to MessagePackCodec:: to_msgpack / from_msgpack with extra error-type adaptation. Now that MessagePackCodec is the canonical entry point, the shims add a second name for the same operation and fragment the error surface (RmpEncodeError vs Box<dyn Error> vs MsgPackError) for no benefit. Removes the inherent shims from CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, KllSketch, HydraKllSketch, and SetAggregator. Internal callers (aggregate_* helpers, tests) now use to_msgpack / from_msgpack directly. The delta_set_aggregator free functions are unchanged — they are not type-method shims and keep their pre-trait signature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(wrapper): drop hashspec bypass; reuse sketches FastPath math `wrapper::CountSketch::update`/`estimate` and `wrapper::CountMinSketch:: update`/`estimate` were each open-coding a Go-compatible hash → (col, sign) derivation via `common::hashspec` self-functions. The same packed bit-slicing math already lives in `MatrixHashType::Packed64` / `sketches::CountMin<.., FastPath>::insert_many`, and produces bit-identical envelope bytes against the existing Go golden fixtures. Changes: - `wrapper::CountSketch::{update,estimate}` use `MatrixHashType::Packed64 + DefaultXxHasher::hash64_seeded(0,..)` directly. No behavior change. - `wrapper::CountMinSketch` switches its backend from `RegularPath` to `FastPath` and routes update/estimate through `backend.insert_many` / `backend.estimate` — the wrapper no longer bypasses the backend. New parity probe in `tests/sketches_go_parity_probe.rs` locks in that `sketches::Count<Vector2D<i64>, FastPath, DefaultXxHasher>` and `sketches::CountMin<Vector2D<f64>, FastPath, DefaultXxHasher>` emit byte-identical SketchEnvelope encodings to the captured sketchlib-go goldens (1577-byte CountSketch, 8275-byte CMS Frequency-Only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: delete mod wrapper; wire DTOs live in message_pack_format The `wrapper/` module was a parallel set of "wire-format-aligned" sketch types added when the runtime sketches in `sketches/` were believed not to produce Go-byte-compatible output. The parity probes added in this branch (`tests/sketches_go_parity_probe.rs`) confirm `sketches::Count<Vector2D<i64>, FastPath>`, `sketches::CountMin<Vector2D<f64>, FastPath>`, `sketches::HyperLogLogImpl<ErtlMLE, P14>`, and `sketches::DDSketch` all emit byte-identical envelopes to `sketchlib-go` for the same goldens, so the parallel layer is redundant. Changes: - Move every type formerly under `src/wrapper/*.rs` into the corresponding `src/message_pack_format/portable/*.rs` file, keeping the wire DTO and runtime ops together. Internal tests come along. - Drop the `crate::wrapper` module entirely. - Re-export the runtime + wire DTO types at the crate root from their new home in `message_pack_format::portable::*`. - Add small public accessors to `sketches::DDSketch` (`alpha`/`sum`/`store_counts`/`store_offset`) and to `sketches::HyperLogLogImpl::registers_as_slice` so the parity probe can build wire envelopes directly from `sketches::*` without reaching into private fields. - Extend `tests/sketches_go_parity_probe.rs` with HLL + DDSketch probes (CountSketch + CountMin already there); all 4 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * sketches: expose KLL::k() accessor The compactor capacity `k` is private inside `sketches::KLL<T>` but needed by downstream consumers that build the Go-compatible `KllSketchData { k, sketch_bytes }` wire envelope around a directly- held `sketches::KLL<f64>`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): rustfmt + rustdoc; refresh docs for deleted wrapper module CI was red on fmt and cargo doc: - `cargo fmt --all -- --check` flagged unrelated formatting drift across lib.rs, kll.rs, countminsketch.rs, countsketch.rs, and msgpack_compat.rs — applied rustfmt. - `cargo doc -D warnings` failed on broken intra-doc links to `crate::wrapper` (the module was deleted in bfe8f37) and an unqualified `MatrixHashType::Packed64` link. Repointed the surviving rustdoc to `crate::message_pack_format::portable` / `crate::MatrixHashType`. Also removed the now-dead `Error::into_encode` helper that clippy/rustc warned about (it was the last caller of the serialize_msgpack shim deleted in f847b18). Doc refresh under docs/: - wrapper.md now describes the wire-format-aligned types in their new home under message_pack_format/portable/ instead of the removed src/wrapper/ tree. - index.md, library_map.md, project_overview.md, and message_pack_format.md updated to drop src/wrapper/ references and reflect that DTO + runtime ops are co-located per-algorithm. - Test file headers (msgpack_compat.rs, sketches_go_parity_probe.rs) no longer reference the deleted wrapper module. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: drop wrapper.md; fold useful bits into message_pack_format.md The wrapper.md page documented a module that no longer exists. The only content not already in message_pack_format.md was the in-process (generic sketches) vs wire-aligned (portable) selection guidance and the per-file type listing — both folded into the `portable/` section of message_pack_format.md. Updated index.md, project_overview.md, and library_map.md to drop their wrapper.md links. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 0.5 of the edge-framework design doc (#213) — retire the standalone
sketch-corecrate and consolidate ASAP's runtime sketches into asap_sketchlib's existingsrc/sketches/layout (no separateasap/namespace).For each sketch that already had a home in
src/sketches/, the wire-format-aligned variant (with public bucket-array fields, sparse multi-cell delta types,apply_delta, msgpack ser/de,from_raw) is appended to the existing file under a clear section header. The existing in-process variants (DDSketch,CountMin<S, Mode, H>,Count,HyperLogLog,KLL<T>,CMSHeap) are unchanged.Changes
Single-file homes (existing + appended wire-format types):
src/sketches/ddsketch.rsDdSketch,DdSketchDeltasrc/sketches/countmin.rsCountMinSketch,CountMinDeltasrc/sketches/count.rsCountSketch,CountSketchDeltasrc/sketches/hll.rsHllSketch,HllSketchDelta,HllVariantsrc/sketches/kll.rsKllSketch,KllSketchData,KllBackendsrc/sketches/cms_heap.rsCountMinSketchWithHeap,CmsHeapItemNew sibling files (no existing home):
hydra_kll.rs,set_aggregator.rs,delta_set_aggregator.rscount_min_sketchlib.rs,count_min_with_heap_sketchlib.rs,kll_sketchlib.rs(FFI helpers)Top-level:
src/asap_runtime.rs—ImplMode(Legacy | Sketchlib) selector for the runtime CountMin / KLL / CountMinWithHeap dispatch.Naming-conflict resolutions
octo_delta::HllDeltaalready exists (single-register, octo path), so the wire-format multi-register delta is renamed toHllSketchDelta.common::input::HeapItemalready exists, so the wire-format CMSHeap item is renamed toCmsHeapItem.Tests
```
cargo test --lib → 376 passed, 0 failed
```
(308 existing + 68 ported wire-format tests; no regressions.)
Followups