Shared-API parity, TAVE tx-window, Datalevin-style query opts, benches - #5

Open

tiensonqin wants to merge 124 commits into

mainfrom

logseq/shared-api-parity-fe5d

Commits on Aug 26, 2026

Add Cloud Agent environment with OCaml 5.5

cursoragent
and
tiensonqin
committed
2180e8e
Add LMDB storage backend and SQLite comparison benchmark

cursoragent
and
tiensonqin
committed
2de5d74
Document non-PSS LMDB design with Scheme A index types

cursoragent
and
tiensonqin
committed
cdc61f4
Remove file storage and transaction tail from LMDB storage

cursoragent
and
tiensonqin
committed
2f53f7d
Fix LMDB index ordering, storage sync, and restore semantics

cursoragent
and
tiensonqin
committed
49de381
Fix LMDB overlay immutability and storage sync

cursoragent
and
tiensonqin
committed
1eeda4a

Commits on Aug 27, 2026

Batch LMDB index writes and cursor range reads (Datalevin-style)

cursoragent
and
tiensonqin
committed
56c365b
Optimize LMDB init with bulk overlay indexes and attr caches

cursoragent
and
tiensonqin
committed
6865015
Optimize bulk LMDB slices and AVET entity-id query path

cursoragent
and
tiensonqin
committed
ea32d01
Fix query fast path and attr grouping for LMDB benchmarks

cursoragent
and
tiensonqin
committed
635b2e8
Add dbval-aligned temporal db API (phase 1)

cursoragent
and
tiensonqin
committed
90dfbd7
Remove LMDB overlay: append-only index with tx-filter reads

cursoragent
and
tiensonqin
committed
6820cd1
Add session pending_datoms staging for fast transact without storage

cursoragent
and
tiensonqin
committed
5e2e430
Store: append-only delta index sync instead of full rewrite

cursoragent
and
tiensonqin
committed
3487d7a
Add Datahike-style purge (excise) API for permanent datom removal

cursoragent
and
tiensonqin
committed
04d0a3f
Expand tx_history tests for dbval/Datahike temporal API coverage

cursoragent
and
tiensonqin
committed
ab4c7ec
Make storage backends pluggable with string kinds and separate packages

cursoragent
and
tiensonqin
committed
2ebfa6b
Fix tx_history since+history test to use exclusive since tx0

cursoragent
and
tiensonqin
committed
2c04b6d
Remove sync_merged overlay path and fix temporal fold_datoms

cursoragent
and
tiensonqin
committed
e1457b5
Add Datahike benchmark parity and avet predicate range fast path

cursoragent
and
tiensonqin
committed
745517b
Add query planner ADR, fix q2 same-entity joins, expand parity tests

cursoragent
and
tiensonqin
committed
da87d76
Fix AVET value-range scans to seek and stop on encoded bounds

cursoragent
and
tiensonqin
committed
e82959c
Optimize AVET range scans and label benchmark storage backends

cursoragent
and
tiensonqin
committed
ac1a172
Align benchmark storage with in-memory nosync LMDB profile

cursoragent
and
tiensonqin
committed
f80411a
Speed up same-entity join queries and fast benchmark defaults

cursoragent
and
tiensonqin
committed
12fbd67
Add single-query filter to Datahike compare benchmarks

cursoragent
and
tiensonqin
committed
73f07b3
Fix same-entity join fast path for mixed constant/value queries

cursoragent
and
tiensonqin
committed
4fc6266
Add fast paths for q5, q-or-join, and q-not-join benchmarks

cursoragent
and
tiensonqin
committed
e8c96a4
Fix attr-only index slice and fold_slice prefix bounds

cursoragent
and
tiensonqin
committed
5d291d9
Add fast path for non-recursive follow rule query (q-rule)

cursoragent
and
tiensonqin
committed
70b3504
Document Datahike vs OCaml query gaps; fast-path bind_var equality

cursoragent
and
tiensonqin
committed
5dc1a05
Simplify query paths to align with Datahike structure

cursoragent
and
tiensonqin
committed
c12e8e7
Use per-entity attr lookup for single-value same-entity queries

cursoragent
and
tiensonqin
committed
49c3356
Align multi-value same-entity queries with Datahike entity-group pattern

cursoragent
and
tiensonqin
committed
5819867
Speed up same-entity value lookups with direct Aevt array search

cursoragent
and
tiensonqin
committed
79fd8c7
Fix LMDB read txn, AVET seek/range, and unify index codec with SQLite

cursoragent
and
tiensonqin
committed
df8d464
Remove legacy PSS Logseq SQLite examples and fix unique checks mid-tx

cursoragent
and
tiensonqin
committed
a60563b
Disable benchmark/perf gates and lein-dependent cross-runtime tests

cursoragent
and
tiensonqin
committed
bf64449
Add Datahike category parity tests and fix as_of query visibility

cursoragent
and
tiensonqin
committed
f5d3a42
Strengthen Datahike result identity and speed qpred2 range scans

cursoragent
and
tiensonqin
committed
f1a2a2d
Speed same-entity and not/not-join query fast paths

cursoragent
and
tiensonqin
committed
1d07749
Speed same-entity/not paths and use shared-API naming

cursoragent
and
tiensonqin
committed
c96fbbe
Add rules-wide-7x3 golden and dense gather for multi-constant joins

cursoragent
and
tiensonqin
committed
83ed53a
Tighten q-5-merge dense gather and bench result consumption

cursoragent
and
tiensonqin
committed
d1e6056
Add query plan IR with cost-ordered where clauses

cursoragent
and
tiensonqin
committed
9071bcd
Apply temporal views to seek/rseek with streaming filter

cursoragent
and
tiensonqin
committed
9b07011
Stabilize attr caches under append-only history views

cursoragent
and
tiensonqin
committed
0d4719f
Add Instant as-of, history purge-before, and LMDB GC sync

cursoragent
and
tiensonqin
committed
cc549a8
Inline non-recursive rules into the relation query path

cursoragent
and
tiensonqin
committed
b000d19
Do not evaluate or-join via plain-or relation path

cursoragent
and
tiensonqin
committed
fe77d25
Harden Cloud Agent OCaml 5.5 env and fix install build

cursoragent
and
tiensonqin
committed
3c53419
Fix js_of_ocaml benches to use the jsoo package

cursoragent
and
tiensonqin
committed
62c1ff2
Update query planner docs for bench_ocaml_js artifact path

cursoragent
and
tiensonqin
committed
34ac9b0
Add LMDB vs SQLite compare for full shared query suite

cursoragent
and
tiensonqin
committed
a682468
Make cloud-agent-install self-bootstrap opam and OCaml 5.5

cursoragent
and
tiensonqin
committed
1c432ed
Add plan for SQLite index without LMDB dependency

cursoragent
and
tiensonqin
committed
4608921
Add SQLite index ported from LMDB index

cursoragent
and
tiensonqin
committed
c924f89
Implement SQLite Share_index_db without LMDB mirror

cursoragent
and
tiensonqin
committed
fbc2236
Harden SQLite Share path (Phase 4) and rename index codec

cursoragent
and
tiensonqin
committed
5b99ef9
Add LMDB vs SQLite operator choice documentation

cursoragent
and
tiensonqin
committed
4fcc6d6
Make LMDB/SQLite compare benches disk-only with durable flush

cursoragent
and
tiensonqin
committed
0c820ce
Add LMDB vs SQLite index-scan microbench and LMDB reopen

cursoragent
and
tiensonqin
committed
841852c
Fix index-scan seek case to use seek_datoms

cursoragent
and
tiensonqin
committed
e7d209b
Add 1M size to index-scan microbench

cursoragent
and
tiensonqin
committed
626dc8f
Use 200k and 500k as index-scan default sizes

cursoragent
and
tiensonqin
committed
3547876
Remove Datascript.q simple_* shape-gate bypasses

cursoragent
and
tiensonqin
committed
b5a5ab5
Align query planner with Datahike logical IR

cursoragent
and
tiensonqin
committed
9c81255
Document Datahike-aligned query planner and fallback

cursoragent
and
tiensonqin
committed
cef4841

Commits on Aug 28, 2026

Speed same-entity lookups, NotJoin fold, and OrJoin relations

cursoragent
and
tiensonqin
committed
48c3394
Speed queries with dense AEVT gather, hash-probe joins, and AVET rows

cursoragent
and
tiensonqin
committed
ee99b2f
Tighten dense AEVT gather specialized row emit path

cursoragent
and
tiensonqin
committed
21e59fe
Use AVET entity-id arrays and unrolled dense row emit

cursoragent
and
tiensonqin
committed
9222bd4
Speed bench queries with AVET OrJoin, lazy bitsets, and AEVT scans

cursoragent
and
tiensonqin
committed
891ebfb
Skip Query_plan.compile on fused fast-path hits

cursoragent
and
tiensonqin
committed
e8b5575
Add AGENTS.md rule: use debug logs to find root cause before fixing

cursoragent
and
tiensonqin
committed
556df05
Speed relation eval with rule, not, and dense gather fast paths

cursoragent
and
tiensonqin
committed
326670c
Fix wildcard pull slow path and add gated query debug logging

cursoragent
and
tiensonqin
committed
24eb404
Add early aligned-constant dense gather for q2-shaped queries

cursoragent
and
tiensonqin
committed
2b699f6
Use const-first AEVT alignment for q-5-merge dense gather

cursoragent
and
tiensonqin
committed
1272379
Add lightweight eval_relation_rows fast path for bench queries

cursoragent
and
tiensonqin
committed
fe8abcc
Add Datahike vs OCaml query pipeline comparison doc

cursoragent
and
tiensonqin
committed
69794cc
Add query_exec layer and simplify query_where fallback

cursoragent
and
tiensonqin
committed
ea12362
Align entity-group execute with Datahike sorted-merge

cursoragent
and
tiensonqin
committed
a571c3e
Tighten Datahike entity-group execute hot paths

cursoragent
and
tiensonqin
committed
f6875ae
Cache last AVET entity-id array in query_exec hot path

cursoragent
and
tiensonqin
committed
2f34a8e
Cache resolved entity-group kernels on the Datahike execute path

cursoragent
and
tiensonqin
committed
f4e845b
Fix compile_plan crash on function-valued query clauses

cursoragent
and
tiensonqin
committed
57b5528
Use Platform.now_seconds in query debug_log

cursoragent
and
tiensonqin
committed
2fe723e
Add Query_exec path probes and fused/fallback parity tests

cursoragent
and
tiensonqin
committed
c15b0f4
Add storage RSS bench for memory/LMDB/SQLite vs main

cursoragent
and
tiensonqin
committed
d0e4bcb

Commits on Aug 29, 2026

Add design for TEAV tx-window index narrowing

cursoragent
and
tiensonqin
committed
3287ba1
Expand TEAV design: index choice, since/history layering, Mode B rules

cursoragent
and
tiensonqin
committed
ba493fa
Document Logseq production query survey vs TAVE

cursoragent
and
tiensonqin
committed
684387b
Require transparent planner path selection; no caller API change

cursoragent
and
tiensonqin
committed
e0e8f87
Implement TAVE rolling index with 30-day retention

cursoragent
and
tiensonqin
committed
43560c0
Add TAVE storage-ratio microbench

cursoragent
and
tiensonqin
committed
d4ba8d1
Add Logseq-shaped query microbench across storages

cursoragent
and
tiensonqin
committed
feae79d
Fix Logseq recent-pages path and LMDB rslice full-scan

cursoragent
and
tiensonqin
committed
6ec4d8f
Bench Logseq queries via nbb+SQLite across three runtimes

cursoragent
and
tiensonqin
committed
12e6e8a
Drop unused path require from nbb CLJS bench

cursoragent
and
tiensonqin
committed
fe13ff4
Make shared SQLite bench build on origin/main

cursoragent
and
tiensonqin
committed
dd455de
Speed up reverse attr scans for Logseq-style recent queries

cursoragent
and
tiensonqin
committed
4c1f58a
Use nbb-logseq PSS storage matching Logseq DB path

cursoragent
and
tiensonqin
committed
e835bbe
Add 5k Logseq shared query 3-way bench results

cursoragent
and
tiensonqin
committed
8498faf
Make Share SQLite and LMDB index scans lazy like upstream

cursoragent
and
tiensonqin
committed
22d82bf
Cover lazy index scans on memory, LMDB, and SQLite

cursoragent
and
tiensonqin
committed
4dd41db
Speed Share SQLite streams and entity attr lookups

cursoragent
and
tiensonqin
committed
b7a59fd
Refresh 5k 3-way Logseq query bench after SQLite/entity fixes

cursoragent
and
tiensonqin
committed
d3958fd
Restore AVET value types and compare Logseq results as EDN

cursoragent
and
tiensonqin
committed
7e30324
Add Datahike and Datalevin Logseq query benches with EDN equality

cursoragent
and
tiensonqin
committed
1a0db92
Cache AVET point lookups and batch entity forward attrs

cursoragent
and
tiensonqin
committed
1fe5de2
Cache EAVT entity slices and entity forward attrs by basis

cursoragent
and
tiensonqin
committed
ea1234b
Warm avet_by_attr for attr-only AVET scans on Share

cursoragent
and
tiensonqin
committed
b0cd9eb
Beat Datahike on all Logseq queries; EDN string equality

cursoragent
and
tiensonqin
committed
550fe6f
Ignore Clojure .cpcache under bench/external

cursoragent
and
tiensonqin
committed
9694c50
Fix Datalevin durable LMDB bench open/cleanup

cursoragent
and
tiensonqin
committed
b0ec910
Add shared people-query 3-way bench vs Datahike and Datalevin

cursoragent
and
tiensonqin
committed
8d68abf
Fix Share SQLite bulk build: O(n) tx_data + batched index writes

cursoragent
and
tiensonqin
committed
9f2a3a0
Push comparison predicates into AVET range scans on fused execute

cursoragent
and
tiensonqin
committed
c41a40a

Commits on Aug 30, 2026

Generalize query execute; add Datalevin-style result cache + debug

cursoragent
and
tiensonqin
committed
31066af
Fix Query_exec hash_join to multi-match; prefer nocache in 3-way

cursoragent
and
tiensonqin
committed
e3def84
Fix query cache keying, bulk card-one re-assert, Int AVET bounds

cursoragent
and
tiensonqin
committed
e7b3c35
Fix filtered DB dense AEVT leak; correct cache/Int tests

cursoragent
and
tiensonqin
committed
f2abdd2
Disable query result cache during memory-bench CI gate

cursoragent
and
tiensonqin
committed
98deacf