Feat(clp) added the capbility to compress unstructured log and analyze any logs with logtype dump

.gitignore

@@ -10,3 +10,4 @@ plugins/clp/.clp-core/
 
 # Common local/runtime noise.
 .DS_Store
+plugins/clp/bin/clp-s

mongodb-skills-eval/logtype-insights-report.md

@@ -0,0 +1,78 @@
+# Logtype Insights Report (App-Agnostic, Logtype-Baseline)
+
+**Skill:** `logtype-insights` (schema discovery → `stats.logtypes` baseline → classify → cache → targeted queries)
+**Input:** same 7-file / 1.88 GB subset → 9.39 MB archive, 4,986,512 records (after the wrapper null-delimiter fix).
+**Executor note:** Subagent delegation unavailable; the two phases (classify, insight) were executed inline. The classification cache round-trip was exercised for real.
+
+## 1. Summary
+
+- **Total records:** 4,986,512; span 2023-03-21T23:34:54 → 2023-03-22T04:47:54 (~5h13m); MongoDB 6.0.5 / hostb22.
+- **Severity:** D3 3.68M, D2 720k, I 242k, D4 167k, D5 150k, D1 25k, W 3, E 0, F 0.
+- **Top logger/component:** STORAGE (3.22M).
+
+## 2. Logtype Baseline — the spine
+
+- **Distinct templates: 199** (via `stats.logtypes`, reads the dictionary, not every record — instant).
+- **Schema discovered** from one sample record: fields `t`, `s`, `c`, `id`, `ctx`, `msg`, `attr` → `timestamp=t.$date`, `severity=s`, `logger=c`, `message=msg`, payload leaves `attr.durationMillis/attr.ns/attr.planSummary/attr.remote`. Time-range flags work (`t.$date` is epoch).
+
+### Top templates by frequency (project `msg`, templatize, `uniq -c`)
+```
+1772320 CUSTOM COMMIT <*>
+ 715525 WT begin_transaction
+ 623649 WT commit_transaction
+ 242792 Slow query
+ 238223 About to run the command
+ 238220 Setting the Client
+ 238215 Released the Client
+ 238207 Received interrupt request for unknown op
+ 150773 Taking ticket.
+ 150750 Waiting for write concern. OpTime: <*>, write concern: <*>
+ 150741 Set last op to system time
+  91876 WT rollback_transaction
+  87276 Using classic engine idhack
+  20945 WiredTiger message
+   9596 flushed journal
+   3942 Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms
+   1111 Trimmed samples. Num: <*>
+   1111 Refreshing tickets. Before: <*> Now: <*>
+```
+
+### Discovered category breakdown (generic + app-specific)
+- **wt-transactions** (debug noise): CUSTOM COMMIT, WT begin/commit/rollback_transaction — ~3.19M templates.
+- **workload / operations**: Slow query (242,792), About to run the command, Setting/Released the Client, Taking ticket — ~1.43M.
+- **write-concern** (REPL): Waiting for write concern (150,750).
+- **storage**: flushed journal (9,596), Slow WT transaction (3,942), WiredTiger message (20,945).
+- **startup**: MongoDB starting, Build Info, Options set by command line, Ran initializers.
+- **network**: compression negotiation, Connection accepted, network-error sessions.
+
+## 3. Classification cache (the reusable artifact)
+
+- **app_key** = `2b94a6fde6c40eea0cf0231ca6d31a82abb8bf7319fead9149a595b89608d4f1` (sha256 of the sorted 199-template set).
+- **First run: cache MISS** → classified the 199 templates into the taxonomy above, built a query plan, and stored it via `logtype-cache put` → `~/.config/.../logtype-cache/2b94a6fd….json`.
+- **Immediate re-get: cache HIT** → the classification was reused verbatim; the classification step would be **skipped on every future run of this same MongoDB 6.0.5 application**. This is the amortization the skill is designed for: the one-time classification cost is paid once and reused across captures.
+
+## 4. Issues & Warnings
+
+- Errors/fatal: 0. Warnings: 3 (access-control disabled, vm.max_map_count too low, legacy wire opcode) — surfaced via `s:W` + projected `id,msg`.
+
+## 5. Performance Signals — grounded, not blind
+
+Every count below is derived from a **real template** in the baseline (no blind `msg:*term*` queries, which would return 0 on the clp-string `msg`):
+
+- **Slow query** (template `Slow query`, id 51803): 242,792; duration 0/0/1/237ms; by ns `ycsb.usertable` 238,013; by plan `IDHACK` 91,843; by client `127.0.0.1:55458` 146,178 + `:45820` 96,406; index effectiveness 1:1:1 (87,274).
+- **Slow WT transaction** (template `Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms`): **3,942** — projected `msg` and grepped the template's static text (`grep -c 'Slow WT transaction'`), since `msg:` KQL is dead. This is a template the blind batteries would never think to query.
+- **flushed journal**: 9,596.
+
+## 6. Replication
+
+- Template `Waiting for write concern. OpTime: <*>, write concern: <*>` → **150,750** (projected `msg` for `c:REPL`, grepped "write concern"). Elections: 0.
+
+## 7. Semantic Search Coverage
+
+Not used — the baseline (199 templates) was sufficient to ground every query. (Also, `semantic()` is broken in this environment — see `mongodb-semantic` report — so the skill correctly avoids depending on it.)
+
+## 8. Follow-up queries (derived from templates)
+
+1. `clp-s-search-kql --projection t.$date,attr.durationMillis,msg ARCHIVE 'id:51803' | grep '^{' | jq -r 'select((.attr.durationMillis//0)>50)|[…]|@tsv'` — slow tail (the 237ms outlier).
+2. `clp-s-search-kql --projection t.$date,msg ARCHIVE 'c:WTEVICT'` — eviction events (cache pressure), a template-driven angle the blind batteries miss.
+3. `clp-s-search-kql --projection msg ARCHIVE 'c:REPL' | grep '^{' | jq -r 'select(.msg|test("write concern";"i"))|.msg' | wc -l` — write-concern wait volume over time (run per-time-bucket with `--tge`/`--tle`).

mongodb-skills-eval/logtype-semantic-vs-kql-report.md

@@ -0,0 +1,55 @@
+# Logtype + Semantic vs Logtype + KQL
+
+**Question:** the logtype method (dump `stats.logtypes` → classify templates → retrieve per category) can retrieve records with either KQL (project `msg` + grep template static text, plus scalar filters) **or** semantic search (`semantic("category description")`). How do the two retrieval modes compare when everything else (baseline, classification, cache) is held constant?
+
+**Setup:** same archive as the other reports — 4,986,512 records, 199-template `stats.logtypes` baseline, classification cached (app_key `2b94a6…`, miss→store→hit). The retrieval step was run two ways:
+- **logtype + kql:** one templatize-frequency pass (`--projection msg` → templatize → `uniq -c`) gives exact counts for every template; per-category counts = sum of member templates. Local, deterministic.
+- **logtype + semantic:** one `semantic("category description")` query per category (`--semantic-top-k 10`, and a `top-k 30` follow-up), counting returned records and inspecting which templates come back.
+
+Semantic search works in this environment after the wrapper compat fix (see `mongodb-semantic-report.md`).
+
+## 1. Per-category results
+
+| Category | logtype + KQL (exact, sum of member templates) | logtype + semantic (top-k=10) | Verdict |
+|---|---|---|---|
+| **storage / WiredTiger** | CUSTOM COMMIT 1,772,320 + WT begin 715,525 + WT commit 623,649 + WT rollback 91,876 + WiredTiger message 20,945 + flushed journal 9,596 + Slow WT txn 3,942 ≈ **3,237,853** | **20,962** — only `WiredTiger message` (20,945) + a few; **missed all 3.19M WT-transaction templates** | semantic **catastrophically incomplete** |
+| **write-concern / replication** | Waiting for write concern **150,750** | **392,965** — Waiting for write concern 150,750 **+ `About to run the command` 238,223 (wrong) + Slow WT txn 3,942 (wrong)** | semantic **imprecise** (238k false matches) |
+| **slow-query / workload** | Slow query **242,792** (+ Using classic engine idhack 87,276 if grouped) | **330,085** — Slow query 242,792 + Using classic engine idhack 87,274 + a few | semantic broader, ~reasonable (groups the plan log) |
+| **startup / config** | MongoDB starting 1 + Build Info 1 + Options set by command line 1 = **3** | **55** — `Setting the Client` 238,220, `WiredTiger message` 19, `CUSTOM COMMIT` 4 … **`MongoDB starting` not retrieved** | semantic **missed the actual startup record** |
+| **errors / assertions** | User assertion 22 + Completed unstable checkpoint 18 + Assertion while executing command 2 + Internal assertion 2 = **44** | **67** — User assertion 22 + Completed unstable checkpoint 18 + Terminating session 6 + WiredTiger message 5 (wrong) + Slow query 2 (wrong) | semantic found the assertions **but with false matches; missed `Internal assertion`/`Assertion while executing command`** |
+| **unstable-checkpoint / recovery** | Completed unstable checkpoint **18** | **174** — WiredTiger message 150 (**wrong**) + Completed unstable checkpoint 18 + Invalidating user cache 1 | semantic **noisy** (150 false matches) |
+| **network / connections** | compression negotiation 181+178 + Connection accepted 11 + Connection ended 6 + Terminating session 6 + Session from remote 6 ≈ **388** | **423** — compression negotiation 181+178 + Connection accepted 11 + User assertion 20 (wrong) + session errors 6 each | semantic ~right **but with false matches** |
+
+## 2. Is the incompleteness a `top-k` artifact or an embedding mismatch?
+
+It's an **embedding mismatch**, not a top-k limit. With `--semantic-top-k 30`:
+- **storage:** still no `CUSTOM COMMIT` / `WT begin_transaction` / `WT commit_transaction` (the 3.19M records). Instead it pulled in *wrong categories* — `Waiting for write concern` (150,750), `User assertion` (8), `Assertion while executing command` (2). The embeddings of `WT begin_transaction` (abbreviated "WT") and `CUSTOM COMMIT {demangleName_typeid_change}` (cryptic) are not nearest to "WiredTiger transaction commit begin journal checkpoint".
+- **startup:** still no `MongoDB starting` (count 1, present in the baseline). It returned `Setting the Client` (238,220), `Using classic engine idhack` (87,276), `WiredTiger message` (96) — templates that share the common words "starting"/"setting" drown out the actual `MongoDB starting` record.
+
+So raising top-k does not recover the missed templates — it only adds more false matches.
+
+## 3. Verdict
+
+**logtype + KQL is the strictly superior retrieval mode; semantic is a conditional classification aid, not a primary retrieval method.**
+
+| Dimension | logtype + KQL | logtype + semantic |
+|---|---|---|
+| **Precision** | Exact — per-template counts from the templatize-frequency pass; no false matches | Imprecise — returns conceptually-nearest logtypes, pulling in wrong categories (`About to run the command` for write-concern; `WiredTiger message` for errors/startup/unstable-checkpoint; `User assertion` for network) |
+| **Coverage** | Complete — captures every template, including the 3.19M WT-transaction records AND the 1-record `MongoDB starting` | **Incomplete** — missed the 3.19M `CUSTOM COMMIT`/`WT begin`/`WT commit` storage templates and the `MongoDB starting` startup record (embedding mismatch, not fixable by top-k) |
+| **Determinism** | High (local computation) | Low (depends on the embedding model + endpoint) |
+| **Cost** | 1 local O(records) pass (templatize-frequency) + optional per-template project+grep | 1 endpoint round-trip per category (N), each decompressing matching records; local cache unavailable on clp-core 0.12.1 |
+| **Low-volume/novel signals** | Found (`User assertion` 22, `Completed unstable checkpoint` 18, `Assertion while executing command` 2, `Internal assertion` 2) — exactly, from the baseline | Found the 22 + 18, but missed `Internal assertion`/`Assertion while executing command` in its top-6 and surrounded them with false matches |
+
+The one place semantic was *reasonable* was the slow-query category, where it grouped `Slow query` with `Using classic engine idhack` (the plan-summary log) — a conceptually-related template. Everywhere else it was either noisier than KQL or materially incomplete.
+
+## 4. Why this validates the `logtype-insights` skill's existing design
+
+The `logtype-insights` skill already prescribes: *"Run `semantic()` ONLY when a template's category is ambiguous or to group similar templates — never as the default. The baseline is small; prefer classifying it directly."* This evaluation is empirical confirmation:
+
+- The **baseline** (`stats.logtypes` + the templatize-frequency pass) is the spine — it gives exact counts for all 199 templates in one local pass, including the high-volume WT-transaction templates and the single-record startup events that semantic retrieval misses entirely.
+- **KQL project+grep** (on the message field, scoped by `level:`/`logger:`/`attr.*` scalar filters) is the correct per-template retrieval — exact and complete.
+- **Semantic** earns its place only as an **optional aid during classification** — e.g., to vote on an ambiguous template's category or to cluster similar templates (the slow-query↔idhack grouping). As the *primary* retrieval it is worse on every axis (precision, coverage, determinism, cost).
+
+## 5. Recommendation
+
+Keep `logtype-insights` as-is: logtype baseline + KQL project+grep as the default retrieval, semantic conditional/optional for ambiguous-template classification. Do **not** add a "logtype + semantic as primary retrieval" variant — it would be strictly worse (imprecise, incomplete on the dominant WT-transaction templates, endpoint-dependent). If a future embedding model ranks `WT begin_transaction`/`CUSTOM COMMIT`/`MongoDB starting` near their conceptual queries, semantic's coverage gap would shrink — but its precision problem (wrong-category nearest-neighbors) is inherent to nearest-logtype retrieval and would remain.

mongodb-skills-eval/mongodb-grep-report.md

@@ -0,0 +1,94 @@
+# MongoDB Grep Insights Report
+
+**Skill:** `mongodb-grep` (raw `jq`/`grep`, no CLP)
+**Input:** `~/clp-demo-mongodb/mongo-subset` — 7 rotated `mongod` files (every ~40 min across the run), 1.88 GB raw.
+**Executor note:** Subagent delegation was unavailable in this environment (`unknown model group: yscope-default-subagent`), so the skill's prescribed `jq`/`grep` query battery was executed inline (codex-style). This report reflects the skill's query design, not subagent overhead.
+
+## 1. Summary
+
+- **Total records:** 4,986,512 (JSON lines with `t` and `s`)
+- **Time span:** 2023-03-21T23:34:54 → 2023-03-22T04:47:54 (~5h13m)
+- **MongoDB:** 6.0.5, host `hostb22`, port 27017, dbPath `/var/lib/mongodb`, pid 29265
+- **Workload:** YCSB benchmark against `ycsb.usertable` (point `find` by `_id` + `insert` + `update`), single primary, clients on `127.0.0.1`.
+
+### Severity
+| s | count |
+|---|---|
+| D3 | 3,680,404 |
+| D2 | 720,329 |
+| I | 242,871 |
+| D4 | 167,182 |
+| D5 | 150,741 |
+| D1 | 24,982 |
+| W | 3 |
+| E | 0 |
+| F | 0 |
+
+### Components (top)
+STORAGE 3,219,273 · `-` 628,348 · COMMAND 476,504 · QUERY 330,056 · WRITE 159,921 · REPL 150,755 · WTWRTLOG 19,434 · WTEVICT 1,169 · NETWORK 433 · WTCHKPT 310 · FTDC 122 · INDEX 52 · ASSERT 24 · CONTROL 9 · ACCESS 2.
+
+## 2. Issues & Warnings
+
+- **Errors/fatal: 0** (`s:E`/`s:F`). No exceptions, assertions, or tracebacks in any record.
+- **Warnings: 3** (all `startupWarnings` at boot):
+  - `22120` Access control is not enabled for the database — data/config access unrestricted.
+  - `5123300` vm.max_map_count is too low (current 65530, recommended ≥102400, maxConns 51200).
+  - `5578800` Legacy wire-protocol op code used — client driver may need an upgrade.
+- **Free-text exception grep caveat:** `grep -rhE 'Exception|Assertion|Fatal|abort|Traceback'` matched **44 lines**, but these are **false positives** — the words appear inside `attr` payloads (e.g. `RegisterErrorExtraInfoFor…` in the startup `Ran initializers` record) and in `ASSERT` debug records, not in any `s:E`/`s:F` record. The structured `s:E OR s:F` count (0) is the truth. This is grep's characteristic blind spot on JSON logs: free-text matches hit payload text, not severity.
+
+## 3. Performance Signals — logged operations
+
+- **"Slow query" (id 51803): 242,792 records.** At debug COMMAND verbosity every operation is logged as "Slow query"; `attr.durationMillis` is the real duration.
+- **Duration distribution:** n=242,792, min=0, median=0, p95=1ms, **max=237ms** (one genuine outlier).
+- **By command verb:** insert 146,173 · find 87,278 · update 4,568 · `q` (internal) 4,569 · ismaster 183 · dropDatabase 9 · listIndexes 8 · buildinfo 2.
+- **By namespace:** `ycsb.usertable` 238,013 · `ycsb.$cmd` 4,566 · `admin.$cmd` 185 · `config.system.sessions` 11.
+- **By plan:** `none` 150,945 (writes) · `IDHACK` 91,843 (the finds) · `EOF` 4.
+- **By client:** `127.0.0.1:55458` 146,176 (inserter) · `127.0.0.1:45820` 96,406 (finder) · a handful of short-lived conns.
+- **Index effectiveness** (`keysExamined:docsExamined:nreturned`): 87,274 at **1:1:1** (perfect IDHACK point lookups), 4,566 at 1:1:0 (updates), 4 at 0:0:0. Indexing is optimal.
+
+## 4. Replication & Elections
+
+- **REPL records: 150,755.** Top message: `Waiting for write concern. OpTime: {replOpTime}, write concern: {writeConcern}` — **150,750** (99.997% of REPL).
+- **Elections/stepdowns/rollbacks: 0.** This is a steady-state primary under heavy write load; the REPL signal is write-concern latency, not elections.
+
+## 5. Storage & WiredTiger
+
+- **STORAGE/WT records: ~3.34M.** Dominant templates: `CUSTOM COMMIT {demangleName_typeid_change}` 1,772,320 · `WT begin_transaction` 715,525 · `WT commit_transaction` 623,649 · `WT rollback_transaction` 91,876 · `flushed journal` 9,596 · `Slow WT transaction. Lifetime of SnapshotId {snapshotId} was {transactionTime}ms` 3,942 · `Trimmed samples` 1,111.
+- The `WTEVICT` component (1,169) and `Slow WT transaction` (3,942) are the operationally interesting storage signals; the rest is high-volume transaction debug noise.
+
+## 6. Connections & Network
+
+- **NETWORK records: 433.** `Starting server-side compression negotiation` 181 · `Compression negotiation not requested by client` 178 · `Connection accepted` 11 · `client metadata` 11 · `Terminating session due to error` 6 · `Session from remote encountered a network error during SourceMessage` 6. The 6 "network error during SourceMessage" events are worth a glance but are on transient connections.
+
+## 7. Configuration & Startup
+
+- `id:4615611` "MongoDB starting": pid 29265, port 27017, dbPath `/var/lib/mongodb`, host `hostb22`, 64-bit.
+- `id:23403` "Build Info": version **6.0.5**, gitVersion c9a99c12…, OpenSSL 1.1.1, allocator tcmalloc, distmod ubuntu1804.
+- No auth, no TLS enforcement (per startup warnings).
+
+## 8. Top repeated messages (logtype-equivalent)
+
+```
+1772320 CUSTOM COMMIT <*>
+ 715525 WT begin_transaction
+ 623649 WT commit_transaction
+ 242792 Slow query
+ 238223 About to run the command
+ 238220 Setting the Client
+ 238215 Released the Client
+ 238207 Received interrupt request for unknown op
+ 150773 Taking ticket.
+ 150750 Waiting for write concern. OpTime: <*>, write concern: <*>
+ 150741 Set last op to system time
+  91876 WT rollback_transaction
+  87276 Using classic engine idhack
+  20945 WiredTiger message
+   9596 flushed journal
+   3942 Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms
+```
+
+## 9. Follow-up commands
+
+1. `find $F -type f -print0 | xargs -0 jq -r 'select(.id==51803 and (.attr.durationMillis//0)>50) | [.t."$date",.attr.durationMillis,.attr.ns,.attr.command|to_entries[0].key]|@tsv'` — the genuinely-slow tail (only the 237ms outlier is >50ms here).
+2. `find $F -type f -print0 | xargs -0 jq -r 'select(.c=="WTEVICT") | [.t."$date",.msg]|@tsv'` — WiredTiger eviction events (potential cache pressure).
+3. `find $F -type f -print0 | xargs -0 jq -r 'select(.c=="NETWORK" and (.msg|test("error|Terminating";"i"))) | [.t."$date",.msg,.attr.remote] | @tsv'` — the 6 network-error sessions with their remote endpoints.