diff --git a/.gitignore b/.gitignore index 0d09b97..4cd31ad 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,4 @@ plugins/clp/.clp-core/ # Common local/runtime noise. .DS_Store +plugins/clp/bin/clp-s diff --git a/mongodb-skills-eval/logtype-insights-report.md b/mongodb-skills-eval/logtype-insights-report.md new file mode 100644 index 0000000..45b26d2 --- /dev/null +++ b/mongodb-skills-eval/logtype-insights-report.md @@ -0,0 +1,78 @@ +# Logtype Insights Report (App-Agnostic, Logtype-Baseline) + +**Skill:** `logtype-insights` (schema discovery → `stats.logtypes` baseline → classify → cache → targeted queries) +**Input:** same 7-file / 1.88 GB subset → 9.39 MB archive, 4,986,512 records (after the wrapper null-delimiter fix). +**Executor note:** Subagent delegation unavailable; the two phases (classify, insight) were executed inline. The classification cache round-trip was exercised for real. + +## 1. Summary + +- **Total records:** 4,986,512; span 2023-03-21T23:34:54 → 2023-03-22T04:47:54 (~5h13m); MongoDB 6.0.5 / hostb22. +- **Severity:** D3 3.68M, D2 720k, I 242k, D4 167k, D5 150k, D1 25k, W 3, E 0, F 0. +- **Top logger/component:** STORAGE (3.22M). + +## 2. Logtype Baseline — the spine + +- **Distinct templates: 199** (via `stats.logtypes`, reads the dictionary, not every record — instant). +- **Schema discovered** from one sample record: fields `t`, `s`, `c`, `id`, `ctx`, `msg`, `attr` → `timestamp=t.$date`, `severity=s`, `logger=c`, `message=msg`, payload leaves `attr.durationMillis/attr.ns/attr.planSummary/attr.remote`. Time-range flags work (`t.$date` is epoch). + +### Top templates by frequency (project `msg`, templatize, `uniq -c`) +``` +1772320 CUSTOM COMMIT <*> + 715525 WT begin_transaction + 623649 WT commit_transaction + 242792 Slow query + 238223 About to run the command + 238220 Setting the Client + 238215 Released the Client + 238207 Received interrupt request for unknown op + 150773 Taking ticket. + 150750 Waiting for write concern. OpTime: <*>, write concern: <*> + 150741 Set last op to system time + 91876 WT rollback_transaction + 87276 Using classic engine idhack + 20945 WiredTiger message + 9596 flushed journal + 3942 Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms + 1111 Trimmed samples. Num: <*> + 1111 Refreshing tickets. Before: <*> Now: <*> +``` + +### Discovered category breakdown (generic + app-specific) +- **wt-transactions** (debug noise): CUSTOM COMMIT, WT begin/commit/rollback_transaction — ~3.19M templates. +- **workload / operations**: Slow query (242,792), About to run the command, Setting/Released the Client, Taking ticket — ~1.43M. +- **write-concern** (REPL): Waiting for write concern (150,750). +- **storage**: flushed journal (9,596), Slow WT transaction (3,942), WiredTiger message (20,945). +- **startup**: MongoDB starting, Build Info, Options set by command line, Ran initializers. +- **network**: compression negotiation, Connection accepted, network-error sessions. + +## 3. Classification cache (the reusable artifact) + +- **app_key** = `2b94a6fde6c40eea0cf0231ca6d31a82abb8bf7319fead9149a595b89608d4f1` (sha256 of the sorted 199-template set). +- **First run: cache MISS** → classified the 199 templates into the taxonomy above, built a query plan, and stored it via `logtype-cache put` → `~/.config/.../logtype-cache/2b94a6fd….json`. +- **Immediate re-get: cache HIT** → the classification was reused verbatim; the classification step would be **skipped on every future run of this same MongoDB 6.0.5 application**. This is the amortization the skill is designed for: the one-time classification cost is paid once and reused across captures. + +## 4. Issues & Warnings + +- Errors/fatal: 0. Warnings: 3 (access-control disabled, vm.max_map_count too low, legacy wire opcode) — surfaced via `s:W` + projected `id,msg`. + +## 5. Performance Signals — grounded, not blind + +Every count below is derived from a **real template** in the baseline (no blind `msg:*term*` queries, which would return 0 on the clp-string `msg`): + +- **Slow query** (template `Slow query`, id 51803): 242,792; duration 0/0/1/237ms; by ns `ycsb.usertable` 238,013; by plan `IDHACK` 91,843; by client `127.0.0.1:55458` 146,178 + `:45820` 96,406; index effectiveness 1:1:1 (87,274). +- **Slow WT transaction** (template `Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms`): **3,942** — projected `msg` and grepped the template's static text (`grep -c 'Slow WT transaction'`), since `msg:` KQL is dead. This is a template the blind batteries would never think to query. +- **flushed journal**: 9,596. + +## 6. Replication + +- Template `Waiting for write concern. OpTime: <*>, write concern: <*>` → **150,750** (projected `msg` for `c:REPL`, grepped "write concern"). Elections: 0. + +## 7. Semantic Search Coverage + +Not used — the baseline (199 templates) was sufficient to ground every query. (Also, `semantic()` is broken in this environment — see `mongodb-semantic` report — so the skill correctly avoids depending on it.) + +## 8. Follow-up queries (derived from templates) + +1. `clp-s-search-kql --projection t.$date,attr.durationMillis,msg ARCHIVE 'id:51803' | grep '^{' | jq -r 'select((.attr.durationMillis//0)>50)|[…]|@tsv'` — slow tail (the 237ms outlier). +2. `clp-s-search-kql --projection t.$date,msg ARCHIVE 'c:WTEVICT'` — eviction events (cache pressure), a template-driven angle the blind batteries miss. +3. `clp-s-search-kql --projection msg ARCHIVE 'c:REPL' | grep '^{' | jq -r 'select(.msg|test("write concern";"i"))|.msg' | wc -l` — write-concern wait volume over time (run per-time-bucket with `--tge`/`--tle`). \ No newline at end of file diff --git a/mongodb-skills-eval/logtype-semantic-vs-kql-report.md b/mongodb-skills-eval/logtype-semantic-vs-kql-report.md new file mode 100644 index 0000000..322d184 --- /dev/null +++ b/mongodb-skills-eval/logtype-semantic-vs-kql-report.md @@ -0,0 +1,55 @@ +# Logtype + Semantic vs Logtype + KQL + +**Question:** the logtype method (dump `stats.logtypes` → classify templates → retrieve per category) can retrieve records with either KQL (project `msg` + grep template static text, plus scalar filters) **or** semantic search (`semantic("category description")`). How do the two retrieval modes compare when everything else (baseline, classification, cache) is held constant? + +**Setup:** same archive as the other reports — 4,986,512 records, 199-template `stats.logtypes` baseline, classification cached (app_key `2b94a6…`, miss→store→hit). The retrieval step was run two ways: +- **logtype + kql:** one templatize-frequency pass (`--projection msg` → templatize → `uniq -c`) gives exact counts for every template; per-category counts = sum of member templates. Local, deterministic. +- **logtype + semantic:** one `semantic("category description")` query per category (`--semantic-top-k 10`, and a `top-k 30` follow-up), counting returned records and inspecting which templates come back. + +Semantic search works in this environment after the wrapper compat fix (see `mongodb-semantic-report.md`). + +## 1. Per-category results + +| Category | logtype + KQL (exact, sum of member templates) | logtype + semantic (top-k=10) | Verdict | +|---|---|---|---| +| **storage / WiredTiger** | CUSTOM COMMIT 1,772,320 + WT begin 715,525 + WT commit 623,649 + WT rollback 91,876 + WiredTiger message 20,945 + flushed journal 9,596 + Slow WT txn 3,942 ≈ **3,237,853** | **20,962** — only `WiredTiger message` (20,945) + a few; **missed all 3.19M WT-transaction templates** | semantic **catastrophically incomplete** | +| **write-concern / replication** | Waiting for write concern **150,750** | **392,965** — Waiting for write concern 150,750 **+ `About to run the command` 238,223 (wrong) + Slow WT txn 3,942 (wrong)** | semantic **imprecise** (238k false matches) | +| **slow-query / workload** | Slow query **242,792** (+ Using classic engine idhack 87,276 if grouped) | **330,085** — Slow query 242,792 + Using classic engine idhack 87,274 + a few | semantic broader, ~reasonable (groups the plan log) | +| **startup / config** | MongoDB starting 1 + Build Info 1 + Options set by command line 1 = **3** | **55** — `Setting the Client` 238,220, `WiredTiger message` 19, `CUSTOM COMMIT` 4 … **`MongoDB starting` not retrieved** | semantic **missed the actual startup record** | +| **errors / assertions** | User assertion 22 + Completed unstable checkpoint 18 + Assertion while executing command 2 + Internal assertion 2 = **44** | **67** — User assertion 22 + Completed unstable checkpoint 18 + Terminating session 6 + WiredTiger message 5 (wrong) + Slow query 2 (wrong) | semantic found the assertions **but with false matches; missed `Internal assertion`/`Assertion while executing command`** | +| **unstable-checkpoint / recovery** | Completed unstable checkpoint **18** | **174** — WiredTiger message 150 (**wrong**) + Completed unstable checkpoint 18 + Invalidating user cache 1 | semantic **noisy** (150 false matches) | +| **network / connections** | compression negotiation 181+178 + Connection accepted 11 + Connection ended 6 + Terminating session 6 + Session from remote 6 ≈ **388** | **423** — compression negotiation 181+178 + Connection accepted 11 + User assertion 20 (wrong) + session errors 6 each | semantic ~right **but with false matches** | + +## 2. Is the incompleteness a `top-k` artifact or an embedding mismatch? + +It's an **embedding mismatch**, not a top-k limit. With `--semantic-top-k 30`: +- **storage:** still no `CUSTOM COMMIT` / `WT begin_transaction` / `WT commit_transaction` (the 3.19M records). Instead it pulled in *wrong categories* — `Waiting for write concern` (150,750), `User assertion` (8), `Assertion while executing command` (2). The embeddings of `WT begin_transaction` (abbreviated "WT") and `CUSTOM COMMIT {demangleName_typeid_change}` (cryptic) are not nearest to "WiredTiger transaction commit begin journal checkpoint". +- **startup:** still no `MongoDB starting` (count 1, present in the baseline). It returned `Setting the Client` (238,220), `Using classic engine idhack` (87,276), `WiredTiger message` (96) — templates that share the common words "starting"/"setting" drown out the actual `MongoDB starting` record. + +So raising top-k does not recover the missed templates — it only adds more false matches. + +## 3. Verdict + +**logtype + KQL is the strictly superior retrieval mode; semantic is a conditional classification aid, not a primary retrieval method.** + +| Dimension | logtype + KQL | logtype + semantic | +|---|---|---| +| **Precision** | Exact — per-template counts from the templatize-frequency pass; no false matches | Imprecise — returns conceptually-nearest logtypes, pulling in wrong categories (`About to run the command` for write-concern; `WiredTiger message` for errors/startup/unstable-checkpoint; `User assertion` for network) | +| **Coverage** | Complete — captures every template, including the 3.19M WT-transaction records AND the 1-record `MongoDB starting` | **Incomplete** — missed the 3.19M `CUSTOM COMMIT`/`WT begin`/`WT commit` storage templates and the `MongoDB starting` startup record (embedding mismatch, not fixable by top-k) | +| **Determinism** | High (local computation) | Low (depends on the embedding model + endpoint) | +| **Cost** | 1 local O(records) pass (templatize-frequency) + optional per-template project+grep | 1 endpoint round-trip per category (N), each decompressing matching records; local cache unavailable on clp-core 0.12.1 | +| **Low-volume/novel signals** | Found (`User assertion` 22, `Completed unstable checkpoint` 18, `Assertion while executing command` 2, `Internal assertion` 2) — exactly, from the baseline | Found the 22 + 18, but missed `Internal assertion`/`Assertion while executing command` in its top-6 and surrounded them with false matches | + +The one place semantic was *reasonable* was the slow-query category, where it grouped `Slow query` with `Using classic engine idhack` (the plan-summary log) — a conceptually-related template. Everywhere else it was either noisier than KQL or materially incomplete. + +## 4. Why this validates the `logtype-insights` skill's existing design + +The `logtype-insights` skill already prescribes: *"Run `semantic()` ONLY when a template's category is ambiguous or to group similar templates — never as the default. The baseline is small; prefer classifying it directly."* This evaluation is empirical confirmation: + +- The **baseline** (`stats.logtypes` + the templatize-frequency pass) is the spine — it gives exact counts for all 199 templates in one local pass, including the high-volume WT-transaction templates and the single-record startup events that semantic retrieval misses entirely. +- **KQL project+grep** (on the message field, scoped by `level:`/`logger:`/`attr.*` scalar filters) is the correct per-template retrieval — exact and complete. +- **Semantic** earns its place only as an **optional aid during classification** — e.g., to vote on an ambiguous template's category or to cluster similar templates (the slow-query↔idhack grouping). As the *primary* retrieval it is worse on every axis (precision, coverage, determinism, cost). + +## 5. Recommendation + +Keep `logtype-insights` as-is: logtype baseline + KQL project+grep as the default retrieval, semantic conditional/optional for ambiguous-template classification. Do **not** add a "logtype + semantic as primary retrieval" variant — it would be strictly worse (imprecise, incomplete on the dominant WT-transaction templates, endpoint-dependent). If a future embedding model ranks `WT begin_transaction`/`CUSTOM COMMIT`/`MongoDB starting` near their conceptual queries, semantic's coverage gap would shrink — but its precision problem (wrong-category nearest-neighbors) is inherent to nearest-logtype retrieval and would remain. \ No newline at end of file diff --git a/mongodb-skills-eval/mongodb-grep-report.md b/mongodb-skills-eval/mongodb-grep-report.md new file mode 100644 index 0000000..e0b55e7 --- /dev/null +++ b/mongodb-skills-eval/mongodb-grep-report.md @@ -0,0 +1,94 @@ +# MongoDB Grep Insights Report + +**Skill:** `mongodb-grep` (raw `jq`/`grep`, no CLP) +**Input:** `~/clp-demo-mongodb/mongo-subset` — 7 rotated `mongod` files (every ~40 min across the run), 1.88 GB raw. +**Executor note:** Subagent delegation was unavailable in this environment (`unknown model group: yscope-default-subagent`), so the skill's prescribed `jq`/`grep` query battery was executed inline (codex-style). This report reflects the skill's query design, not subagent overhead. + +## 1. Summary + +- **Total records:** 4,986,512 (JSON lines with `t` and `s`) +- **Time span:** 2023-03-21T23:34:54 → 2023-03-22T04:47:54 (~5h13m) +- **MongoDB:** 6.0.5, host `hostb22`, port 27017, dbPath `/var/lib/mongodb`, pid 29265 +- **Workload:** YCSB benchmark against `ycsb.usertable` (point `find` by `_id` + `insert` + `update`), single primary, clients on `127.0.0.1`. + +### Severity +| s | count | +|---|---| +| D3 | 3,680,404 | +| D2 | 720,329 | +| I | 242,871 | +| D4 | 167,182 | +| D5 | 150,741 | +| D1 | 24,982 | +| W | 3 | +| E | 0 | +| F | 0 | + +### Components (top) +STORAGE 3,219,273 · `-` 628,348 · COMMAND 476,504 · QUERY 330,056 · WRITE 159,921 · REPL 150,755 · WTWRTLOG 19,434 · WTEVICT 1,169 · NETWORK 433 · WTCHKPT 310 · FTDC 122 · INDEX 52 · ASSERT 24 · CONTROL 9 · ACCESS 2. + +## 2. Issues & Warnings + +- **Errors/fatal: 0** (`s:E`/`s:F`). No exceptions, assertions, or tracebacks in any record. +- **Warnings: 3** (all `startupWarnings` at boot): + - `22120` Access control is not enabled for the database — data/config access unrestricted. + - `5123300` vm.max_map_count is too low (current 65530, recommended ≥102400, maxConns 51200). + - `5578800` Legacy wire-protocol op code used — client driver may need an upgrade. +- **Free-text exception grep caveat:** `grep -rhE 'Exception|Assertion|Fatal|abort|Traceback'` matched **44 lines**, but these are **false positives** — the words appear inside `attr` payloads (e.g. `RegisterErrorExtraInfoFor…` in the startup `Ran initializers` record) and in `ASSERT` debug records, not in any `s:E`/`s:F` record. The structured `s:E OR s:F` count (0) is the truth. This is grep's characteristic blind spot on JSON logs: free-text matches hit payload text, not severity. + +## 3. Performance Signals — logged operations + +- **"Slow query" (id 51803): 242,792 records.** At debug COMMAND verbosity every operation is logged as "Slow query"; `attr.durationMillis` is the real duration. +- **Duration distribution:** n=242,792, min=0, median=0, p95=1ms, **max=237ms** (one genuine outlier). +- **By command verb:** insert 146,173 · find 87,278 · update 4,568 · `q` (internal) 4,569 · ismaster 183 · dropDatabase 9 · listIndexes 8 · buildinfo 2. +- **By namespace:** `ycsb.usertable` 238,013 · `ycsb.$cmd` 4,566 · `admin.$cmd` 185 · `config.system.sessions` 11. +- **By plan:** `none` 150,945 (writes) · `IDHACK` 91,843 (the finds) · `EOF` 4. +- **By client:** `127.0.0.1:55458` 146,176 (inserter) · `127.0.0.1:45820` 96,406 (finder) · a handful of short-lived conns. +- **Index effectiveness** (`keysExamined:docsExamined:nreturned`): 87,274 at **1:1:1** (perfect IDHACK point lookups), 4,566 at 1:1:0 (updates), 4 at 0:0:0. Indexing is optimal. + +## 4. Replication & Elections + +- **REPL records: 150,755.** Top message: `Waiting for write concern. OpTime: {replOpTime}, write concern: {writeConcern}` — **150,750** (99.997% of REPL). +- **Elections/stepdowns/rollbacks: 0.** This is a steady-state primary under heavy write load; the REPL signal is write-concern latency, not elections. + +## 5. Storage & WiredTiger + +- **STORAGE/WT records: ~3.34M.** Dominant templates: `CUSTOM COMMIT {demangleName_typeid_change}` 1,772,320 · `WT begin_transaction` 715,525 · `WT commit_transaction` 623,649 · `WT rollback_transaction` 91,876 · `flushed journal` 9,596 · `Slow WT transaction. Lifetime of SnapshotId {snapshotId} was {transactionTime}ms` 3,942 · `Trimmed samples` 1,111. +- The `WTEVICT` component (1,169) and `Slow WT transaction` (3,942) are the operationally interesting storage signals; the rest is high-volume transaction debug noise. + +## 6. Connections & Network + +- **NETWORK records: 433.** `Starting server-side compression negotiation` 181 · `Compression negotiation not requested by client` 178 · `Connection accepted` 11 · `client metadata` 11 · `Terminating session due to error` 6 · `Session from remote encountered a network error during SourceMessage` 6. The 6 "network error during SourceMessage" events are worth a glance but are on transient connections. + +## 7. Configuration & Startup + +- `id:4615611` "MongoDB starting": pid 29265, port 27017, dbPath `/var/lib/mongodb`, host `hostb22`, 64-bit. +- `id:23403` "Build Info": version **6.0.5**, gitVersion c9a99c12…, OpenSSL 1.1.1, allocator tcmalloc, distmod ubuntu1804. +- No auth, no TLS enforcement (per startup warnings). + +## 8. Top repeated messages (logtype-equivalent) + +``` +1772320 CUSTOM COMMIT <*> + 715525 WT begin_transaction + 623649 WT commit_transaction + 242792 Slow query + 238223 About to run the command + 238220 Setting the Client + 238215 Released the Client + 238207 Received interrupt request for unknown op + 150773 Taking ticket. + 150750 Waiting for write concern. OpTime: <*>, write concern: <*> + 150741 Set last op to system time + 91876 WT rollback_transaction + 87276 Using classic engine idhack + 20945 WiredTiger message + 9596 flushed journal + 3942 Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms +``` + +## 9. Follow-up commands + +1. `find $F -type f -print0 | xargs -0 jq -r 'select(.id==51803 and (.attr.durationMillis//0)>50) | [.t."$date",.attr.durationMillis,.attr.ns,.attr.command|to_entries[0].key]|@tsv'` — the genuinely-slow tail (only the 237ms outlier is >50ms here). +2. `find $F -type f -print0 | xargs -0 jq -r 'select(.c=="WTEVICT") | [.t."$date",.msg]|@tsv'` — WiredTiger eviction events (potential cache pressure). +3. `find $F -type f -print0 | xargs -0 jq -r 'select(.c=="NETWORK" and (.msg|test("error|Terminating";"i"))) | [.t."$date",.msg,.attr.remote] | @tsv'` — the 6 network-error sessions with their remote endpoints. \ No newline at end of file diff --git a/mongodb-skills-eval/mongodb-kql-report.md b/mongodb-skills-eval/mongodb-kql-report.md new file mode 100644 index 0000000..301f714 --- /dev/null +++ b/mongodb-skills-eval/mongodb-kql-report.md @@ -0,0 +1,83 @@ +# MongoDB KQL Insights Report + +**Skill:** `mongodb-kql` (native CLP compress + keyword KQL only, no semantic) +**Input:** `~/clp-demo-mongodb/mongo-subset` (7 files, 1.88 GB) → compressed natively with `--extensions '*' --timestamp-key t.$date` → 9.39 MB archive (200×), **4,986,512 records**. +**Executor note:** Subagent delegation unavailable (`unknown model group: yscope-default-subagent`); the skill's KQL battery was executed inline. + +> **Critical caveat — resolved during the run:** the first compression produced an archive with only **683,022 records (file 1 only, 14%)**, not 4.99M. Root cause: `clp-s-compress-folder` wrote the file list **null-delimited** (`find -print0`) but `clp-s -f` expects **newline-delimited**, so only the first file was ingested. Confirmed directly (newline list of 2 files → 1,359,713 records; null list → 683,022) and **fixed** in the wrapper (one null→newline conversion). All numbers below are from the corrected full archive. See the comparison report. + +## 1. Summary + +- **Total records:** 4,986,512 (search `*` = 4,986,512; decompress-count = 4,986,512 — consistent) +- **Time span:** 2023-03-21T23:34:54 → 2023-03-22T04:47:54 (~5h13m) +- **MongoDB:** 6.0.5, host `hostb22`, port 27017 + +### Severity (KQL `s:`) +| s | count | +|---|---| +| D3 | 3,680,404 | +| D2 | 720,329 | +| I | 242,871 | +| D4 | 167,182 | +| D5 | 150,741 | +| D1 | 24,982 | +| W | 3 | +| E | 0 | +| F | 0 | + +## 2. Issues & Warnings + +- **Errors/fatal: 0** (`s:E OR s:F`). KQL severity filtering is exact — no false positives (contrast `mongodb-grep`'s 44 free-text "exception" matches). +- **Warnings: 3** (projected `id,msg` for `s:W`): `22120` access-control disabled · `5123300` vm.max_map_count too low · `5578800` legacy wire opcode. + +## 3. Performance Signals — logged operations + +- **"Slow query" (`id:51803`): 242,792.** Filtered by `id:51803` (not `msg:`, which is a clp-string and returns 0). +- **Duration dist:** n=242,792, min=0, median=0, p95=1ms, max=237ms. +- **By namespace** (`attr.ns`): `ycsb.usertable` 238,013 · `ycsb.$cmd` 4,566 · `admin.$cmd` 185 · `config.system.sessions` 11. +- **By plan** (`attr.planSummary`): `IDHACK` 91,843 · `EOF` 4 (writes have no planSummary). +- **By client** (`attr.remote`): `127.0.0.1:55458` 146,178 · `127.0.0.1:45820` 96,406 · … +- **By command verb** (existence `attr.command.:*`): insert 146,173 · find 87,278 · update 4,568 · ismaster 183 · dropDatabase 9 · listIndexes 8 · buildinfo 2. (`attr.command` is a nested **object** — projecting it returns null; counted each verb via an existence query on its leaf, per the skill.) +- **Index effectiveness** (`attr.keysExamined,attr.docsExamined,attr.nreturned`): 87,274 at 1:1:1, 4,566 at 1:1:0, 7 at 0:0:0. + +## 4. Replication & Elections + +- **REPL (`c:REPL`): 150,755.** Top: `Waiting for write concern. OpTime: {replOpTime}, write concern: {writeConcern}` = **150,750**. Elections/stepdowns: **0** (steady-state primary). + +## 5. Storage & WiredTiger + +- `c:STORAGE OR c:WT OR c:WTWRTLOG OR c:WTCHKPT OR c:WTRECOV OR c:WTTS`: ~3.34M. Dominant: `CUSTOM COMMIT` 1,772,320, `WT begin/commit_transaction` 715k/623k, `WT rollback_transaction` 91,876, `flushed journal` 9,596, `Slow WT transaction …` 3,942. + +## 6. Connections & Network + +- `c:NETWORK OR c:ACCESS`: 435. `Starting server-side compression negotiation` 181 · `Compression negotiation not requested` 178 · `Connection accepted` 11 · `Terminating session due to error` 6 · `Session from remote encountered a network error during SourceMessage` 6. + +## 7. Configuration & Startup + +- `id:4615611` "MongoDB starting": pid 29265, port 27017, dbPath `/var/lib/mongodb`, host `hostb22`. Build Info `id:23403`: version 6.0.5. +- Note: projecting the `attr` **object** wholesale returns `{}` (only leaf paths like `attr.version`/`attr.host`/`attr.port` project). The skill prescribes leaf projection; a naive `--projection attr` returns nothing. + +## 8. Logtype dictionary + +- `stats.logtypes`: **199 distinct templates** (works natively on the Mongo archive — logtypes built from `msg`). + +## 9. Top templates (project `msg`, templatize, `uniq -c`) + +``` +1772320 CUSTOM COMMIT <*> + 715525 WT begin_transaction + 623649 WT commit_transaction + 242792 Slow query + 238223 About to run the command + 150750 Waiting for write concern. OpTime: <*>, write concern: <*> + 91876 WT rollback_transaction + 20945 WiredTiger message + 9596 flushed journal + 3942 Slow WT transaction. Lifetime of SnapshotId <*> was <*>ms +``` + +## 10. Follow-up KQL queries + +1. `clp-s-search-kql --projection t.$date,attr.durationMillis,attr.ns,msg ARCHIVE 'id:51803' | grep '^{' | jq -r 'select((.attr.durationMillis//0)>50)|[…]|@tsv'` — the genuinely-slow tail. +2. `clp-s-search-kql ARCHIVE 'c:WTEVICT'` — WiredTiger eviction (cache pressure). +3. `clp-s-search-kql --projection t.$date,msg,attr.remote ARCHIVE 'c:NETWORK' | grep '^{' | jq -r 'select(.msg|test("error|Terminating";"i"))|[…]|@tsv'` — the 6 network-error sessions. \ No newline at end of file diff --git a/mongodb-skills-eval/mongodb-semantic-report.md b/mongodb-skills-eval/mongodb-semantic-report.md new file mode 100644 index 0000000..4a56c99 --- /dev/null +++ b/mongodb-skills-eval/mongodb-semantic-report.md @@ -0,0 +1,47 @@ +# MongoDB Semantic Insights Report + +**Skill:** `mongodb-semantic` (native CLP compress + KQL **+ semantic search**) +**Input:** same 7-file / 1.88 GB subset → 9.39 MB archive, 4,986,512 records (after the wrapper null-delimiter fix). +**Executor note:** Subagent delegation unavailable this session (`unknown model group: yscope-default-subagent`); battery executed inline. Subagent config since fixed (see comparison report). + +## ✅ Headline: semantic search works (after a wrapper compat fix) + +The first runs failed every `semantic("…")` query with `Unknown OUTPUT_HANDLER: `. Root cause: the `clp-s-search-kql` wrapper passed `--semantic-cache-dir` / `--semantic-cache-cold-capacity` by default, but the installed `clp-s` (clp-core **0.12.1**) does **not** support those flags (absent from `clp-s s --help`), causing a positional misparse. **Fix applied to the wrapper:** it now probes `clp-s s --help` for `--semantic-cache-dir` support and skips the cache flags (falling back to remote-only `/v1/similarity`) when unsupported, emitting a warning. After the fix, semantic search works by default: + +``` +warning: clp-s does not support --semantic-cache-dir; using remote-only semantic search (no local cache) +{"msg":"Slow query"} … (results returned) +``` + +The endpoint auto-selected `https://ca-central-1-semantic-cache.yscope.ai` and is reachable. (This is the same toolchain incompatibility the previous vLLM eval's `vllm-insights` hit — §4.4 — now fixed at the wrapper so the skills don't need a per-call `--semantic-cache-dir none` workaround.) + +## 1–9. Keyword findings (identical to `mongodb-kql`) + +The keyword battery is the same as `mongodb-kql`, so keyword findings match: total 4,986,512; severity D3-dominant, 0 E/F; 3 startup warnings; 242,792 "Slow query" (id 51803, duration 0/0/1/237ms, ycsb.usertable IDHACK + insert + update, 2 main local clients, index effectiveness 1:1:1); 150,750 write-concern waits, 0 elections; ~3.34M storage/WT; 199 templates. + +## 10. Semantic Search Coverage — semantic-only findings (the value) + +Each `semantic()` query was run with `--semantic-top-k 10`. Semantic search found several issue signals the keyword battery (`s:E OR s:F` = 0, and component-specific queries) **missed**, because they live at D1/D2 severity under the `ASSERT`/`RECOVERY`/`COMMAND` components — not `E`/`F`: + +| semantic query | semantic-only finding | count | severity/component | keyword missed? | +|---|---|---|---|---| +| `errors failures exceptions fatal` | **User assertion** | 12 | D1 / ASSERT / id 23074 | ✅ keyword `s:E/F`=0; no `c:ASSERT` query | +| `errors failures exceptions fatal` | **Completed unstable checkpoint.** | 18 | D2 / RECOVERY | ✅ no `c:RECOVERY` query | +| `errors failures exceptions fatal` | Terminating session due to error / Session from remote encountered a network error / Connection ended / Ending session | 6 each | I / NETWORK | ⚠ keyword `c:NETWORK` found these, but not grouped as failures | +| `connection failures network errors authentication denied` | **Assertion while executing command** | 2 | D1 / COMMAND | ✅ keyword `c:COMMAND` was scoped to `id:51803` | +| `connection failures …` | Access control is not enabled for the database | 1 | W / CONTROL | (also found by `s:W`) | +| `index build creation` | Index build: done building; Reconciling collection and index idents; Creating profile collection | 1–2 each | I / INDEX | (also findable via `c:INDEX`) | + +**The actionable semantic-only signals:** `User assertion` (12, ASSERT), `Completed unstable checkpoint` (18, RECOVERY), and `Assertion while executing command` (2, COMMAND) — these are exactly the kind of low-severity-but-meaningful events that a blind `s:E OR s:F` keyword filter (which returned 0) hides, and that semantic search surfaces by concept. `c:ASSERT` has 24 records total; the keyword battery never queried it. + +Other semantic results confirmed/extended the keyword picture: `semantic("slow query long running operation")` → 242,792 Slow query + lifecycle ("thread awake", "Starting thread"); `semantic("WiredTiger cache eviction…")` → 48 "WiredTiger message"; `semantic("startup…")` → "MongoDB starting" + "Setting the Client" (238k). + +## 11. Cost note + +Semantic adds one embedding round-trip per `semantic()` query (7 here) on top of the keyword battery. The local in-process cache is unavailable on clp-core 0.12.1 (flag unsupported), so each semantic query hits the remote endpoint. On a clp-core build that supports `--semantic-cache-dir`, repeated semantic queries would hit in-process. + +## 12. Follow-up + +1. `clp-s-search-kql --projection t.$date,c,msg ARCHIVE 'c:ASSERT'` — pull all 24 ASSERT records semantic surfaced (the "User assertion" / "Assertion while executing command" tail). +2. `clp-s-search-kql --projection t.$date,msg ARCHIVE 'c:RECOVERY'` — the "Completed unstable checkpoint" events with timestamps. +3. `clp-s-search-kql --projection t.$date,msg,attr.remote ARCHIVE 'semantic("connection failures") AND c:NETWORK'` — the 6 network-error sessions with their remote endpoints. \ No newline at end of file diff --git a/mongodb-skills-eval/mongodb-skills-comparison-report.md b/mongodb-skills-eval/mongodb-skills-comparison-report.md new file mode 100644 index 0000000..37fd798 --- /dev/null +++ b/mongodb-skills-eval/mongodb-skills-comparison-report.md @@ -0,0 +1,136 @@ +# Comparative Review: Four MongoDB Insight Skills + +A side-by-side evaluation of **`mongodb-grep`**, **`mongodb-kql`**, **`mongodb-semantic`**, and **`logtype-insights`**, each run against the **same input**: a 7-file, 1.88 GB subset of `~/clp-demo-mongodb` (4,986,512 `mongod` records, ~5h13m, a YCSB benchmark on a single MongoDB 6.0.5 primary). The four per-skill reports live alongside this one in `mongodb-skills-eval/`. + +Mirrors the previous vLLM four-skill evaluation, now on MongoDB logs with the logtype skill generalized and caching-enabled. + +## What changed in this revision (vs the first review) + +Three issues raised after the first review were addressed: +1. **No token count → subagent model fixed.** The first review ran inline because every Agent spawn failed with `unknown model group: yscope-default-subagent`. Root cause: `CLAUDE_CODE_SUBAGENT_MODEL=yscope-default-subagent` in `/home/robin/.ccs/yscope-default.settings.json`, but only `yscope-default-{haiku,opus,sonnet}` are valid model groups. **Fixed** the settings file → `yscope-default-haiku`. The current session's env is frozen at launch, so full subagent-run token totals still require a session restart to take effect; this revision measures subagent **input-prompt** token costs directly via the API instead (§5). +2. **Repo updated.** Merged `origin/main` (commit `5b95220` — clp+ skills, `dev` skill, `references/`, `--experimental`/`--clp-s-bin` wrapper options). No conflicts with local work. +3. **Semantic search fixed.** It was broken (`Unknown OUTPUT_HANDLER`). Root cause: the wrapper passed `--semantic-cache-dir`/`--semantic-cache-cold-capacity` that clp-core 0.12.1 doesn't support. **Fixed** the wrapper to probe `clp-s s --help` and skip those flags when unsupported. Semantic now works and finds signals the keyword battery missed (§3.2, §4). + +## Method & caveats + +1. **Subagent delegation** was unavailable this session (env frozen). All four skills' prescribed query batteries were executed **inline** (codex-style); the subagent config is fixed for the next session. Token cost is therefore reported as **subagent input-prompt tokens** measured via the API (§5), not full subagent-run totals. +2. **Scale:** the full corpus is 65 GB / 186M records (vs 57 MB in the vLLM eval). Skills ran on a representative 7-file / 5M-record subset; full-corpus ground truth gathered earlier (186M records, 9.25M "Slow query", 5.2M REPL, 0 E/F) is consistent with the subset. +3. **A wrapper bug was found and fixed mid-run** (§3.1). CLP-skill numbers are from the corrected, full-data archive. + +## 1. Verdict (up front) + +| Skill | Setup | Coverage | Finding quality | Determinism | Cost | Status | +|---|---|---|---|---|---|---| +| `logtype-insights` | CLP compress + baseline + cache | 100% (after fix) | **Best** — 199-template spine, grounded queries, reusable cached classification | High (semantic optional, unused) | ~1 dump + handful of targeted queries | ✅ Works; cache amortizes | +| `mongodb-semantic` | CLP compress + KQL + semantic | 100% (after fixes) | **Good + semantic-only signals** — finds D1/D2 assertions/checkpoints keyword misses | Medium (endpoint; cache unavailable on 0.12.1) | keyword + 7 semantic round-trips | ✅ Works (after wrapper compat fix) | +| `mongodb-kql` | CLP compress (keyword only) | 100% (after fix) | Good — exact, matches grep; precise severity | High (no network) | compress + ~20 cheap queries | ✅ Works (after wrapper fix) | +| `mongodb-grep` | none (raw jq/grep) | 100% (all files, always) | Good — but 44 false-positive "exception" hits | High (no network) | ~13 jq passes over 1.88 GB (slowest) | ✅ Works; only zero-setup option | + +**Headline (revised):** `logtype-insights` remains the strongest (grounded, waste-free, reusable cached classification). `mongodb-semantic` is no longer broken — after the wrapper compat fix it works and is the **only** skill that surfaced the D1/D2 `User assertion` (12), `Completed unstable checkpoint` (18), and `Assertion while executing command` (2) signals that `s:E OR s:F` (=0) hides. `mongodb-kql` is the reliable keyword workhorse. `mongodb-grep` is the zero-setup but slower/noisier option. + +## 2. The four skills at a glance (by design) + +| Dimension | `mongodb-grep` | `mongodb-kql` | `mongodb-semantic` | `logtype-insights` | +|---|---|---|---|---| +| Tooling | `jq`/`grep` on raw JSON | CLP compress + keyword KQL | CLP compress + KQL + `semantic()` | CLP compress + `stats.logtypes` baseline + classify + cache | +| `msg` field | plain text (no limit) | clp-string (`msg:term`→0; project+grep) | clp-string; semantic reaches it | clp-string; baseline reads it directly | +| Hard constraint | no CLP | no semantic | none | semantic optional (not used here) | +| Reusable artifact | none | none | none | **cached classification** (per-app) | + +## 3. Critical findings & fixes + +### 3.1 `clp-s-compress-folder` silently compressed only the FIRST file of a multi-file folder — found & fixed +- **Symptom:** compressing the 7-file/1.88 GB subset produced a 1.16 MB archive with **683,022 records** — exactly file 1 (14% of the data). Files 2–7 silently dropped (1/260 = 0.4% of the full corpus). +- **Root cause:** the wrapper wrote the file list **null-delimited** (`find -print0`), but `clp-s -f` expects **newline-delimited** → clp-s read only the first null-terminated entry. **Proof:** newline list of 2 files → 1,359,713 records; null list of the same 2 files → 683,022. +- **Why the vLLM eval missed it:** it compressed a single 57 MB file. +- **Fix applied:** one null→newline conversion in `clp-s-compress-folder` (after the discovery/structurize paths). After the fix, the same subset compresses to 9.39 MB with all **4,986,512 records** (verified by decompress-count and the last file's 04:47 timestamp). The `--structurize` path shares the bug and is fixed by the same conversion. This affected all CLP skills (incl. the vLLM CLP skills). + +### 3.2 `mongodb-semantic` was broken by a wrapper/clp-s flag incompatibility — fixed +- **Symptom:** every `semantic("…")` query failed with `Unknown OUTPUT_HANDLER: `. +- **Root cause:** the wrapper passed `--semantic-cache-dir`/`--semantic-cache-cold-capacity` by default; clp-core **0.12.1** does **not** support those flags (absent from `clp-s s --help`), causing a positional misparse. (Same class of bug the vLLM eval's `vllm-insights` hit, §4.4.) +- **Fix applied to `clp-s-search-kql`:** probe `clp-s s --help` for `--semantic-cache-dir`; if unsupported, skip the cache flags (remote-only `/v1/similarity`) with a warning. Semantic now works by default on 0.12.1 and keeps the local cache on newer binaries that support it. +- **Result:** semantic search works and produced **semantic-only findings** (§4) — `User assertion` (12, ASSERT), `Completed unstable checkpoint` (18, RECOVERY), `Assertion while executing command` (2, COMMAND), all at D1/D2 severity invisible to `s:E OR s:F`. + +### 3.3 "Slow query" ≠ slow (debug-verbosity semantics) — corrected in the skills +`msg:"Slow query"` (id 51803) is emitted for **every** operation (242,792 here) at debug COMMAND verbosity; `attr.durationMillis` is 0–3ms (p95=1) with one 237ms outlier. The skills were corrected to filter by `id:51803` and report the workload (verb / `attr.ns` / `attr.planSummary` / `attr.remote` / index effectiveness), not just a >100ms threshold. `logtype-insights` needs no such fix — it discovers the `Slow query` template from the baseline. + +### 3.4 `msg` clp-string; `attr.*` leaves searchable; `attr` object is not +`msg:term`/`msg:*term*` → 0 (project+grep). Nested leaves are KQL-searchable (`attr.ns:ycsb.usertable`→238,013, `attr.planSummary:IDHACK`→91,843, `attr.remote:*`). Projecting the `attr` **object** returns `{}` — must project leaves. Verb counted via existence `attr.command.:*` (the object can't be projected). + +### 3.5 `logtype-insights` classification cache works on real data +5M-record archive: schema discovery → `stats.logtypes` = **199 templates** → `app_key = sha256(sorted templates)` → **cache MISS** → classified + stored → **immediate re-get = HIT**. The one-time classification is reusable across future captures of the same MongoDB 6.0.5 app. This realizes the amortization the vLLM `logtypes` skill only claimed. + +### 3.6 Subagent model config — fixed (takes effect next session) +`CLAUDE_CODE_SUBAGENT_MODEL` was `yscope-default-subagent` (unmapped). Changed to `yscope-default-haiku` in `/home/robin/.ccs/yscope-default.settings.json`. The current session's env is frozen at launch, so subagent spawning still fails here; a session restart activates it. (Note: this gateway routes `yscope-default-haiku` → `nemotron-3-nano:30b-cloud`, so future subagent-run token totals reflect Nemotron-30B, not the vLLM eval's Haiku — not directly comparable.) + +### 3.7 Repo updated +Merged `origin/main` (`5b95220`: clp+ skills, `dev` skill, `references/`, `--experimental`/`--clp-s-bin` wrapper options, `--projection` repeatable, `--project` removed). No conflicts with the local MongoDB-skill work or the wrapper fixes above. + +## 4. Finding-by-finding coverage (same 4,986,512-record subset) + +| Finding (ground truth) | grep | kql | semantic | logtype | +|---|---|---|---|---| +| Total 4,986,512 | ✅ | ✅ (after fix) | ✅ | ✅ | +| Severity (D3-dominant, 0 E/F) | ✅ | ✅ | ✅ | ✅ | +| Errors/fatals = 0 | ✅ | ✅ exact | ✅ | ✅ | +| Free-text "exception" matches | ⚠ 44 false positives | — | — | — | +| Warnings (3 startupWarnings) | ✅ | ✅ | ✅ | ✅ | +| **User assertion (12, ASSERT, D1)** | ❌ | ❌ | ✅ **semantic-only** | ❌ | +| **Completed unstable checkpoint (18, RECOVERY, D2)** | ❌ | ❌ | ✅ **semantic-only** | ❌ | +| **Assertion while executing command (2, COMMAND, D1)** | ❌ | ❌ | ✅ **semantic-only** | ❌ | +| "Slow query" = every op (debug verbosity) | ✅ | ✅ | ✅ | ✅ (template) | +| Slow-query duration dist + workload | ✅ | ✅ | ✅ | ✅ | +| Write-concern waits 150,750 | ✅ | ✅ | ✅ | ✅ (template) | +| Elections = 0 | ✅ | ✅ | ✅ | ✅ | +| 199-template dictionary + spine | ⚠ templatize only | ⚠ templatize only | ⚠ templatize only | ✅ **`stats.logtypes`** | +| Classification cache (reusable) | ❌ | ❌ | ❌ | ✅ **proven** | + +**Coverage (revised):** `logtype` best (template spine + cache); `semantic` now adds genuine value (the only skill surfacing D1/D2 assertions/checkpoints); `kql` and `grep` are strong on the structured signals but blind to those low-severity conceptual events. `grep` uniquely mis-fires on free-text exceptions (44 false positives vs exact `s:E/F`=0). + +## 5. Token cost (subagent input-prompt tokens, measured via the API) + +Full subagent-run totals (à la the vLLM eval's 62k–150k across tool rounds) require the subagent fix to take effect after a session restart. What's measurable now is the **subagent input-prompt token cost** — the prompt each skill's Haiku subagent receives — measured by sending each skill's prompt template (with the real data the subagent would be pasted) to `/v1/messages` (`max_tokens=1`) and reading `usage.input_tokens`: + +| Skill | Subagent input-prompt tokens | Notes | +|---|---|---| +| `mongodb-grep` | ~3,551 | fixed query-sequence prompt | +| `mongodb-kql` | ~3,570 | fixed query-sequence prompt | +| `mongodb-semantic` | ~2,877 | fixed prompt (extraction slightly shorter) | +| `logtype-insights` — classify phase | ~7,445 | includes the **199-template baseline** pasted in | +| `logtype-insights` — insight phase | ~1,060 | taxonomy + query plan pasted in | + +**Reading:** the three blind-battery skills have comparable ~3k input-prompt cost. `logtype-insights` splits into a one-time **classify** pass (~7.4k, includes the 199-template baseline) and a recurring **insight** pass (~1.1k). On a **cache hit** (same app, run ≥2), the classify pass is skipped → recurring input-prompt cost drops to ~1.1k — the cheapest, mirroring the vLLM eval's amortization finding. These are input-prompt only; the full run adds output + tool-round I/O (the vLLM eval's totals were ~20–40× the prompt size due to tool rounds), so treat these as relative prompt-cost, not full-run totals. + +## 6. Cost / determinism + +| | grep | kql | semantic | logtype | +|---|---|---|---|---| +| Compress | none | ~1–2 min → 9.4 MB (200×) | same | same | +| Query cost | ~13 `jq` passes over 1.88 GB (slowest) | ~20 queries on 9.4 MB (fast) | same + 7 semantic endpoint round-trips | 1 `stats.logtypes` + ~5 targeted (fastest per analysis) | +| Network | none | none | required (remote-only on 0.12.1) | none | +| Determinism | high | high | medium | high | +| Reusable artifact | none | none | none | **cached classification** | + +## 7. Skill deficiencies & fixes + +1. **`clp-s-compress-folder` null-delimiter bug** (§3.1) — **fixed**. Shared wrapper bug affecting all CLP skills. +2. **`clp-s-search-kql` semantic cache-flag incompat** (§3.2) — **fixed** (probe + skip on unsupported binaries). +3. **`mongodb-grep` free-text exception grep over-counts** (44 payload false positives). Fix: lead with `s:E OR s:F` and treat the free-text grep as secondary. +4. **`attr` object projection returns `{}`** — always project `attr.` paths. +5. **`--count` not exposed by the wrapper** — counting needs `grep -c '^{'` over dumped records (fine for small sets, expensive for whole-archive distributions on huge archives). +6. **`logtype-insights` cache key is exact-match** on the template set — a shorter capture (subset of templates) misses; future: key on the logger set + incrementally classify new templates. +7. **Subagent config** (§3.6) — **fixed** in settings; takes effect next session. + +## 8. Recommendations by use case + +| Situation | Use | Why | +|---|---|---| +| Deepest grounded report; CLP available | **`logtype-insights`** | 199-template spine, no blind queries, reusable cached classification, no endpoint dependency | +| Surface low-severity assertions/checkpoints `s:E/F` hides | **`mongodb-semantic`** | the only skill that found the D1/D2 ASSERT/RECOVERY signals; needs the (now-fixed) endpoint | +| Fast deterministic keyword pass | **`mongodb-kql`** | exact counts, precise severity, no network | +| No CLP, quick raw read | **`mongodb-grep`** | zero setup; mind free-text false positives + multi-GB jq cost | +| Repeatedly analyzing the same MongoDB app | **`logtype-insights`** (cache) | one-time classification amortizes away | + +## 9. One-line summary + +> On a 5M-record / 1.88 GB MongoDB (YCSB) subset, `logtype-insights` gave the deepest grounded report (199-template baseline + reusable cached classification, proven miss→store→hit); `mongodb-semantic` — after a wrapper fix that stops `clp-s-search-kql` passing `--semantic-cache-dir` to clp-core 0.12.1 (which doesn't support it) — now works and was the **only** skill to surface the D1/D2 `User assertion` (12), `Completed unstable checkpoint` (18), and `Assertion while executing command` (2) signals hidden by `s:E OR s:F`=0; `mongodb-kql` was the reliable keyword workhorse; and `mongodb-grep` was the zero-setup but slower, noisier option (44 free-text false positives). The review also found and **fixed** a critical shared-wrapper bug (`clp-s-compress-folder` passed a null-delimited file list to `clp-s -f`, silently archiving only the first file of any multi-file folder — 14% here, 0.4% of the full corpus — masked in the vLLM eval by its single-file input), **fixed** the subagent model config (was an unmapped `yscope-default-subagent` group), and **updated** the repo to `origin/main` (`5b95220`). Subagent input-prompt token costs were measured (~3k for grep/kql/semantic; ~7.4k one-time + ~1.1k recurring for logtype, dropping to ~1.1k on a cache hit); full subagent-run totals need a session restart to activate the subagent fix. \ No newline at end of file diff --git a/plugins/clp/README.md b/plugins/clp/README.md index bc569dc..496790f 100644 --- a/plugins/clp/README.md +++ b/plugins/clp/README.md @@ -13,6 +13,7 @@ The plugin exposes only: - list recent Claude Code and Codex session JSONL files. - compress one selected session with `clp-s c --timestamp-key timestamp`. +- compress log files from an arbitrary folder with `clp-s c --remove-path-prefix FOLDER -f FILE_LIST OUTPUT_DIR`. - search local CLP archives with KQL (including `semantic("query")`) and stdout results. - decompress a local CLP archive directory. @@ -25,6 +26,7 @@ metadata sinks, or arbitrary `clp-s` option passthrough. | Skill | Scope | | --- | --- | | `compress` | Compress a session JSONL file into a CLP archive directory. | +| `compress-folder` | Compress log files from an arbitrary folder into a CLP archive directory. | | `search` | Search CLP archives with KQL, including `semantic("query")`. | | `decompress` | Decompress a CLP archive directory for raw inspection. | | `claude-code-trajectory` | End-to-end Claude Code session analysis: list → compress → search → decompress, plus Claude-specific query starters. | @@ -65,6 +67,7 @@ claude --plugin-dir ./plugins/clp - `bin/clp-s-list-sessions` - `bin/clp-s-compress-session` +- `bin/clp-s-compress-folder` - `bin/clp-s-search-kql` - `bin/clp-s-decompress` @@ -122,6 +125,49 @@ wrappers resolve the inner `clp-s` archive directory automatically. Metadata in `.yscope-clp-archive.json` maps archive to session file, agent, roots, timestamp key, SHA-256, compression stats, command, and resolved inner archive. +## Folder Logs + +Compress log files from an arbitrary folder: + +```bash +./plugins/clp/bin/clp-s-compress-folder --folder /var/log/myapp +``` + +Defaults: + +- extensions: `log,jsonl,json,txt,ndjson,out,err` (override with `--extensions`, + or use `--extensions '*'` to include every regular file). +- recursive: yes (use `--no-recursive` for top-level only). +- timestamp key: none (pass `--timestamp-key KEY` if your logs have a known + timestamp field; required for time-range search). +- archive root: `${TMPDIR:-/tmp}/yscope-clp-archives`. Ask only when the user + wants persistent storage or a different root. + +After compression, report: + +- `Raw input bytes` +- `Archive bytes` +- `Compression ratio` +- `File size reduction` +- `Input files` +- `Archives dir` +- `Archive metadata` + +The resulting archive is compatible with `clp-s-search-kql` and +`clp-s-decompress`. Use the printed top-level `Archives dir` for search and +decompression. Metadata in `.yscope-clp-archive.json` records the source +folder, extensions, file count, compression stats, command, and resolved inner +archive. + +Useful commands: + +```bash +./plugins/clp/bin/clp-s-compress-folder --show-archives-root +./plugins/clp/bin/clp-s-compress-folder --set-archives-root ~/clp-archives +./plugins/clp/bin/clp-s-compress-folder --folder /var/log/myapp --dry-run +./plugins/clp/bin/clp-s-compress-folder --folder ./logs --extensions log,txt +``` + ## Search ```bash diff --git a/plugins/clp/bin/clp-s-compress-folder b/plugins/clp/bin/clp-s-compress-folder new file mode 100755 index 0000000..b5f0f94 --- /dev/null +++ b/plugins/clp/bin/clp-s-compress-folder @@ -0,0 +1,567 @@ +#!/usr/bin/env bash +set -euo pipefail + +CLP_PLUGIN_BIN_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck disable=SC1091 +source "${CLP_PLUGIN_BIN_DIR}/lib/clp-common.sh" + +usage() { + cat <<'EOF' +Usage: + clp-s-compress-folder --folder PATH [options] + +Compress log files from an arbitrary folder into a searchable CLP archive +directory: + + clp-s c --remove-path-prefix FOLDER -f FILE_LIST OUTPUT_DIR + +Options: + --folder PATH Directory containing log files (required). + --extensions EXT,EXT,... Comma-separated extension filter. + Default: log,jsonl,json,txt,ndjson,out,err + Use '*' to include every regular file. + --no-recursive Search only the top level of --folder. + --timestamp-key KEY Pass --timestamp-key KEY to clp-s. + No default; only set this if your logs have a + known timestamp field. + --compression-level LEVEL Pass --compression-level LEVEL to clp-s. + --target-encoded-size N Pass --target-encoded-size N to clp-s. + --output-dir DIR Top-level archive directory. Overrides + --archives-root and saved settings. + --archives-root DIR Parent directory for auto-named archive outputs. + Default order: this option, CLP_S_ARCHIVES_ROOT, + saved config, then ${TMPDIR:-/tmp}/yscope-clp-archives + --save-archives-root Save --archives-root as the future default. + --set-archives-root DIR Save DIR as the future default and exit. + --show-archives-root Print the current archive root setting and exit. + --print-archive-stats Pass --print-archive-stats to clp-s. + --structurize Pre-process files through structurize.py to + convert unstructured text logs into structured + JSONL before compression. Automatically sets + --timestamp-key timestamp. + --dry-run Print the planned compression operation only. + -h, --help Show this help. + +Examples: + clp-s-compress-folder --folder /var/log/myapp + clp-s-compress-folder --folder ./logs --extensions log,txt + clp-s-compress-folder --folder ./logs --extensions '*' --dry-run + clp-s-compress-folder --folder /var/log/myapp --timestamp-key ts + clp-s-compress-folder --show-archives-root + clp-s-compress-folder --set-archives-root ~/clp-archives +EOF +} + +DEFAULT_EXTENSIONS="log,jsonl,json,txt,ndjson,out,err" + +folder="" +extensions="$DEFAULT_EXTENSIONS" +recursive=1 +timestamp_key="" +compression_level="" +target_encoded_size="" +output_dir="" +archives_root="" +archives_root_source="" +archives_root_explicit=0 +resolved_archives_root="" +save_archives_root=0 +set_archives_root="" +show_archives_root=0 +print_archive_stats=0 +dry_run=0 +structurize=0 +structurize_tmpdir="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --folder) + [[ $# -ge 2 ]] || { echo "error: --folder requires a value" >&2; exit 2; } + folder="$2" + shift 2 + ;; + --extensions) + [[ $# -ge 2 ]] || { echo "error: --extensions requires a value" >&2; exit 2; } + extensions="$2" + shift 2 + ;; + --no-recursive) + recursive=0 + shift + ;; + --timestamp-key) + [[ $# -ge 2 ]] || { echo "error: --timestamp-key requires a value" >&2; exit 2; } + timestamp_key="$2" + shift 2 + ;; + --compression-level) + [[ $# -ge 2 ]] || { echo "error: --compression-level requires a value" >&2; exit 2; } + compression_level="$2" + shift 2 + ;; + --target-encoded-size) + [[ $# -ge 2 ]] || { echo "error: --target-encoded-size requires a value" >&2; exit 2; } + target_encoded_size="$2" + shift 2 + ;; + --output-dir|--archives-dir) + [[ $# -ge 2 ]] || { echo "error: $1 requires a value" >&2; exit 2; } + output_dir="$2" + shift 2 + ;; + --archives-root) + [[ $# -ge 2 ]] || { echo "error: --archives-root requires a value" >&2; exit 2; } + archives_root="$2" + archives_root_explicit=1 + shift 2 + ;; + --save-archives-root) + save_archives_root=1 + shift + ;; + --set-archives-root) + [[ $# -ge 2 ]] || { echo "error: --set-archives-root requires a value" >&2; exit 2; } + set_archives_root="$2" + shift 2 + ;; + --show-archives-root) + show_archives_root=1 + shift + ;; + --print-archive-stats) + print_archive_stats=1 + shift + ;; + --dry-run) + dry_run=1 + shift + ;; + --structurize) + structurize=1 + shift + ;; + -h|--help) + usage + exit 0 + ;; + --) + echo "error: arbitrary clp-s option passthrough is not exposed by this wrapper" >&2 + exit 2 + ;; + *) + echo "error: unknown argument: $1" >&2 + echo >&2 + usage >&2 + exit 2 + ;; + esac +done + +resolve_archives_root() { + local configured + + if [[ -n "$archives_root" ]]; then + archives_root_source="argument" + resolved_archives_root="$(canonicalize_output_path "$archives_root")" + return $? + fi + if [[ -n "${CLP_S_ARCHIVES_ROOT:-}" ]]; then + archives_root_source="env" + resolved_archives_root="$(canonicalize_output_path "$CLP_S_ARCHIVES_ROOT")" + return $? + fi + if configured="$(read_configured_archives_root 2>/dev/null)"; then + archives_root_source="config" + resolved_archives_root="$(canonicalize_output_path "$configured")" + return $? + fi + + archives_root_source="default" + resolved_archives_root="$(canonicalize_output_path "${TMPDIR:-/tmp}/yscope-clp-archives")" +} + +configure_archives_root() { + local requested="$1" + local resolved + + resolved="$(canonicalize_output_path "$requested")" || { + echo "error: archive root parent directory does not exist: $requested" >&2 + exit 1 + } + if is_broad_output_dir "$resolved"; then + echo "error: refusing broad archive root: $resolved" >&2 + exit 2 + fi + mkdir -p "$resolved" + write_configured_archives_root "$resolved" + echo "Archives root: $resolved" + echo "Archives root source: config" + echo "Config file: $(clp_archives_root_config_file)" +} + +if [[ -n "$set_archives_root" ]]; then + configure_archives_root "$set_archives_root" + exit 0 +fi + +if [[ "$save_archives_root" -eq 1 && "$archives_root_explicit" -eq 0 ]]; then + echo "error: --save-archives-root requires --archives-root DIR" >&2 + exit 2 +fi + +if [[ "$show_archives_root" -eq 1 ]]; then + resolve_archives_root || { + echo "error: archive root parent directory does not exist" >&2 + exit 1 + } + echo "Archives root: $resolved_archives_root" + echo "Archives root source: $archives_root_source" + echo "Config file: $(clp_archives_root_config_file)" + exit 0 +fi + +# --- Validate required arguments --- + +if [[ -z "$folder" ]]; then + echo "error: --folder PATH is required" >&2 + echo >&2 + usage >&2 + exit 2 +fi + +if [[ ! -d "$folder" ]]; then + echo "error: --folder is not a directory: $folder" >&2 + exit 1 +fi +if [[ ! -r "$folder" ]]; then + echo "error: --folder is not readable: $folder" >&2 + exit 1 +fi + +folder_real="$(realpath "$folder")" +folder_name="$(basename "$folder_real")" + +# --- Validate extensions --- + +all_files=0 +if [[ "$extensions" == "*" ]]; then + all_files=1 +else + # Validate each extension is alphanumeric (no wildcards, paths, or dots) + IFS=',' read -ra ext_array <<< "$extensions" + for ext in "${ext_array[@]}"; do + if [[ -z "$ext" ]]; then + echo "error: empty extension in --extensions list" >&2 + exit 2 + fi + if [[ "$ext" =~ [^a-zA-Z0-9_] ]]; then + echo "error: invalid extension (use only alphanumeric/underscore): $ext" >&2 + exit 2 + fi + done +fi + +# --- Resolve output directory --- + +if [[ -n "$output_dir" && "$save_archives_root" -eq 1 ]]; then + echo "error: --save-archives-root cannot be used with --output-dir" >&2 + exit 2 +fi + +if [[ -n "$output_dir" ]]; then + output_dir="$(canonicalize_output_path "$output_dir")" || { + echo "error: output directory parent does not exist: $output_dir" >&2 + exit 1 + } + if is_broad_output_dir "$output_dir"; then + echo "error: refusing broad output directory: $output_dir" >&2 + exit 2 + fi +else + run_stamp="$(date -u +%Y%m%dT%H%M%SZ)" + resolve_archives_root || { + echo "error: archive root parent directory does not exist" >&2 + exit 1 + } + if is_broad_output_dir "$resolved_archives_root"; then + echo "error: refusing broad archive root: $resolved_archives_root" >&2 + exit 2 + fi + if [[ "$save_archives_root" -eq 1 ]]; then + mkdir -p "$resolved_archives_root" + write_configured_archives_root "$resolved_archives_root" + fi + output_dir="${resolved_archives_root}/folder-${folder_name}-${run_stamp}" +fi + +# --- Discover files --- + +file_list="$(mktemp "${TMPDIR:-/tmp}/clp-s-files.XXXXXX")" +trap 'rm -f "$file_list"; [[ -n "${structurize_tmpdir:-}" ]] && rm -rf "${structurize_tmpdir}"' EXIT + +if [[ "$all_files" -eq 1 ]]; then + if [[ "$recursive" -eq 1 ]]; then + find "$folder_real" -type f -print0 > "$file_list" + else + find "$folder_real" -maxdepth 1 -type f -print0 > "$file_list" + fi +else + # Build find extension predicates (case-insensitive) + find_args=("$folder_real") + if [[ "$recursive" -eq 0 ]]; then + find_args+=(-maxdepth 1) + fi + find_args+=(-type f \() + first=1 + for ext in "${ext_array[@]}"; do + if [[ "$first" -eq 1 ]]; then + find_args+=(-iname "*.${ext}") + first=0 + else + find_args+=(-o -iname "*.${ext}") + fi + done + find_args+=(\)) + # shellcheck disable=SC2086 + find "${find_args[@]}" -print0 > "$file_list" +fi + +file_count="$(tr '\0' '\n' < "$file_list" | grep -c . || true)" +if [[ "$file_count" -eq 0 ]]; then + echo "error: no log files matched in $folder_real" >&2 + if [[ "$all_files" -eq 1 ]]; then + echo " (searched all files, recursive=$recursive)" >&2 + else + echo " (extensions: $extensions, recursive=$recursive)" >&2 + fi + exit 1 +fi + +# --- Structurize unstructured logs --- + +if [[ "$structurize" -eq 1 ]]; then + structurize_script="${CLP_PLUGIN_BIN_DIR}/structurize.py" + if [[ ! -f "$structurize_script" ]]; then + echo "error: --structurize requires structurize.py but it was not found: $structurize_script" >&2 + exit 1 + fi + if ! command -v python3 >/dev/null 2>&1; then + echo "error: --structurize requires python3 but it is not installed" >&2 + exit 1 + fi + + structurize_tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/clp-s-structurize.XXXXXX")" + structurize_ok=0 + structurize_skip=0 + new_file_list="$(mktemp "${TMPDIR:-/tmp}/clp-s-files.XXXXXX")" + original_folder_real="$folder_real" + + while IFS= read -r -d '' src_path; do + src_name="$(basename "$src_path")" + dst_path="${structurize_tmpdir}/${src_name}.json" + if python3 "$structurize_script" "$src_path" "$dst_path" >/dev/null 2>&1; then + if [[ -s "$dst_path" ]]; then + printf '%s\0' "$dst_path" >> "$new_file_list" + structurize_ok=$((structurize_ok + 1)) + else + echo "warning: structurize produced empty output for $src_name, skipping" >&2 + structurize_skip=$((structurize_skip + 1)) + fi + else + echo "warning: structurize failed for $src_name, skipping" >&2 + structurize_skip=$((structurize_skip + 1)) + fi + done < "$file_list" + + if [[ "$structurize_ok" -eq 0 ]]; then + echo "error: structurize did not produce any output files" >&2 + rm -f "$new_file_list" + exit 1 + fi + + if [[ "$structurize_skip" -gt 0 ]]; then + echo "warning: structurize skipped $structurize_skip file(s); proceeding with $structurize_ok structured file(s)" >&2 + fi + + # Replace file list with structurize output + rm -f "$file_list" + file_list="$new_file_list" + + # Point remove-path-prefix at the structurize temp dir so archive paths + # show just the filename (e.g. "app.log.json") instead of a deep tmp path + folder_real="$structurize_tmpdir" + + # Recompute file count from the structurized list + file_count="$structurize_ok" + + # Auto-set timestamp key for structured JSONL output + if [[ -z "$timestamp_key" ]]; then + timestamp_key="timestamp" + fi + + echo "Structurize: converted $structurize_ok file(s) to structured JSONL" + if [[ "$structurize_skip" -gt 0 ]]; then + echo "Structurize skipped: $structurize_skip file(s)" + fi +fi + +# clp-s -f expects a NEWLINE-delimited file list. The list above is +# null-delimited (built with `find -print0` / `printf '%s\0'` to survive +# filenames containing spaces). Convert it to newline-delimited here so +# clp-s ingests EVERY file — otherwise it reads only the first null-terminated +# entry and silently drops the rest of the folder. (Log filenames do not +# contain newlines, so this conversion is safe.) +nl_file_list="$(mktemp "${TMPDIR:-/tmp}/clp-s-files.XXXXXX")" +tr '\0' '\n' < "$file_list" > "$nl_file_list" +rm -f "$file_list" +file_list="$nl_file_list" + +# Compute total bytes +total_bytes=0 +while IFS= read -r path; do + size="$(wc -c < "$path" | tr -d '[:space:]')" + total_bytes=$((total_bytes + size)) +done < "$file_list" + +# --- Build clp-s command --- + +clp_s_bin="$(resolve_clp_s)" || exit $? + +cmd=("$clp_s_bin" c) +if [[ -n "$timestamp_key" ]]; then + cmd+=(--timestamp-key "$timestamp_key") +fi +if [[ -n "$compression_level" ]]; then + cmd+=(--compression-level "$compression_level") +fi +if [[ -n "$target_encoded_size" ]]; then + cmd+=(--target-encoded-size "$target_encoded_size") +fi +if [[ "$print_archive_stats" -eq 1 ]]; then + cmd+=(--print-archive-stats) +fi +cmd+=(--remove-path-prefix "$folder_real") +cmd+=(-f "$file_list" "$output_dir") + +# --- Plan / dry-run output --- + +echo "Folder: $folder_real" +echo "Extensions: $extensions" +echo "Recursive: $([ "$recursive" -eq 1 ] && echo 'yes' || echo 'no')" +echo "Input files: $file_count" +echo "Input bytes: $total_bytes" +if [[ -n "${resolved_archives_root:-}" ]]; then + echo "Archives root: $resolved_archives_root" + echo "Archives root source: $archives_root_source" +fi +echo "Archives dir: $output_dir" +echo "Timestamp key: ${timestamp_key:-none}" +echo "Structurize: $([ "$structurize" -eq 1 ] && echo 'yes' || echo 'no')" +echo "Archive mode: regular directory archive (--single-file-archive disabled)" +printf 'Command:' +printf ' %q' "${cmd[@]}" +echo + +if [[ "$dry_run" -eq 1 ]]; then + echo "Dry run only; no archive was created." + exit 0 +fi + +# --- Run compression --- + +mkdir -p "$output_dir" +"${cmd[@]}" + +# --- Compute stats and write metadata --- + +archive_bytes="$(directory_file_bytes "$output_dir")" +compression_ratio="$(awk -v raw="$total_bytes" -v archive="$archive_bytes" 'BEGIN { if (archive > 0) printf "%.2fx", raw / archive; else printf "n/a" }')" +reduction_bytes=$((total_bytes - archive_bytes)) +reduction_percent="$(awk -v raw="$total_bytes" -v reduction="$reduction_bytes" 'BEGIN { if (raw > 0) printf "%.2f%%", reduction * 100 / raw; else printf "n/a" }')" +clp_archive_dir="$(resolve_clp_s_archive_dir "$output_dir" 2>/dev/null || true)" + +# Build extensions JSON array +if [[ "$all_files" -eq 1 ]]; then + extensions_json='"*"' +else + extensions_json="$(printf '%s\n' "${ext_array[@]}" | jq -R . | jq -s .)" +fi + +# Preserve the original source path for metadata when structurize is used, +# since folder_real has been repointed at the structurize temp directory. +if [[ "$structurize" -eq 1 ]]; then + metadata_source_path="$original_folder_real" +else + metadata_source_path="$folder_real" +fi + +metadata_file="$(clp_archive_metadata_file "$output_dir")" +command_json="$(printf '%s\n' "${cmd[@]}" | jq -R . | jq -s .)" +jq -n \ + --arg schemaVersion "1" \ + --arg createdAt "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg plugin "yscope-clp" \ + --arg sourceType "folder" \ + --arg sourcePath "$metadata_source_path" \ + --arg sourceName "$folder_name" \ + --argjson sourceRecursive "$recursive" \ + --argjson sourceExtensions "$extensions_json" \ + --argjson sourceFileCount "$file_count" \ + --arg archiveRoot "${resolved_archives_root:-}" \ + --arg archiveRootSource "${archives_root_source:-output-dir}" \ + --arg archiveDir "$output_dir" \ + --arg clpArchiveDir "$clp_archive_dir" \ + --arg timestampKey "${timestamp_key:-}" \ + --argjson inputBytes "$total_bytes" \ + --argjson archiveBytes "$archive_bytes" \ + --argjson reductionBytes "$reduction_bytes" \ + --arg compressionRatio "$compression_ratio" \ + --arg reductionPercent "$reduction_percent" \ + --argjson command "$command_json" \ + --argjson structurizeFlag "$structurize" \ + '{ + schemaVersion: ($schemaVersion | tonumber), + createdAt: $createdAt, + plugin: $plugin, + source: { + type: $sourceType, + path: $sourcePath, + name: $sourceName, + recursive: $sourceRecursive, + extensions: $sourceExtensions, + fileCount: $sourceFileCount + }, + archiveRoot: (if $archiveRoot == "" then null else $archiveRoot end), + archiveRootSource: $archiveRootSource, + archiveDir: $archiveDir, + clpArchiveDir: (if $clpArchiveDir == "" then null else $clpArchiveDir end), + timestampKey: (if $timestampKey == "" then null else $timestampKey end), + structurize: ($structurizeFlag == 1), + compression: { + rawBytes: $inputBytes, + archiveBytes: $archiveBytes, + ratio: $compressionRatio, + reductionBytes: $reductionBytes, + reductionPercent: $reductionPercent + }, + command: $command + }' > "$metadata_file" + +echo "Raw input bytes: $total_bytes" +echo "Archive bytes: $archive_bytes" +echo "Compression ratio: $compression_ratio" +echo "File size reduction: $reduction_bytes bytes ($reduction_percent)" +echo "Input files: $file_count" +if [[ "$structurize" -eq 1 ]]; then + echo "Structurize: yes (converted to structured JSONL)" +fi +if [[ -n "$clp_archive_dir" && "$clp_archive_dir" != "$output_dir" ]]; then + echo "Resolved clp-s archive dir: $clp_archive_dir" +fi +echo "Archive metadata: $metadata_file" +echo "Archive entries:" +while IFS= read -r -d '' entry; do + entry_type="$(file_type_letter "$entry")" + entry_size="$(file_size_bytes "$entry" 2>/dev/null || printf '0')" + printf '%s\t%s\t%s bytes\n' "$entry" "$entry_type" "$entry_size" +done < <(find_immediate_children "$output_dir") | LC_ALL=C sort \ No newline at end of file diff --git a/plugins/clp/bin/clp-s-search-kql b/plugins/clp/bin/clp-s-search-kql index d02e523..f3883bd 100755 --- a/plugins/clp/bin/clp-s-search-kql +++ b/plugins/clp/bin/clp-s-search-kql @@ -295,9 +295,18 @@ if [[ -n "$archive_id" ]]; then fi if [[ "$semantic_active" -eq 1 ]]; then search_options+=(--semantic-endpoint "$semantic_endpoint") + # Only pass the local-cache flags when this clp-s binary supports them. + # Older builds (e.g. clp-core 0.12.1) lack --semantic-cache-dir and abort + # with "Unknown OUTPUT_HANDLER: " when it is passed, which breaks + # semantic search entirely. Probe the binary's --help and fall back to + # remote-only /v1/similarity (no local cache) when the flag is unsupported. if [[ "$semantic_cache_dir" != "none" ]]; then - search_options+=(--semantic-cache-dir "$semantic_cache_dir") - search_options+=(--semantic-cache-cold-capacity "$semantic_cache_cold_capacity") + if grep -q -- '--semantic-cache-dir' <("$clp_s_bin" s --help 2>&1); then + search_options+=(--semantic-cache-dir "$semantic_cache_dir") + search_options+=(--semantic-cache-cold-capacity "$semantic_cache_cold_capacity") + else + echo "warning: clp-s does not support --semantic-cache-dir; using remote-only semantic search (no local cache)" >&2 + fi fi fi if [[ -n "$semantic_top_k" ]]; then diff --git a/plugins/clp/bin/logtype-cache b/plugins/clp/bin/logtype-cache new file mode 100755 index 0000000..e9ef429 --- /dev/null +++ b/plugins/clp/bin/logtype-cache @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +""" +logtype-cache - persistent cache for the logtype-insights skill's template +classification. + +The classification step (grouping the archive's logtype templates into +categories and deriving a targeted query plan) is reusable across runs of the +same application: the same app build emits the same message templates, so the +classification is paid once and amortized over every future analysis of that +app. This helper stores and retrieves those classifications. + +Cache key (the "app fingerprint") is sha256 of the sorted set of distinct +logtype strings from `clp-s-search-kql ARCHIVE 'stats.logtypes'`. Same app +build -> same templates -> same key -> cache hit -> classification skipped. + +Cache layout: + /.json +where each entry is the classification JSON produced by the classification +subagent, e.g.: + { + "app_key": "...", + "schema": {"timestamp": "t.$date", "severity": "s", ...}, + "taxonomy": [{"category": "errors", "description": "..."}], + "templates": [{"logtype": "...", "category": "errors"}], + "query_plan": [{"label": "Errors", "kql": "s:E OR s:F", "method": "count"}], + "classified_at": "2026-08-07T..." + } + +Subcommands: + key --logtypes-file F Print the app_key for the given stats.logtypes + NDJSON file (one {"id":N,"logtype":"..."} per + line). Empty/whitespace logtypes are skipped. + get KEY [--cache-dir D] Print cached JSON to stdout (exit 0); exit 1 + on miss. + put KEY [--cache-dir D] Read classification JSON from stdin and store + it under /KEY.json (atomic write). + Stamps/refreshes `classified_at`. + list [--cache-dir D] List cached entries: key, template count, + classified_at. + show KEY [--cache-dir D] Pretty-print one cached entry. + +--cache-dir overrides the default +~/.config/yscope-clp-plugin/logtype-cache (or $CLP_LOGTYPE_CACHE_DIR). +""" +import argparse +import hashlib +import json +import os +import sys +import tempfile +from datetime import datetime, timezone + + +def default_cache_dir(): + env = os.environ.get("CLP_LOGTYPE_CACHE_DIR") + if env: + return os.path.expanduser(env) + return os.path.expanduser("~/.config/yscope-clp-plugin/logtype-cache") + + +def load_logtypes(path): + """Return the sorted set of distinct logtype strings from a stats.logtypes + NDJSON file. Returns None if the file has no parseable logtype lines.""" + logtypes = set() + with open(path, "r", encoding="utf-8") as f: + for line in f: + line = line.strip() + if not line or not line.startswith("{"): + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + lt = obj.get("logtype") + if isinstance(lt, str) and lt.strip(): + logtypes.add(lt) + return logtypes + + +def app_key_for_logtypes(logtypes): + h = hashlib.sha256() + for lt in sorted(logtypes): + h.update(lt.encode("utf-8")) + h.update(b"\n") + return h.hexdigest() + + +def cmd_key(args): + if not os.path.exists(args.logtypes_file): + print(f"error: logtypes file not found: {args.logtypes_file}", file=sys.stderr) + return 2 + logtypes = load_logtypes(args.logtypes_file) + if not logtypes: + print("error: no logtype entries found in file", file=sys.stderr) + return 1 + print(app_key_for_logtypes(logtypes)) + return 0 + + +def entry_path(cache_dir, key): + return os.path.join(cache_dir, f"{key}.json") + + +def cmd_get(args): + path = entry_path(args.cache_dir, args.key) + if not os.path.exists(path): + return 1 + with open(path, "r", encoding="utf-8") as f: + sys.stdout.write(f.read()) + if not sys.stdout.isatty(): + # Ensure trailing newline for pipelines; the stored file already ends + # with one from put(), so nothing extra is needed. + pass + return 0 + + +def cmd_put(args): + raw = sys.stdin.read() + try: + obj = json.loads(raw) + except json.JSONDecodeError as e: + print(f"error: stdin is not valid JSON: {e}", file=sys.stderr) + return 2 + if not isinstance(obj, dict): + print("error: classification JSON must be a JSON object", file=sys.stderr) + return 2 + obj["app_key"] = args.key + obj["classified_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + os.makedirs(args.cache_dir, exist_ok=True) + path = entry_path(args.cache_dir, args.key) + # Atomic write: temp file in the same dir, then rename. + fd, tmp = tempfile.mkstemp(prefix=".tmp-", suffix=".json", dir=args.cache_dir) + try: + with os.fdopen(fd, "w", encoding="utf-8") as f: + json.dump(obj, f, indent=2, ensure_ascii=False) + f.write("\n") + os.replace(tmp, path) + except BaseException: + try: + os.unlink(tmp) + except OSError: + pass + raise + print(f"Stored classification for app_key {args.key} -> {path}", file=sys.stderr) + return 0 + + +def cmd_list(args): + if not os.path.isdir(args.cache_dir): + print(f"(cache dir does not exist: {args.cache_dir})", file=sys.stderr) + return 0 + rows = [] + for name in sorted(os.listdir(args.cache_dir)): + if not name.endswith(".json"): + continue + key = name[:-5] + path = os.path.join(args.cache_dir, name) + try: + with open(path, "r", encoding="utf-8") as f: + obj = json.load(f) + except (OSError, json.JSONDecodeError): + continue + templates = obj.get("templates", []) + classified_at = obj.get("classified_at", "?") + rows.append((key, len(templates) if isinstance(templates, list) else "?", classified_at)) + if not rows: + print(f"(no cached classifications in {args.cache_dir})", file=sys.stderr) + return 0 + print(f"{'app_key':<64} templates classified_at") + for key, n, ts in rows: + print(f"{key:<64} {str(n):>9} {ts}") + return 0 + + +def cmd_show(args): + path = entry_path(args.cache_dir, args.key) + if not os.path.exists(path): + print(f"error: no cache entry for key {args.key}", file=sys.stderr) + return 1 + with open(path, "r", encoding="utf-8") as f: + obj = json.load(f) + print(json.dumps(obj, indent=2, ensure_ascii=False)) + return 0 + + +def main(argv=None): + p = argparse.ArgumentParser( + prog="logtype-cache", + description="Persistent classification cache for the logtype-insights skill.", + ) + sub = p.add_subparsers(dest="command", required=True) + + pk = sub.add_parser("key", help="Compute the app_key for a stats.logtypes NDJSON file.") + pk.add_argument("--logtypes-file", required=True, help="Path to a stats.logtypes NDJSON file.") + pk.set_defaults(func=cmd_key, cache_dir=None) + + pg = sub.add_parser("get", help="Print a cached classification (exit 1 on miss).") + pg.add_argument("key", help="app_key (from the `key` subcommand).") + pg.set_defaults(func=cmd_get) + + pp = sub.add_parser("put", help="Store a classification read from stdin.") + pp.add_argument("key", help="app_key (from the `key` subcommand).") + pp.set_defaults(func=cmd_put) + + pl = sub.add_parser("list", help="List cached classifications.") + pl.set_defaults(func=cmd_list) + + ps = sub.add_parser("show", help="Pretty-print one cached classification.") + ps.add_argument("key", help="app_key (from the `key` subcommand).") + ps.set_defaults(func=cmd_show) + + # Shared --cache-dir for get/put/list/show (key doesn't need it). + for subp in (pg, pp, pl, ps): + subp.add_argument( + "--cache-dir", + default=default_cache_dir(), + help="Cache directory (default: %(default)s, or $CLP_LOGTYPE_CACHE_DIR).", + ) + + args = p.parse_args(argv) + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) \ No newline at end of file diff --git a/plugins/clp/bin/structurize.py b/plugins/clp/bin/structurize.py new file mode 100644 index 0000000..0ba0153 --- /dev/null +++ b/plugins/clp/bin/structurize.py @@ -0,0 +1,134 @@ +import json +import re +import os +import datetime + +def convert_vllm_log_to_json(input_filepath, output_filepath): + # Regex to match the outer wrapper log format + # Example: 2026-06-09 10:02:41,887 - sflow.task.vllm_worker_3 - INFO - 0: ... + outer_pattern = re.compile( + r'^(?P\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s+-\s+' + r'(?P[\w\.]+)\s+-\s+' + r'(?P\w+)\s+-\s+' + r'(?P\d+):\s+' + r'(?P.*)$' + ) + + # Regex to catch inner duplicate timestamps generated by the internal vLLM logger + # Matches formats like: + # 1. ISO: "2026-06-09T17:02:52.521329Z DEBUG __init__..." + # 2. Short: "DEBUG 06-09 10:02:45 [plugins/..." + inner_timestamp_pattern = re.compile( + r'^(?:' + r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z\s+(?PINFO|DEBUG|WARN|WARNING|ERROR)\s+|' + r'(?PINFO|DEBUG|WARN|WARNING|ERROR)\s+\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s+' + r')(?P.*)$' + ) + + # Regex to match RAW vLLM engine logs that have NO outer wrapper (e.g. logs + # captured straight from `vllm serve` in CI, without an sflow-style wrapper). + # Matches, with an optional leading ISO timestamp (GitHub Actions line prefix) + # and/or an optional "(Component pid=N)" prefix: + # INFO 06-30 03:55:00 [importing.py:81] Triton not installed ... + # (APIServer pid=20533) INFO 06-30 04:12:03 [launcher.py:46] Route: /v1/... + # 2026-06-30T03:55:12.7099980Z INFO 06-30 03:55:00 [importing.py:81] ... + raw_vllm_pattern = re.compile( + r'^(?:(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+)?' + r'(?P\([A-Za-z_][A-Za-z0-9_]*(?:\s+pid=\d+)?\)\s+)?' + r'(?PINFO|WARNING|ERROR|DEBUG|CRITICAL)\s+' + r'(?P\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}(?:\.\d+)?)\s+' + r'(?P\[[^\]]+\])(?:\s+(?P.*))?$' + ) + + structured_logs = [] + seen_timestamps = set() + + # Raw vLLM engine timestamps have no year ("06-15 03:52:34"), which CLP + # cannot parse. Infer a year: prefer a leading ISO timestamp on the line, + # then a YYYY-MM-DD date in the filename, else the current year. The + # resulting timestamp is normalized to "YYYY-MM-DD HH:MM:SS,mmm". + fn_year_match = re.search(r'(\d{4})-\d{2}-\d{2}', os.path.basename(input_filepath)) + fallback_year = (fn_year_match.group(1) if fn_year_match + else str(datetime.datetime.now().year)) + + def _normalize_raw_ts(raw_ts, lead_ts): + year = lead_ts[:4] if lead_ts else fallback_year + ts = f"{year}-{raw_ts}" # raw_ts == "MM-DD HH:MM:SS" or "MM-DD HH:MM:SS.frac" + if '.' in ts: + ts = ts.replace('.', ',', 1) # -> "YYYY-MM-DD HH:MM:SS,frac" + else: + ts = ts + ',000' + return ts + + with open(input_filepath, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if not line: + continue + + outer_match = outer_pattern.match(line) + if outer_match: + log_entry = outer_match.groupdict() + raw_message = log_entry.pop('message') + timestamp = log_entry['timestamp'] + + # Optional: If you want to completely skip new lines that have the exact + # same wrapper timestamp as a previous line, uncomment the following block: + # if timestamp in seen_timestamps: + # continue + # seen_timestamps.add(timestamp) + + # Check for duplicate internal timestamps inside the message + inner_match = inner_timestamp_pattern.match(raw_message) + if inner_match: + # Update the level to the inner vLLM core level (e.g. wrapper might say INFO, but core says DEBUG) + inner_level = inner_match.group('level_iso') or inner_match.group('level_short') + log_entry['level'] = inner_level.strip() + # Keep only the clean message, dropping the redundant inner timestamp + log_entry['message'] = inner_match.group('inner_message').strip() + else: + log_entry['message'] = raw_message + + structured_logs.append(log_entry) + else: + raw_match = raw_vllm_pattern.match(line) + if raw_match: + component = (raw_match.group('component') or '').strip() + message = raw_match.group('raw_message') or '' + if component: + message = f"{component} {message}".strip() + structured_logs.append({ + 'timestamp': _normalize_raw_ts( + raw_match.group('raw_ts'), raw_match.group('lead_ts')), + 'logger': raw_match.group('raw_logger').strip('[]'), + 'level': raw_match.group('raw_level'), + 'message': message, + }) + else: + # Handle multi-line strings (like large JSON dumps) by appending to the previous message + if structured_logs: + structured_logs[-1]['message'] += '\n' + line + + # Write out as JSON Lines (JSONL) + with open(output_filepath, 'w', encoding='utf-8') as out_f: + for log in structured_logs: + out_f.write(json.dumps(log) + '\n') + + if len(structured_logs) == 0: + print(f"Warning: no structured log entries produced from {input_filepath}.", file=sys.stderr) + sys.exit(1) + + print(f"Successfully processed {len(structured_logs)} structured log entries.") + +if __name__ == "__main__": + # Replace these filenames with your actual paths + INPUT_LOG_FILE = 'vllm-log.txt' + OUTPUT_JSON_FILE = 'vllm-structured-log.json' + # take the input and output file paths from command line arguments if provided + import sys + if len(sys.argv) > 1: + INPUT_LOG_FILE = sys.argv[1] + if len(sys.argv) > 2: + OUTPUT_JSON_FILE = sys.argv[2] + # usage: python structurize.py input_log.txt output_log.json + convert_vllm_log_to_json(INPUT_LOG_FILE, OUTPUT_JSON_FILE) \ No newline at end of file diff --git a/plugins/clp/skills-claude/compress-folder/SKILL.md b/plugins/clp/skills-claude/compress-folder/SKILL.md new file mode 100644 index 0000000..0299ca9 --- /dev/null +++ b/plugins/clp/skills-claude/compress-folder/SKILL.md @@ -0,0 +1,124 @@ +--- +name: compress-folder +description: Compress log files from an arbitrary folder into a searchable CLP archive directory. +allowed-tools: ["Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder:*)"] +--- + +# Compress Folder + +Use only the plugin wrappers. Do not call bare `clp-s` or expose arbitrary CLP +commands/options. + +## Rules + +- Compress log files from one folder. Do not use this skill for session JSONL + files; use `compress` for sessions. +- Do not pass `--single-file-archive`; search uses regular archive directories. +- `--timestamp-key` has no default. Only pass it when the user says their logs + have a known timestamp field. Omit it otherwise — `clp-s` will still + compress and search, but time-range flags (`--tge`/`--tle`) will not work. +- Default file extensions: `log`, `jsonl`, `json`, `txt`, `ndjson`, `out`, + `err`. Override with `--extensions`. +- Default archive root: `${TMPDIR:-/tmp}/yscope-clp-archives`. +- Ask about archive location only if the user wants persistent storage or a + change. + +## Structurize (Unstructured Text Logs) + +Use `--structurize` when compressing **unstructured text logs** — plain-text +log files that lack a regular structured format (e.g. vLLM wrapper logs, +application logs with interleaved timestamps and messages). The flag runs each +input file through `bin/structurize.py`, which parses out timestamp, logger, +level, worker, and message fields and writes structured JSONL. This gives +`clp-s` proper timestamp extraction and better compression. + +When `--structurize` is active: + +- Each input file is converted to a `.json` sidecar in a temp directory. +- `--timestamp-key timestamp` is set automatically (do not override it). +- Files that structurize cannot parse are skipped with a warning. +- The archive's `source.path` metadata still records the original folder path. + +Do **not** use `--structurize` for files that are already structured +(JSON, JSONL, NDJSON) — structurize is designed for unstructured text formats. + +## Workflow + +1. If the user does not specify a folder, ask for one. + +2. Run compression: + + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" --folder /path/to/logs + ``` + + Override extensions or add a timestamp key as needed: + + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/logs \ + --extensions log,txt \ + --timestamp-key ts + ``` + + For unstructured text logs (vLLM logs, plain-text app logs): + + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/logs \ + --structurize + ``` + +3. After compression, always report: + + - `Raw input bytes` + - `Archive bytes` + - `Compression ratio` + - `File size reduction` + - `Input files` + - `Archives dir` + - `Archive metadata` + +4. Use the printed top-level `Archives dir` for search and decompression. The + wrappers resolve the inner `clp-s` archive directory automatically. + +## Useful Commands + +Show archive root: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" --show-archives-root +``` + +Set persistent archive root: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" --set-archives-root ~/clp-archives +``` + +Dry run: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/logs \ + --dry-run +``` + +Compress only top-level `.log` files with a timestamp field: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /var/log/myapp \ + --extensions log \ + --no-recursive \ + --timestamp-key timestamp +``` + +Compress unstructured text logs (e.g. vLLM wrapper logs): + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /var/log/vllm \ + --extensions log,txt \ + --structurize +``` \ No newline at end of file diff --git a/plugins/clp/skills-claude/logtype-insights/SKILL.md b/plugins/clp/skills-claude/logtype-insights/SKILL.md new file mode 100644 index 0000000..f5f06c4 --- /dev/null +++ b/plugins/clp/skills-claude/logtype-insights/SKILL.md @@ -0,0 +1,529 @@ +--- +name: logtype-insights +description: App-agnostic logtype-baseline log analysis with CLP. Dump the archive's logtype dictionary first, classify the real templates into (generic + app-discovered) categories, and drive targeted KQL from them — no blind queries. Caches the classification so re-analyzing the same application skips it. Works on any structurized or native-JSON CLP archive (vLLM, MongoDB, nginx, …). +allowed-tools: + - "Agent" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-decompress:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache:*)" + - "Bash(jq:*)" + - "Bash(grep:*)" + - "Bash(sort:*)" + - "Bash(uniq:*)" + - "Bash(head:*)" + - "Bash(tail:*)" + - "Bash(cat:*)" + - "Bash(wc:*)" + - "Bash(sed:*)" +--- + +# Logtype Insights (App-Agnostic, Logtype-Baseline) + +End-to-end analysis of **any** CLP archive — structurized text logs (vLLM +wrapper logs → `timestamp/logger/level/message`), native JSON logs (MongoDB → +`t.$date/s/c/msg/attr`), or other JSON — using the **logtype baseline** method: +dump the archive's logtype dictionary first, classify those *real* message +templates into categories, and derive every later query from a template that is +guaranteed to exist. No blind keyword batteries, no queries wasted on keywords +that aren't there. + +This is the generalized successor to the old vLLM-only `vllm-insights-logtypes` +skill. The logtype method is **not application-specific**: the dictionary dump, +the generic category taxonomy, the classification cache, and the project+grep +retrieval pattern all work on any archive. The only thing that changes between +applications is the set of templates — which the skill reads from the archive +itself rather than guessing. + +For a single ad-hoc KQL query, use the `search` skill. For app-specific +batteries on vLLM logs use `vllm-insights`/`vllm-kql`; on MongoDB use +`mongodb-semantic`/`mongodb-kql`. + +## Why a logtype baseline beats blind search + +A CLP logtype is a message template with variables replaced by `<*>`, e.g. +`Triton not installed or not compatible; certain GPU-related functions ...` or +MongoDB's `Slow query`, `attr.durationMillis=<*>`. The logtype dictionary is the +**complete vocabulary** of distinct message shapes in the archive — for a +typical run, tens to a few hundred templates, no matter how many millions of +records. Dumping it gives you, in one cheap pass that reads the dictionary +rather than every record: + +- Every kind of event the run actually produced (no guessing keywords). +- The static tokens of each template, which you turn into queries that always + match — so counts are exact and zero queries return zero by surprise. +- A natural unit for "top repeated messages": frequency per template. + +The blind variants (`*-insights`, `*-kql`) run a fixed battery of hardcoded +queries; on an unfamiliar archive many return nothing. This skill runs **1 dump ++ schema discovery + a handful of targeted queries**, each grounded in a real +template. + +## Why the classification is cached + +Classifying the templates into categories and deriving a query plan is the one +expensive step, and it is a property of the **application**, not the individual +capture: the same app build emits the same message templates on every run, so +the same classification applies. This skill caches the classification keyed by a +fingerprint of the template set (`sha256` of the sorted logtypes). On a cache +hit (same app), classification is skipped entirely and the skill goes straight +to the insight pass with the pre-made plan — so re-analyzing the same +application costs only the cheap Haiku insight pass, not the classification. + +## Supported inputs + +- A CLP archive directory (any kind). Primary input. +- A folder of raw logs — compress first with the app-appropriate settings, since + compression is the one app-specific step: + - vLLM wrapper text logs: `--structurize` (produces `timestamp/logger/level/message`). + - MongoDB JSON: `--extensions '*' --timestamp-key t.$date` (native). + - Generic JSON with a known timestamp field: `--timestamp-key `. + - Then point this skill at the resulting archive. +- If nothing was provided, ask for an archive or folder path. + +## Workflow + +1. Determine the input: + - If the user provided an archive path, use it. + - If the user provided a folder, compress it with the app-appropriate + settings (above) and use the resulting archive. If the app is unknown, ask + the user how the logs should be compressed (structurize vs native + `--timestamp-key`), or have them compress first and pass the archive. + - If nothing was provided, ask for an archive or folder path. + +2. Report compression stats when you compressed the folder: + - `Raw input bytes`, `Archive bytes`, `Compression ratio`, + `File size reduction`, `Input files`, `Archives dir`, `Archive metadata`. + +3. **Discover the schema** (cheap; do this in the parent). A no-projection + search returns the full original record, so one sample line reveals the + field names: + + ```bash + ARCHIVE= + SEARCH="${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" + + # One full record (reveals the JSON keys / structurized fields): + "$SEARCH" "$ARCHIVE" '*' 2>/dev/null | grep '^{' | head -1 + ``` + + Identify and record, as `schema`, the field names for: + - **timestamp** — e.g. `timestamp` (vLLM structurized) or `t.$date` (Mongo). + If it is a real epoch (native JSON), `--tge`/`--tle` work; if it is a + structurized string, they do not. + - **severity** — e.g. `level` (vLLM) or `s` (Mongo). + - **logger/component** — e.g. `logger` (vLLM) or `c` (Mongo). + - **message** — the clp-string field whose logtypes appear in + `stats.logtypes` — e.g. `message` (vLLM) or `msg` (Mongo). + - **payload** (optional) — e.g. `attr` (Mongo); note the useful leaf paths + (e.g. `attr.durationMillis`, `attr.host`). + Also note the distinct values of the severity and logger fields (one count + query each) so the classifier and insight pass can use the real vocabularies: + ```bash + "$SEARCH" --projection "$ARCHIVE" '*' | grep '^{' | jq -r '.' | sort | uniq -c | sort -rn + "$SEARCH" --projection "$ARCHIVE" '*' | grep '^{' | jq -r '.' | sort | uniq -c | sort -rn + ``` + +4. **Dump the logtype baseline** (cheap; reads the dictionary, not every record): + + ```bash + # Canonical template dictionary — one JSON object per logtype: + # {"id":0,"logtype":"Triton not installed ... <*> functions ..."} + "$SEARCH" "$ARCHIVE" 'stats.logtypes' > /tmp/logtypes.ndjson 2>/tmp/logtypes.err + + # Summary: how many distinct templates, and the templates themselves. + jq -s 'length' /tmp/logtypes.ndjson + jq -r '.logtype' /tmp/logtypes.ndjson + ``` + + **Fallback if `stats.logtypes` emits no NDJSON** (some builds only print a + `[stats]` dictionary-size summary to stderr). If `/tmp/logtypes.ndjson` has + zero JSON lines, build an approximate baseline by projecting the message + field for all records and templatizing the variable runs (O(records), but + produces templates AND counts in one pass): + + ```bash + MSG= # e.g. message (vLLM) or msg (Mongo) + "$SEARCH" --projection "$MSG" "$ARCHIVE" '*' \ + | grep '^{' | jq -r --arg f "$MSG" '.[$f]' \ + | sed -E 's/\{[^}]+\}/<*>/g; s/0x[0-9a-fA-F]+/<*>/g; s/\b[0-9]+\b/<*>/g' \ + | sort | uniq -c | sort -rn > /tmp/logtype-freqs.txt + ``` + + (The wrapper prints archive-metadata header lines to stdout, so `grep '^{'` + filters to JSON records before `jq` — same idiom as `grep -c '^{'` for + counting.) + +5. **Classification cache lookup** (cheap; parent): + + ```bash + CACHE="${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache" + APP_KEY="$("$CACHE" key --logtypes-file /tmp/logtypes.ndjson)" + # If you used the templatize fallback instead, build the key from it: + # APP_KEY="$(sort /tmp/logtype-freqs.txt | ... )" # see logtype-cache `key` + + if CLASSIFICATION="$("$CACHE" get "$APP_KEY" 2>/dev/null)"; then + echo "CACHE HIT — reusing cached classification for $APP_KEY" + echo "$CLASSIFICATION" > /tmp/logtype-classification.json + else + echo "CACHE MISS — will classify and store for $APP_KEY" + fi + ``` + + On a **cache hit**, verify the cached `schema` field matches the schema you + discovered in step 3 (same field names). If it matches, skip to step 7 — the + classification is reused as-is and the expensive classification subagent is + skipped entirely. If the schema differs, treat it as a miss (reclassify). + +6. **(Cache miss only) Classify the templates.** Spawn a **classification + subagent** with the Agent tool, model `sonnet` (fall back to `haiku`). It + takes the baseline templates + the discovered schema + the generic taxonomy, + classifies each template, discovers any app-specific categories, builds a + query plan expressed in the discovered field names, and writes the result as + structured JSON to `/tmp/logtype-classification.json`. + + Classification subagent prompt template (fill in `ARCHIVE`, the `schema`, and + paste the baseline from step 4): + + ``` + You are classifying the logtype templates of a CLP archive so a later insight + pass can run targeted queries. Do NOT write the final report — only the + classification JSON. + + Archive: ARCHIVE + Discovered schema (field names in this archive): + timestamp: + severity: + logger: + message: (the clp-string field whose logtypes these are) + payload: (leaf paths if any, e.g. attr.durationMillis) + Severity values seen: + Logger values seen: + + LOGTYPE BASELINE (every distinct message template in the archive; <*> marks + variables). Classify each into a category and derive a query plan from its + static tokens. + + + Method: + 1. Classify each template into the best-fitting category. Use this GENERIC + default taxonomy, AND add any APP-SPECIFIC categories you discover from the + templates (e.g. for MongoDB: workload/operations (slow query, write-concern + waits), replication/election, sharding, indexing, WiredTiger/storage; for + vLLM: worker-health, kv-cache, model-loading). Name + app-specific categories descriptively. + Generic defaults: + - errors / exceptions / failures + - warnings + - performance (latency / throughput / timing / "took <*> ms") + - config / startup / initialization + - network / connectivity / timeout + - resource (memory / disk / file-descriptors / storage pressure) + - lifecycle / state-transitions (start/stop/election/stepdown/restart) + - security / auth / access + - other (note but don't deep-search) + 2. Build a QUERY PLAN: a list of targeted queries, each derived from one or + more real templates, expressed in the discovered field names. For each plan + entry give: label, the KQL filter (using the searchable scalar fields — + severity/logger/payload leaves; NOT message:term which is a clp-string and + returns 0), the columns to --projection, and the method: + - "count" -> count matches via `... | grep -c '^{'` + - "project+grep" -> project the message field and grep its static text + - "project+jq" -> project message/payload and jq-filter (e.g. a + numeric threshold on a payload leaf) + - "semantic" -> semantic("...") AND , ONLY for an + ambiguous template or to group similar ones + Example plan entry (Mongo schema): + {"label":"Slow queries","kql":"attr.durationMillis:*", + "project":"t.$date,attr.durationMillis,msg", + "jq":"select((.attr.durationMillis//0)>100)","method":"project+jq"} + Example plan entry (vLLM schema): + {"label":"Memory warnings","kql":"level:WARNING", + "project":"timestamp,level,message","grep":"memory|OOM|KV", + "method":"project+grep"} + 3. Total records: '*'. Severity breakdown: one count per severity value seen. + Logger breakdown: project logger + uniq -c. Time span: project the + timestamp field and use head/tail (records are chronological; do NOT sort). + 4. Remember: the message field is a clp-string. KQL `message:term` / + `msg:term` and `message:*term*` / `msg:*term*` return 0. Only the scalar + fields (severity, logger, payload leaves) are KQL-searchable. Retrieve + message content by projecting the message field and grepping. + + Write the result as valid JSON to /tmp/logtype-classification.json with EXACTLY + this shape, then print "DONE" and nothing else: + { + "app_key": "", + "schema": {"timestamp":"","severity":"","logger":"","message":"","payload":["",...]}, + "taxonomy": [{"category":"","description":""}], + "templates": [{"logtype":"