From db06dda005725a21b65b4e83ba78a20fa875faa3 Mon Sep 17 00:00:00 2001 From: Robin Li Date: Fri, 26 Jun 2026 07:18:19 -0400 Subject: [PATCH 1/5] feat(clp): add compress-folder skill and vllm-insights skill - Add clp-s-compress-folder wrapper script for compressing arbitrary folder logs (not just sessions) with --structurize and --timestamp-key support - Add structurize.py for vLLM-style log structuring (timestamp, logger, level, worker, message fields) - Add compress-folder skills for both Claude and Codex agents - Add vllm-insights skills for both Claude and Codex agents with KQL query starters, analysis patterns, and report format - Update README with folder logs documentation and skill table entries - Add bin/clp-s binary artifact to .gitignore --- .gitignore | 1 + plugins/clp/README.md | 46 ++ plugins/clp/bin/clp-s-compress-folder | 557 ++++++++++++++++++ plugins/clp/bin/structurize.py | 86 +++ .../skills-claude/compress-folder/SKILL.md | 124 ++++ .../clp/skills-claude/vllm-insights/SKILL.md | 209 +++++++ .../clp/skills-codex/compress-folder/SKILL.md | 129 ++++ .../clp/skills-codex/vllm-insights/SKILL.md | 195 ++++++ 8 files changed, 1347 insertions(+) create mode 100755 plugins/clp/bin/clp-s-compress-folder create mode 100644 plugins/clp/bin/structurize.py create mode 100644 plugins/clp/skills-claude/compress-folder/SKILL.md create mode 100644 plugins/clp/skills-claude/vllm-insights/SKILL.md create mode 100644 plugins/clp/skills-codex/compress-folder/SKILL.md create mode 100644 plugins/clp/skills-codex/vllm-insights/SKILL.md diff --git a/.gitignore b/.gitignore index 0d09b97..4cd31ad 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,4 @@ plugins/clp/.clp-core/ # Common local/runtime noise. .DS_Store +plugins/clp/bin/clp-s diff --git a/plugins/clp/README.md b/plugins/clp/README.md index 654177f..057a636 100644 --- a/plugins/clp/README.md +++ b/plugins/clp/README.md @@ -13,6 +13,7 @@ The plugin exposes only: - list recent Claude Code and Codex session JSONL files. - compress one selected session with `clp-s c --timestamp-key timestamp`. +- compress log files from an arbitrary folder with `clp-s c --remove-path-prefix FOLDER -f FILE_LIST OUTPUT_DIR`. - search local CLP archives with KQL (including `semantic("query")`) and stdout results. - decompress a local CLP archive directory. @@ -25,6 +26,7 @@ metadata sinks, or arbitrary `clp-s` option passthrough. | Skill | Scope | | --- | --- | | `compress` | Compress a session JSONL file into a CLP archive directory. | +| `compress-folder` | Compress log files from an arbitrary folder into a CLP archive directory. | | `search` | Search CLP archives with KQL, including `semantic("query")`. | | `decompress` | Decompress a CLP archive directory for raw inspection. | | `claude-code-trajectory` | End-to-end Claude Code session analysis: list → compress → search → decompress, plus Claude-specific query starters. | @@ -65,6 +67,7 @@ claude --plugin-dir ./plugins/clp - `bin/clp-s-list-sessions` - `bin/clp-s-compress-session` +- `bin/clp-s-compress-folder` - `bin/clp-s-search-kql` - `bin/clp-s-decompress` @@ -122,6 +125,49 @@ wrappers resolve the inner `clp-s` archive directory automatically. Metadata in `.yscope-clp-archive.json` maps archive to session file, agent, roots, timestamp key, SHA-256, compression stats, command, and resolved inner archive. +## Folder Logs + +Compress log files from an arbitrary folder: + +```bash +./plugins/clp/bin/clp-s-compress-folder --folder /var/log/myapp +``` + +Defaults: + +- extensions: `log,jsonl,json,txt,ndjson,out,err` (override with `--extensions`, + or use `--extensions '*'` to include every regular file). +- recursive: yes (use `--no-recursive` for top-level only). +- timestamp key: none (pass `--timestamp-key KEY` if your logs have a known + timestamp field; required for time-range search). +- archive root: `${TMPDIR:-/tmp}/yscope-clp-archives`. Ask only when the user + wants persistent storage or a different root. + +After compression, report: + +- `Raw input bytes` +- `Archive bytes` +- `Compression ratio` +- `File size reduction` +- `Input files` +- `Archives dir` +- `Archive metadata` + +The resulting archive is compatible with `clp-s-search-kql` and +`clp-s-decompress`. Use the printed top-level `Archives dir` for search and +decompression. Metadata in `.yscope-clp-archive.json` records the source +folder, extensions, file count, compression stats, command, and resolved inner +archive. + +Useful commands: + +```bash +./plugins/clp/bin/clp-s-compress-folder --show-archives-root +./plugins/clp/bin/clp-s-compress-folder --set-archives-root ~/clp-archives +./plugins/clp/bin/clp-s-compress-folder --folder /var/log/myapp --dry-run +./plugins/clp/bin/clp-s-compress-folder --folder ./logs --extensions log,txt +``` + ## Search ```bash diff --git a/plugins/clp/bin/clp-s-compress-folder b/plugins/clp/bin/clp-s-compress-folder new file mode 100755 index 0000000..516bb48 --- /dev/null +++ b/plugins/clp/bin/clp-s-compress-folder @@ -0,0 +1,557 @@ +#!/usr/bin/env bash +set -euo pipefail + +CLP_PLUGIN_BIN_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck disable=SC1091 +source "${CLP_PLUGIN_BIN_DIR}/lib/clp-common.sh" + +usage() { + cat <<'EOF' +Usage: + clp-s-compress-folder --folder PATH [options] + +Compress log files from an arbitrary folder into a searchable CLP archive +directory: + + clp-s c --remove-path-prefix FOLDER -f FILE_LIST OUTPUT_DIR + +Options: + --folder PATH Directory containing log files (required). + --extensions EXT,EXT,... Comma-separated extension filter. + Default: log,jsonl,json,txt,ndjson,out,err + Use '*' to include every regular file. + --no-recursive Search only the top level of --folder. + --timestamp-key KEY Pass --timestamp-key KEY to clp-s. + No default; only set this if your logs have a + known timestamp field. + --compression-level LEVEL Pass --compression-level LEVEL to clp-s. + --target-encoded-size N Pass --target-encoded-size N to clp-s. + --output-dir DIR Top-level archive directory. Overrides + --archives-root and saved settings. + --archives-root DIR Parent directory for auto-named archive outputs. + Default order: this option, CLP_S_ARCHIVES_ROOT, + saved config, then ${TMPDIR:-/tmp}/yscope-clp-archives + --save-archives-root Save --archives-root as the future default. + --set-archives-root DIR Save DIR as the future default and exit. + --show-archives-root Print the current archive root setting and exit. + --print-archive-stats Pass --print-archive-stats to clp-s. + --structurize Pre-process files through structurize.py to + convert unstructured text logs into structured + JSONL before compression. Automatically sets + --timestamp-key timestamp. + --dry-run Print the planned compression operation only. + -h, --help Show this help. + +Examples: + clp-s-compress-folder --folder /var/log/myapp + clp-s-compress-folder --folder ./logs --extensions log,txt + clp-s-compress-folder --folder ./logs --extensions '*' --dry-run + clp-s-compress-folder --folder /var/log/myapp --timestamp-key ts + clp-s-compress-folder --show-archives-root + clp-s-compress-folder --set-archives-root ~/clp-archives +EOF +} + +DEFAULT_EXTENSIONS="log,jsonl,json,txt,ndjson,out,err" + +folder="" +extensions="$DEFAULT_EXTENSIONS" +recursive=1 +timestamp_key="" +compression_level="" +target_encoded_size="" +output_dir="" +archives_root="" +archives_root_source="" +archives_root_explicit=0 +resolved_archives_root="" +save_archives_root=0 +set_archives_root="" +show_archives_root=0 +print_archive_stats=0 +dry_run=0 +structurize=0 +structurize_tmpdir="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --folder) + [[ $# -ge 2 ]] || { echo "error: --folder requires a value" >&2; exit 2; } + folder="$2" + shift 2 + ;; + --extensions) + [[ $# -ge 2 ]] || { echo "error: --extensions requires a value" >&2; exit 2; } + extensions="$2" + shift 2 + ;; + --no-recursive) + recursive=0 + shift + ;; + --timestamp-key) + [[ $# -ge 2 ]] || { echo "error: --timestamp-key requires a value" >&2; exit 2; } + timestamp_key="$2" + shift 2 + ;; + --compression-level) + [[ $# -ge 2 ]] || { echo "error: --compression-level requires a value" >&2; exit 2; } + compression_level="$2" + shift 2 + ;; + --target-encoded-size) + [[ $# -ge 2 ]] || { echo "error: --target-encoded-size requires a value" >&2; exit 2; } + target_encoded_size="$2" + shift 2 + ;; + --output-dir|--archives-dir) + [[ $# -ge 2 ]] || { echo "error: $1 requires a value" >&2; exit 2; } + output_dir="$2" + shift 2 + ;; + --archives-root) + [[ $# -ge 2 ]] || { echo "error: --archives-root requires a value" >&2; exit 2; } + archives_root="$2" + archives_root_explicit=1 + shift 2 + ;; + --save-archives-root) + save_archives_root=1 + shift + ;; + --set-archives-root) + [[ $# -ge 2 ]] || { echo "error: --set-archives-root requires a value" >&2; exit 2; } + set_archives_root="$2" + shift 2 + ;; + --show-archives-root) + show_archives_root=1 + shift + ;; + --print-archive-stats) + print_archive_stats=1 + shift + ;; + --dry-run) + dry_run=1 + shift + ;; + --structurize) + structurize=1 + shift + ;; + -h|--help) + usage + exit 0 + ;; + --) + echo "error: arbitrary clp-s option passthrough is not exposed by this wrapper" >&2 + exit 2 + ;; + *) + echo "error: unknown argument: $1" >&2 + echo >&2 + usage >&2 + exit 2 + ;; + esac +done + +resolve_archives_root() { + local configured + + if [[ -n "$archives_root" ]]; then + archives_root_source="argument" + resolved_archives_root="$(canonicalize_output_path "$archives_root")" + return $? + fi + if [[ -n "${CLP_S_ARCHIVES_ROOT:-}" ]]; then + archives_root_source="env" + resolved_archives_root="$(canonicalize_output_path "$CLP_S_ARCHIVES_ROOT")" + return $? + fi + if configured="$(read_configured_archives_root 2>/dev/null)"; then + archives_root_source="config" + resolved_archives_root="$(canonicalize_output_path "$configured")" + return $? + fi + + archives_root_source="default" + resolved_archives_root="$(canonicalize_output_path "${TMPDIR:-/tmp}/yscope-clp-archives")" +} + +configure_archives_root() { + local requested="$1" + local resolved + + resolved="$(canonicalize_output_path "$requested")" || { + echo "error: archive root parent directory does not exist: $requested" >&2 + exit 1 + } + if is_broad_output_dir "$resolved"; then + echo "error: refusing broad archive root: $resolved" >&2 + exit 2 + fi + mkdir -p "$resolved" + write_configured_archives_root "$resolved" + echo "Archives root: $resolved" + echo "Archives root source: config" + echo "Config file: $(clp_archives_root_config_file)" +} + +if [[ -n "$set_archives_root" ]]; then + configure_archives_root "$set_archives_root" + exit 0 +fi + +if [[ "$save_archives_root" -eq 1 && "$archives_root_explicit" -eq 0 ]]; then + echo "error: --save-archives-root requires --archives-root DIR" >&2 + exit 2 +fi + +if [[ "$show_archives_root" -eq 1 ]]; then + resolve_archives_root || { + echo "error: archive root parent directory does not exist" >&2 + exit 1 + } + echo "Archives root: $resolved_archives_root" + echo "Archives root source: $archives_root_source" + echo "Config file: $(clp_archives_root_config_file)" + exit 0 +fi + +# --- Validate required arguments --- + +if [[ -z "$folder" ]]; then + echo "error: --folder PATH is required" >&2 + echo >&2 + usage >&2 + exit 2 +fi + +if [[ ! -d "$folder" ]]; then + echo "error: --folder is not a directory: $folder" >&2 + exit 1 +fi +if [[ ! -r "$folder" ]]; then + echo "error: --folder is not readable: $folder" >&2 + exit 1 +fi + +folder_real="$(realpath "$folder")" +folder_name="$(basename "$folder_real")" + +# --- Validate extensions --- + +all_files=0 +if [[ "$extensions" == "*" ]]; then + all_files=1 +else + # Validate each extension is alphanumeric (no wildcards, paths, or dots) + IFS=',' read -ra ext_array <<< "$extensions" + for ext in "${ext_array[@]}"; do + if [[ -z "$ext" ]]; then + echo "error: empty extension in --extensions list" >&2 + exit 2 + fi + if [[ "$ext" =~ [^a-zA-Z0-9_] ]]; then + echo "error: invalid extension (use only alphanumeric/underscore): $ext" >&2 + exit 2 + fi + done +fi + +# --- Resolve output directory --- + +if [[ -n "$output_dir" && "$save_archives_root" -eq 1 ]]; then + echo "error: --save-archives-root cannot be used with --output-dir" >&2 + exit 2 +fi + +if [[ -n "$output_dir" ]]; then + output_dir="$(canonicalize_output_path "$output_dir")" || { + echo "error: output directory parent does not exist: $output_dir" >&2 + exit 1 + } + if is_broad_output_dir "$output_dir"; then + echo "error: refusing broad output directory: $output_dir" >&2 + exit 2 + fi +else + run_stamp="$(date -u +%Y%m%dT%H%M%SZ)" + resolve_archives_root || { + echo "error: archive root parent directory does not exist" >&2 + exit 1 + } + if is_broad_output_dir "$resolved_archives_root"; then + echo "error: refusing broad archive root: $resolved_archives_root" >&2 + exit 2 + fi + if [[ "$save_archives_root" -eq 1 ]]; then + mkdir -p "$resolved_archives_root" + write_configured_archives_root "$resolved_archives_root" + fi + output_dir="${resolved_archives_root}/folder-${folder_name}-${run_stamp}" +fi + +# --- Discover files --- + +file_list="$(mktemp "${TMPDIR:-/tmp}/clp-s-files.XXXXXX")" +trap 'rm -f "$file_list"; [[ -n "${structurize_tmpdir:-}" ]] && rm -rf "${structurize_tmpdir}"' EXIT + +if [[ "$all_files" -eq 1 ]]; then + if [[ "$recursive" -eq 1 ]]; then + find "$folder_real" -type f -print0 > "$file_list" + else + find "$folder_real" -maxdepth 1 -type f -print0 > "$file_list" + fi +else + # Build find extension predicates (case-insensitive) + find_args=("$folder_real") + if [[ "$recursive" -eq 0 ]]; then + find_args+=(-maxdepth 1) + fi + find_args+=(-type f \() + first=1 + for ext in "${ext_array[@]}"; do + if [[ "$first" -eq 1 ]]; then + find_args+=(-iname "*.${ext}") + first=0 + else + find_args+=(-o -iname "*.${ext}") + fi + done + find_args+=(\)) + # shellcheck disable=SC2086 + find "${find_args[@]}" -print0 > "$file_list" +fi + +file_count="$(tr '\0' '\n' < "$file_list" | grep -c . || true)" +if [[ "$file_count" -eq 0 ]]; then + echo "error: no log files matched in $folder_real" >&2 + if [[ "$all_files" -eq 1 ]]; then + echo " (searched all files, recursive=$recursive)" >&2 + else + echo " (extensions: $extensions, recursive=$recursive)" >&2 + fi + exit 1 +fi + +# --- Structurize unstructured logs --- + +if [[ "$structurize" -eq 1 ]]; then + structurize_script="${CLP_PLUGIN_BIN_DIR}/structurize.py" + if [[ ! -f "$structurize_script" ]]; then + echo "error: --structurize requires structurize.py but it was not found: $structurize_script" >&2 + exit 1 + fi + if ! command -v python3 >/dev/null 2>&1; then + echo "error: --structurize requires python3 but it is not installed" >&2 + exit 1 + fi + + structurize_tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/clp-s-structurize.XXXXXX")" + structurize_ok=0 + structurize_skip=0 + new_file_list="$(mktemp "${TMPDIR:-/tmp}/clp-s-files.XXXXXX")" + original_folder_real="$folder_real" + + while IFS= read -r -d '' src_path; do + src_name="$(basename "$src_path")" + dst_path="${structurize_tmpdir}/${src_name}.json" + if python3 "$structurize_script" "$src_path" "$dst_path" >/dev/null 2>&1; then + if [[ -s "$dst_path" ]]; then + printf '%s\0' "$dst_path" >> "$new_file_list" + structurize_ok=$((structurize_ok + 1)) + else + echo "warning: structurize produced empty output for $src_name, skipping" >&2 + structurize_skip=$((structurize_skip + 1)) + fi + else + echo "warning: structurize failed for $src_name, skipping" >&2 + structurize_skip=$((structurize_skip + 1)) + fi + done < "$file_list" + + if [[ "$structurize_ok" -eq 0 ]]; then + echo "error: structurize did not produce any output files" >&2 + rm -f "$new_file_list" + exit 1 + fi + + if [[ "$structurize_skip" -gt 0 ]]; then + echo "warning: structurize skipped $structurize_skip file(s); proceeding with $structurize_ok structured file(s)" >&2 + fi + + # Replace file list with structurize output + rm -f "$file_list" + file_list="$new_file_list" + + # Point remove-path-prefix at the structurize temp dir so archive paths + # show just the filename (e.g. "app.log.json") instead of a deep tmp path + folder_real="$structurize_tmpdir" + + # Recompute file count from the structurized list + file_count="$structurize_ok" + + # Auto-set timestamp key for structured JSONL output + if [[ -z "$timestamp_key" ]]; then + timestamp_key="timestamp" + fi + + echo "Structurize: converted $structurize_ok file(s) to structured JSONL" + if [[ "$structurize_skip" -gt 0 ]]; then + echo "Structurize skipped: $structurize_skip file(s)" + fi +fi + +# Convert null-delimited file list to newline-delimited for wc/calculation +# and compute total bytes +total_bytes=0 +while IFS= read -r -d '' path; do + size="$(wc -c < "$path" | tr -d '[:space:]')" + total_bytes=$((total_bytes + size)) +done < "$file_list" + +# --- Build clp-s command --- + +clp_s_bin="$(resolve_clp_s)" || exit $? + +cmd=("$clp_s_bin" c) +if [[ -n "$timestamp_key" ]]; then + cmd+=(--timestamp-key "$timestamp_key") +fi +if [[ -n "$compression_level" ]]; then + cmd+=(--compression-level "$compression_level") +fi +if [[ -n "$target_encoded_size" ]]; then + cmd+=(--target-encoded-size "$target_encoded_size") +fi +if [[ "$print_archive_stats" -eq 1 ]]; then + cmd+=(--print-archive-stats) +fi +cmd+=(--remove-path-prefix "$folder_real") +cmd+=(-f "$file_list" "$output_dir") + +# --- Plan / dry-run output --- + +echo "Folder: $folder_real" +echo "Extensions: $extensions" +echo "Recursive: $([ "$recursive" -eq 1 ] && echo 'yes' || echo 'no')" +echo "Input files: $file_count" +echo "Input bytes: $total_bytes" +if [[ -n "${resolved_archives_root:-}" ]]; then + echo "Archives root: $resolved_archives_root" + echo "Archives root source: $archives_root_source" +fi +echo "Archives dir: $output_dir" +echo "Timestamp key: ${timestamp_key:-none}" +echo "Structurize: $([ "$structurize" -eq 1 ] && echo 'yes' || echo 'no')" +echo "Archive mode: regular directory archive (--single-file-archive disabled)" +printf 'Command:' +printf ' %q' "${cmd[@]}" +echo + +if [[ "$dry_run" -eq 1 ]]; then + echo "Dry run only; no archive was created." + exit 0 +fi + +# --- Run compression --- + +mkdir -p "$output_dir" +"${cmd[@]}" + +# --- Compute stats and write metadata --- + +archive_bytes="$(directory_file_bytes "$output_dir")" +compression_ratio="$(awk -v raw="$total_bytes" -v archive="$archive_bytes" 'BEGIN { if (archive > 0) printf "%.2fx", raw / archive; else printf "n/a" }')" +reduction_bytes=$((total_bytes - archive_bytes)) +reduction_percent="$(awk -v raw="$total_bytes" -v reduction="$reduction_bytes" 'BEGIN { if (raw > 0) printf "%.2f%%", reduction * 100 / raw; else printf "n/a" }')" +clp_archive_dir="$(resolve_clp_s_archive_dir "$output_dir" 2>/dev/null || true)" + +# Build extensions JSON array +if [[ "$all_files" -eq 1 ]]; then + extensions_json='"*"' +else + extensions_json="$(printf '%s\n' "${ext_array[@]}" | jq -R . | jq -s .)" +fi + +# Preserve the original source path for metadata when structurize is used, +# since folder_real has been repointed at the structurize temp directory. +if [[ "$structurize" -eq 1 ]]; then + metadata_source_path="$original_folder_real" +else + metadata_source_path="$folder_real" +fi + +metadata_file="$(clp_archive_metadata_file "$output_dir")" +command_json="$(printf '%s\n' "${cmd[@]}" | jq -R . | jq -s .)" +jq -n \ + --arg schemaVersion "1" \ + --arg createdAt "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg plugin "yscope-clp" \ + --arg sourceType "folder" \ + --arg sourcePath "$metadata_source_path" \ + --arg sourceName "$folder_name" \ + --argjson sourceRecursive "$recursive" \ + --argjson sourceExtensions "$extensions_json" \ + --argjson sourceFileCount "$file_count" \ + --arg archiveRoot "${resolved_archives_root:-}" \ + --arg archiveRootSource "${archives_root_source:-output-dir}" \ + --arg archiveDir "$output_dir" \ + --arg clpArchiveDir "$clp_archive_dir" \ + --arg timestampKey "${timestamp_key:-}" \ + --argjson inputBytes "$total_bytes" \ + --argjson archiveBytes "$archive_bytes" \ + --argjson reductionBytes "$reduction_bytes" \ + --arg compressionRatio "$compression_ratio" \ + --arg reductionPercent "$reduction_percent" \ + --argjson command "$command_json" \ + --argjson structurizeFlag "$structurize" \ + '{ + schemaVersion: ($schemaVersion | tonumber), + createdAt: $createdAt, + plugin: $plugin, + source: { + type: $sourceType, + path: $sourcePath, + name: $sourceName, + recursive: $sourceRecursive, + extensions: $sourceExtensions, + fileCount: $sourceFileCount + }, + archiveRoot: (if $archiveRoot == "" then null else $archiveRoot end), + archiveRootSource: $archiveRootSource, + archiveDir: $archiveDir, + clpArchiveDir: (if $clpArchiveDir == "" then null else $clpArchiveDir end), + timestampKey: (if $timestampKey == "" then null else $timestampKey end), + structurize: ($structurizeFlag == 1), + compression: { + rawBytes: $inputBytes, + archiveBytes: $archiveBytes, + ratio: $compressionRatio, + reductionBytes: $reductionBytes, + reductionPercent: $reductionPercent + }, + command: $command + }' > "$metadata_file" + +echo "Raw input bytes: $total_bytes" +echo "Archive bytes: $archive_bytes" +echo "Compression ratio: $compression_ratio" +echo "File size reduction: $reduction_bytes bytes ($reduction_percent)" +echo "Input files: $file_count" +if [[ "$structurize" -eq 1 ]]; then + echo "Structurize: yes (converted to structured JSONL)" +fi +if [[ -n "$clp_archive_dir" && "$clp_archive_dir" != "$output_dir" ]]; then + echo "Resolved clp-s archive dir: $clp_archive_dir" +fi +echo "Archive metadata: $metadata_file" +echo "Archive entries:" +while IFS= read -r -d '' entry; do + entry_type="$(file_type_letter "$entry")" + entry_size="$(file_size_bytes "$entry" 2>/dev/null || printf '0')" + printf '%s\t%s\t%s bytes\n' "$entry" "$entry_type" "$entry_size" +done < <(find_immediate_children "$output_dir") | LC_ALL=C sort \ No newline at end of file diff --git a/plugins/clp/bin/structurize.py b/plugins/clp/bin/structurize.py new file mode 100644 index 0000000..ab89f1d --- /dev/null +++ b/plugins/clp/bin/structurize.py @@ -0,0 +1,86 @@ +import json +import re + +def convert_vllm_log_to_json(input_filepath, output_filepath): + # Regex to match the outer wrapper log format + # Example: 2026-06-09 10:02:41,887 - sflow.task.vllm_worker_3 - INFO - 0: ... + outer_pattern = re.compile( + r'^(?P\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s+-\s+' + r'(?P[\w\.]+)\s+-\s+' + r'(?P\w+)\s+-\s+' + r'(?P\d+):\s+' + r'(?P.*)$' + ) + + # Regex to catch inner duplicate timestamps generated by the internal vLLM logger + # Matches formats like: + # 1. ISO: "2026-06-09T17:02:52.521329Z DEBUG __init__..." + # 2. Short: "DEBUG 06-09 10:02:45 [plugins/..." + inner_timestamp_pattern = re.compile( + r'^(?:' + r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z\s+(?PINFO|DEBUG|WARN|WARNING|ERROR)\s+|' + r'(?PINFO|DEBUG|WARN|WARNING|ERROR)\s+\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s+' + r')(?P.*)$' + ) + + structured_logs = [] + seen_timestamps = set() + + with open(input_filepath, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if not line: + continue + + outer_match = outer_pattern.match(line) + if outer_match: + log_entry = outer_match.groupdict() + raw_message = log_entry.pop('message') + timestamp = log_entry['timestamp'] + + # Optional: If you want to completely skip new lines that have the exact + # same wrapper timestamp as a previous line, uncomment the following block: + # if timestamp in seen_timestamps: + # continue + # seen_timestamps.add(timestamp) + + # Check for duplicate internal timestamps inside the message + inner_match = inner_timestamp_pattern.match(raw_message) + if inner_match: + # Update the level to the inner vLLM core level (e.g. wrapper might say INFO, but core says DEBUG) + inner_level = inner_match.group('level_iso') or inner_match.group('level_short') + log_entry['level'] = inner_level.strip() + # Keep only the clean message, dropping the redundant inner timestamp + log_entry['message'] = inner_match.group('inner_message').strip() + else: + log_entry['message'] = raw_message + + structured_logs.append(log_entry) + else: + # Handle multi-line strings (like large JSON dumps) by appending to the previous message + if structured_logs: + structured_logs[-1]['message'] += '\n' + line + + # Write out as JSON Lines (JSONL) + with open(output_filepath, 'w', encoding='utf-8') as out_f: + for log in structured_logs: + out_f.write(json.dumps(log) + '\n') + + if len(structured_logs) == 0: + print(f"Warning: no structured log entries produced from {input_filepath}.", file=sys.stderr) + sys.exit(1) + + print(f"Successfully processed {len(structured_logs)} structured log entries.") + +if __name__ == "__main__": + # Replace these filenames with your actual paths + INPUT_LOG_FILE = 'vllm-log.txt' + OUTPUT_JSON_FILE = 'vllm-structured-log.json' + # take the input and output file paths from command line arguments if provided + import sys + if len(sys.argv) > 1: + INPUT_LOG_FILE = sys.argv[1] + if len(sys.argv) > 2: + OUTPUT_JSON_FILE = sys.argv[2] + # usage: python structurize.py input_log.txt output_log.json + convert_vllm_log_to_json(INPUT_LOG_FILE, OUTPUT_JSON_FILE) \ No newline at end of file diff --git a/plugins/clp/skills-claude/compress-folder/SKILL.md b/plugins/clp/skills-claude/compress-folder/SKILL.md new file mode 100644 index 0000000..0299ca9 --- /dev/null +++ b/plugins/clp/skills-claude/compress-folder/SKILL.md @@ -0,0 +1,124 @@ +--- +name: compress-folder +description: Compress log files from an arbitrary folder into a searchable CLP archive directory. +allowed-tools: ["Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder:*)"] +--- + +# Compress Folder + +Use only the plugin wrappers. Do not call bare `clp-s` or expose arbitrary CLP +commands/options. + +## Rules + +- Compress log files from one folder. Do not use this skill for session JSONL + files; use `compress` for sessions. +- Do not pass `--single-file-archive`; search uses regular archive directories. +- `--timestamp-key` has no default. Only pass it when the user says their logs + have a known timestamp field. Omit it otherwise — `clp-s` will still + compress and search, but time-range flags (`--tge`/`--tle`) will not work. +- Default file extensions: `log`, `jsonl`, `json`, `txt`, `ndjson`, `out`, + `err`. Override with `--extensions`. +- Default archive root: `${TMPDIR:-/tmp}/yscope-clp-archives`. +- Ask about archive location only if the user wants persistent storage or a + change. + +## Structurize (Unstructured Text Logs) + +Use `--structurize` when compressing **unstructured text logs** — plain-text +log files that lack a regular structured format (e.g. vLLM wrapper logs, +application logs with interleaved timestamps and messages). The flag runs each +input file through `bin/structurize.py`, which parses out timestamp, logger, +level, worker, and message fields and writes structured JSONL. This gives +`clp-s` proper timestamp extraction and better compression. + +When `--structurize` is active: + +- Each input file is converted to a `.json` sidecar in a temp directory. +- `--timestamp-key timestamp` is set automatically (do not override it). +- Files that structurize cannot parse are skipped with a warning. +- The archive's `source.path` metadata still records the original folder path. + +Do **not** use `--structurize` for files that are already structured +(JSON, JSONL, NDJSON) — structurize is designed for unstructured text formats. + +## Workflow + +1. If the user does not specify a folder, ask for one. + +2. Run compression: + + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" --folder /path/to/logs + ``` + + Override extensions or add a timestamp key as needed: + + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/logs \ + --extensions log,txt \ + --timestamp-key ts + ``` + + For unstructured text logs (vLLM logs, plain-text app logs): + + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/logs \ + --structurize + ``` + +3. After compression, always report: + + - `Raw input bytes` + - `Archive bytes` + - `Compression ratio` + - `File size reduction` + - `Input files` + - `Archives dir` + - `Archive metadata` + +4. Use the printed top-level `Archives dir` for search and decompression. The + wrappers resolve the inner `clp-s` archive directory automatically. + +## Useful Commands + +Show archive root: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" --show-archives-root +``` + +Set persistent archive root: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" --set-archives-root ~/clp-archives +``` + +Dry run: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/logs \ + --dry-run +``` + +Compress only top-level `.log` files with a timestamp field: + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /var/log/myapp \ + --extensions log \ + --no-recursive \ + --timestamp-key timestamp +``` + +Compress unstructured text logs (e.g. vLLM wrapper logs): + +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /var/log/vllm \ + --extensions log,txt \ + --structurize +``` \ No newline at end of file diff --git a/plugins/clp/skills-claude/vllm-insights/SKILL.md b/plugins/clp/skills-claude/vllm-insights/SKILL.md new file mode 100644 index 0000000..90c4ecf --- /dev/null +++ b/plugins/clp/skills-claude/vllm-insights/SKILL.md @@ -0,0 +1,209 @@ +--- +name: vllm-insights +description: Analyze structurized vLLM wrapper logs from a CLP archive — or compress a folder of raw vLLM logs first — and produce actionable insights. +allowed-tools: + - "Agent" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-decompress:*)" +--- + +# vLLM Insights + +End-to-end analysis of structurized vLLM wrapper logs with CLP. Use this when +the user wants to understand what a vLLM run did: errors, warnings, +performance signals, startup configuration, worker behavior, downloads, and +other actionable issues. + +For a single ad-hoc KQL query, use the `search` skill instead. + +## Supported inputs + +- A CLP archive directory that was produced with `--structurize` (fields: + `timestamp`, `logger`, `level`, `worker`, `message`). +- A folder of raw vLLM wrapper text logs. The skill compresses it with + `--structurize` first. +- Already-structured JSONL/NDJSON vLLM logs (compress with + `--timestamp-key timestamp` instead of `--structurize`). + +## Workflow + +1. Determine the input: + - If the user provided an archive path, use it. + - If the user provided a folder, compress it: + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder" \ + --folder /path/to/vllm/logs \ + --structurize + ``` + - If nothing was provided, ask for a folder or archive path. + +2. Report compression stats when you compressed the folder: + - `Raw input bytes` + - `Archive bytes` + - `Compression ratio` + - `File size reduction` + - `Input files` + - `Archives dir` + - `Archive metadata` + +3. **Spawn a subagent for the insight pass.** Use the Agent tool with model + `haiku` (fall back to `sonnet`). The subagent runs a focused KQL sequence + and returns only the compact Markdown report — keeping the parent context + clean. + + Subagent prompt template (fill in `ARCHIVE` and `GOAL`): + + ``` + Analyze this structurized vLLM CLP archive: ARCHIVE + + Search wrapper: /home/robin/coding-agent-integration/plugins/clp/bin/clp-s-search-kql + Goal: GOAL + + Efficiency rules (follow strictly): + - Use compound KQL instead of many separate queries: level:WARN AND message:*memory* + - Count matches with: clp-s-search-kql ARCHIVE 'KQL' | grep -c '^{' + - Project aggressively. Pass --project for each column you need + (timestamp, level, logger, worker, message). Omit --project only when you + genuinely need the full record. + - Records are stored in chronological order. If you need the first or last + timestamp, project timestamp and use head/tail; do NOT sort. + - Do NOT use --tge or --tle. The timestamp field is a string + ("YYYY-MM-DD HH:MM:SS,mmm"), not epoch ms, so time-range flags do not work. + - Use message:*term* for substring search. Bare message:term matches whole + tokens only. + - Add --ignore-case when case is uncertain. + - Use semantic("query") for exploratory searches or when exact field values + are unknown. + + KQL syntax rules: + - Available fields: timestamp, logger, level, worker, message + - String match: field:value + - Substring: field:*value* (required for message substring) + - Prefix: field:value* + - Phrase: "exact phrase" + - Boolean: A AND B, A OR B, NOT A + - No array fields exist in this archive; all fields are scalar. + + Required query sequence: + 1. Total records: '*' + 2. Level breakdown, one query per level: + level:INFO, level:DEBUG, level:WARN, level:WARNING, level:ERROR + 3. Errors / exceptions: + level:ERROR OR level:WARN OR level:WARNING OR message:*Exception* OR message:*Traceback* + 4. Performance signals: + message:*ms* OR message:*latency* OR message:*throughput* OR message:*slow* OR message:*took* OR message:*second* + 5. Configuration / startup: + message:*engine* OR message:*model* OR message:*dtype* OR message:*quantization* OR message:*tp* OR message:*pp* OR message:*cuda* OR message:*GPU* + 6. Worker distribution: project logger and count distinct logger values. + 7. Connectivity / downloads: + message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection* OR message:*timeout* + 8. Memory / KV cache: + message:*memory* OR message:*KV* OR message:*cache* OR message:*OOM* OR message:*allocation* + 9. Dynamo / Pyxis (if relevant): + message:*dynamo* OR message:*pyxis* + + Return ONLY a Markdown vLLM Insights Report with these sections: + 1. Summary — total records, level counts, time span, top logger + 2. Issues & Warnings — error count, warning count, top 3 warning patterns, + any actionable problems + 3. Performance Signals — latencies, throughput, slow operations, counts + 4. Configuration & Startup — inferred model, dtype, TP/PP, GPU, engine args + 5. Worker & Health Notes — logger/worker distribution, connectivity, + download issues + 6. Top 3 follow-up KQL queries worth running + ``` + +4. Present the subagent's report to the user. Offer to: + - Drill deeper with another subagent pass on a specific finding. + - Decompress the archive for raw inspection: + ```bash + "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-decompress" \ + /tmp/vllm-archive \ + /tmp/vllm-archive-decompressed + ``` + +## Query Starters + +| Goal | KQL | +| --- | --- | +| All records | `*` | +| Errors | `level:ERROR` | +| Warnings | `level:WARN OR level:WARNING` | +| Info messages | `level:INFO` | +| Debug messages | `level:DEBUG` | +| Substring in message | `message:*term*` | +| Phrase in message | `"exact phrase"` | +| Worker by logger | `logger:sflow.task.vllm_worker_N` | +| Any worker logger | `logger:sflow.task.vllm_worker_*` | +| Startup / engine init | `message:*engine* OR message:*Initializing* OR message:*vLLM* OR message:*config*` | +| Model / dtype / parallelism | `message:*model* OR message:*dtype* OR message:*quantization* OR message:*tp* OR message:*pp*` | +| GPU / CUDA | `message:*cuda* OR message:*GPU* OR message:*device*` | +| Memory / KV cache | `message:*memory* OR message:*KV* OR message:*cache* OR message:*OOM*` | +| Performance / latency | `message:*ms* OR message:*latency* OR message:*throughput* OR message:*slow* OR message:*took*` | +| Requests / prefill-decode | `message:*request* OR message:*prefill* OR message:*decode* OR message:*batch* OR message:*sequence*` | +| Connectivity / downloads | `message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection*` | +| Dynamo / Pyxis | `message:*dynamo* OR message:*pyxis*` | +| Failures / exceptions | `message:*error* OR message:*Error* OR message:*Exception* OR message:*Traceback* OR message:*failed*` | +| Semantic: slow operations | `semantic("slow operations")` | +| Semantic: download failures | `semantic("download or connection failures")` | +| Semantic: GPU memory | `semantic("GPU memory issues")` | + +Combine any starter with a user-supplied term using `AND`, for example: +`level:WARN AND message:*memory*`. + +## Analysis Patterns + +CLP searches the compressed archive — unmatched records are never decompressed. +Push logic into KQL rather than fetching all records and post-filtering in +shell or Python. + +**For analyses that run 3+ queries, spawn a subagent:** +- Prefer Haiku model (`haiku`); fall back to Sonnet (`sonnet`) if unavailable. +- Brief the subagent with the archive path and the analysis goal. +- Ask it to return only the structured Markdown report and follow-up queries. +- This keeps the parent context lean. + +**Count matches without fetching full records:** +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'level:ERROR' | grep -c '^{' +``` + +**Project only the columns you need:** +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" \ + --projection timestamp,level,logger,message \ + ARCHIVE 'level:WARN' +``` + +**Compound KQL — one query instead of several:** +```bash +# Warnings about memory +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'level:WARN AND message:*memory*' +# Dynamo or Pyxis messages +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'message:*dynamo* OR message:*pyxis*' +``` + +**Important: no time-range flags for structurized vLLM archives** +The `timestamp` field is a string (`"YYYY-MM-DD HH:MM:SS,mmm"`), not epoch ms. +Do not use `--tge` / `--tle`. To obtain the span, project `timestamp` and use +`head -n 1` / `tail -n 1` on the JSONL output. + +**Use semantic search for exploratory questions:** +```bash +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'semantic("slow operations")' +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'semantic("download failures") AND level:WARN' +``` + +## Report format + +Present subagent results in this order: + +1. **Summary** — total records, level counts, archive span, top logger. +2. **Issues & Warnings** — errors, warnings, top repeated messages, anything + that needs action. +3. **Performance Signals** — latencies, throughput, slow operations, counts. +4. **Configuration & Startup** — inferred model, dtype, TP/PP, GPUs, engine + args. +5. **Worker & Health Notes** — logger distribution, connectivity, downloads. +6. **Follow-up queries** — 2–3 concrete KQL queries the user can run next. \ No newline at end of file diff --git a/plugins/clp/skills-codex/compress-folder/SKILL.md b/plugins/clp/skills-codex/compress-folder/SKILL.md new file mode 100644 index 0000000..dd78a23 --- /dev/null +++ b/plugins/clp/skills-codex/compress-folder/SKILL.md @@ -0,0 +1,129 @@ +--- +name: compress-folder +description: Compress log files from an arbitrary folder into a searchable CLP archive directory. +--- + +# Compress Folder + +Use only the plugin wrappers. Standard plugin root: + +```text +~/.codex/marketplaces/yscope/plugins/clp +``` + +If installed elsewhere, resolve the same `bin/` wrappers from that plugin root. +Do not call bare `clp-s` or expose arbitrary CLP commands/options. + +## Rules + +- Compress log files from one folder. Do not use this skill for session JSONL + files; use `compress` for sessions. +- Do not pass `--single-file-archive`; search uses regular archive directories. +- `--timestamp-key` has no default. Only pass it when the user says their logs + have a known timestamp field. Omit it otherwise — `clp-s` will still + compress and search, but time-range flags (`--tge`/`--tle`) will not work. +- Default file extensions: `log`, `jsonl`, `json`, `txt`, `ndjson`, `out`, + `err`. Override with `--extensions`. +- Default archive root: `${TMPDIR:-/tmp}/yscope-clp-archives`. +- Ask about archive location only if the user wants persistent storage or a + change. + +## Structurize (Unstructured Text Logs) + +Use `--structurize` when compressing **unstructured text logs** — plain-text +log files that lack a regular structured format (e.g. vLLM wrapper logs, +application logs with interleaved timestamps and messages). The flag runs each +input file through `bin/structurize.py`, which parses out timestamp, logger, +level, worker, and message fields and writes structured JSONL. This gives +`clp-s` proper timestamp extraction and better compression. + +When `--structurize` is active: + +- Each input file is converted to a `.json` sidecar in a temp directory. +- `--timestamp-key timestamp` is set automatically (do not override it). +- Files that structurize cannot parse are skipped with a warning. +- The archive's `source.path` metadata still records the original folder path. + +Do **not** use `--structurize` for files that are already structured +(JSON, JSONL, NDJSON) — structurize is designed for unstructured text formats. + +## Workflow + +1. If the user does not specify a folder, ask for one. + +2. Run compression: + + ```bash + ~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder --folder /path/to/logs + ``` + + Override extensions or add a timestamp key as needed: + + ```bash + ~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder \ + --folder /path/to/logs \ + --extensions log,txt \ + --timestamp-key ts + ``` + + For unstructured text logs (vLLM logs, plain-text app logs): + + ```bash + ~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder \ + --folder /path/to/logs \ + --structurize + ``` + +3. After compression, always report: + + - `Raw input bytes` + - `Archive bytes` + - `Compression ratio` + - `File size reduction` + - `Input files` + - `Archives dir` + - `Archive metadata` + +4. Use the printed top-level `Archives dir` for search and decompression. The + wrappers resolve the inner `clp-s` archive directory automatically. + +## Useful Commands + +Show archive root: + +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder --show-archives-root +``` + +Set persistent archive root: + +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder --set-archives-root ~/clp-archives +``` + +Dry run: + +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder \ + --folder /path/to/logs \ + --dry-run +``` + +Compress only top-level `.log` files with a timestamp field: + +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder \ + --folder /var/log/myapp \ + --extensions log \ + --no-recursive \ + --timestamp-key timestamp +``` + +Compress unstructured text logs (e.g. vLLM wrapper logs): + +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder \ + --folder /var/log/vllm \ + --extensions log,txt \ + --structurize +``` \ No newline at end of file diff --git a/plugins/clp/skills-codex/vllm-insights/SKILL.md b/plugins/clp/skills-codex/vllm-insights/SKILL.md new file mode 100644 index 0000000..c5b1b0a --- /dev/null +++ b/plugins/clp/skills-codex/vllm-insights/SKILL.md @@ -0,0 +1,195 @@ +--- +name: vllm-insights +description: Analyze structurized vLLM wrapper logs from a CLP archive — or compress a folder of raw vLLM logs first — and produce actionable insights. +--- + +# vLLM Insights + +End-to-end analysis of structurized vLLM wrapper logs with CLP. Use this when +the user wants to understand what a vLLM run did: errors, warnings, +performance signals, startup configuration, worker behavior, downloads, and +other actionable issues. + +For a single ad-hoc KQL query, use the `search` skill instead. + +## Supported inputs + +- A CLP archive directory that was produced with `--structurize` (fields: + `timestamp`, `logger`, `level`, `worker`, `message`). +- A folder of raw vLLM wrapper text logs. The skill compresses it with + `--structurize` first. +- Already-structured JSONL/NDJSON vLLM logs (compress with + `--timestamp-key timestamp` instead of `--structurize`). + +## Workflow + +1. Determine the input: + - If the user provided an archive path, use it. + - If the user provided a folder, compress it: + ```bash + ~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-compress-folder \ + --folder /path/to/vllm/logs \ + --structurize + ``` + - If nothing was provided, ask for a folder or archive path. + +2. Report compression stats when you compressed the folder: + - `Raw input bytes` + - `Archive bytes` + - `Compression ratio` + - `File size reduction` + - `Input files` + - `Archives dir` + - `Archive metadata` + +3. **Run the insight pass.** Execute a focused KQL sequence and return a + compact Markdown report. + + Search wrapper: + + ```bash + ~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql ARCHIVE 'KQL_QUERY' + ``` + + Efficiency rules (follow strictly): + - Use compound KQL instead of many separate queries: `level:WARN AND message:*memory*` + - Count matches with: `clp-s-search-kql ARCHIVE 'KQL' | grep -c '^{'` + - Project aggressively. Pass `--project` for each column you need + (timestamp, level, logger, worker, message). Omit `--project` only when + you genuinely need the full record. + - Records are stored in chronological order. If you need the first or last + timestamp, project timestamp and use `head`/`tail`; do NOT sort. + - Do NOT use `--tge` or `--tle`. The timestamp field is a string + (`"YYYY-MM-DD HH:MM:SS,mmm"`), not epoch ms, so time-range flags do not + work. + - Use `message:*term*` for substring search. Bare `message:term` matches + whole tokens only. + - Add `--ignore-case` when case is uncertain. + - Use `semantic("query")` for exploratory searches or when exact field + values are unknown. + + KQL syntax rules: + - Available fields: `timestamp`, `logger`, `level`, `worker`, `message` + - String match: `field:value` + - Substring: `field:*value*` (required for message substring) + - Prefix: `field:value*` + - Phrase: `"exact phrase"` + - Boolean: `A AND B`, `A OR B`, `NOT A` + - No array fields exist in this archive; all fields are scalar. + + Required query sequence: + 1. Total records: `*` + 2. Level breakdown, one query per level: + `level:INFO`, `level:DEBUG`, `level:WARN`, `level:WARNING`, `level:ERROR` + 3. Errors / exceptions: + `level:ERROR OR level:WARN OR level:WARNING OR message:*Exception* OR message:*Traceback*` + 4. Performance signals: + `message:*ms* OR message:*latency* OR message:*throughput* OR message:*slow* OR message:*took* OR message:*second*` + 5. Configuration / startup: + `message:*engine* OR message:*model* OR message:*dtype* OR message:*quantization* OR message:*tp* OR message:*pp* OR message:*cuda* OR message:*GPU*` + 6. Worker distribution: project logger and count distinct logger values. + 7. Connectivity / downloads: + `message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection* OR message:*timeout*` + 8. Memory / KV cache: + `message:*memory* OR message:*KV* OR message:*cache* OR message:*OOM* OR message:*allocation*` + 9. Dynamo / Pyxis (if relevant): + `message:*dynamo* OR message:*pyxis*` + +4. Present the results as a Markdown vLLM Insights Report with these sections: + 1. **Summary** — total records, level counts, time span, top logger + 2. **Issues & Warnings** — error count, warning count, top 3 warning + patterns, any actionable problems + 3. **Performance Signals** — latencies, throughput, slow operations, counts + 4. **Configuration & Startup** — inferred model, dtype, TP/PP, GPU, engine + args + 5. **Worker & Health Notes** — logger/worker distribution, connectivity, + download issues + 6. **Top 3 follow-up KQL queries worth running** + +5. Offer to drill deeper on a specific finding or decompress for raw + inspection: + + ```bash + ~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-decompress \ + /tmp/vllm-archive \ + /tmp/vllm-archive-decompressed + ``` + +## Query Starters + +| Goal | KQL | +| --- | --- | +| All records | `*` | +| Errors | `level:ERROR` | +| Warnings | `level:WARN OR level:WARNING` | +| Info messages | `level:INFO` | +| Debug messages | `level:DEBUG` | +| Substring in message | `message:*term*` | +| Phrase in message | `"exact phrase"` | +| Worker by logger | `logger:sflow.task.vllm_worker_N` | +| Any worker logger | `logger:sflow.task.vllm_worker_*` | +| Startup / engine init | `message:*engine* OR message:*Initializing* OR message:*vLLM* OR message:*config*` | +| Model / dtype / parallelism | `message:*model* OR message:*dtype* OR message:*quantization* OR message:*tp* OR message:*pp*` | +| GPU / CUDA | `message:*cuda* OR message:*GPU* OR message:*device*` | +| Memory / KV cache | `message:*memory* OR message:*KV* OR message:*cache* OR message:*OOM*` | +| Performance / latency | `message:*ms* OR message:*latency* OR message:*throughput* OR message:*slow* OR message:*took*` | +| Requests / prefill-decode | `message:*request* OR message:*prefill* OR message:*decode* OR message:*batch* OR message:*sequence*` | +| Connectivity / downloads | `message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection*` | +| Dynamo / Pyxis | `message:*dynamo* OR message:*pyxis*` | +| Failures / exceptions | `message:*error* OR message:*Error* OR message:*Exception* OR message:*Traceback* OR message:*failed*` | +| Semantic: slow operations | `semantic("slow operations")` | +| Semantic: download failures | `semantic("download or connection failures")` | +| Semantic: GPU memory | `semantic("GPU memory issues")` | + +Combine any starter with a user-supplied term using `AND`, for example: +`level:WARN AND message:*memory*`. + +## Analysis Patterns + +CLP searches the compressed archive — unmatched records are never decompressed. +Push logic into KQL rather than fetching all records and post-filtering in +shell or Python. + +**Count matches without fetching full records:** +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql ARCHIVE 'level:ERROR' | grep -c '^{' +``` + +**Project only the columns you need:** +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql \ + --projection timestamp,level,logger,message \ + ARCHIVE 'level:WARN' +``` + +**Compound KQL — one query instead of several:** +```bash +# Warnings about memory +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql ARCHIVE 'level:WARN AND message:*memory*' +# Dynamo or Pyxis messages +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql ARCHIVE 'message:*dynamo* OR message:*pyxis*' +``` + +**Important: no time-range flags for structurized vLLM archives** +The `timestamp` field is a string (`"YYYY-MM-DD HH:MM:SS,mmm"`), not epoch ms. +Do not use `--tge` / `--tle`. To obtain the span, project `timestamp` and use +`head -n 1` / `tail -n 1` on the JSONL output. + +**Use semantic search for exploratory questions:** +```bash +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql ARCHIVE 'semantic("slow operations")' +~/.codex/marketplaces/yscope/plugins/clp/bin/clp-s-search-kql ARCHIVE 'semantic("download failures") AND level:WARN' +``` + +## Report format + +Present results in this order: + +1. **Summary** — total records, level counts, archive span, top logger. +2. **Issues & Warnings** — errors, warnings, top repeated messages, anything + that needs action. +3. **Performance Signals** — latencies, throughput, slow operations, counts. +4. **Configuration & Startup** — inferred model, dtype, TP/PP, GPUs, engine + args. +5. **Worker & Health Notes** — logger distribution, connectivity, downloads. +6. **Follow-up queries** — 2–3 concrete KQL queries the user can run next. \ No newline at end of file From 97e2e4e482adcd3c8bfa70ea288444a8bc0eb91f Mon Sep 17 00:00:00 2001 From: Robin Li Date: Fri, 3 Jul 2026 16:01:18 -0400 Subject: [PATCH 2/5] FIx some issues --- plugins/clp/bin/structurize.py | 54 +++++- .../clp/skills-claude/vllm-insights/SKILL.md | 180 +++++++++++++++--- 2 files changed, 200 insertions(+), 34 deletions(-) diff --git a/plugins/clp/bin/structurize.py b/plugins/clp/bin/structurize.py index ab89f1d..0ba0153 100644 --- a/plugins/clp/bin/structurize.py +++ b/plugins/clp/bin/structurize.py @@ -1,5 +1,7 @@ import json import re +import os +import datetime def convert_vllm_log_to_json(input_filepath, output_filepath): # Regex to match the outer wrapper log format @@ -23,9 +25,41 @@ def convert_vllm_log_to_json(input_filepath, output_filepath): r')(?P.*)$' ) + # Regex to match RAW vLLM engine logs that have NO outer wrapper (e.g. logs + # captured straight from `vllm serve` in CI, without an sflow-style wrapper). + # Matches, with an optional leading ISO timestamp (GitHub Actions line prefix) + # and/or an optional "(Component pid=N)" prefix: + # INFO 06-30 03:55:00 [importing.py:81] Triton not installed ... + # (APIServer pid=20533) INFO 06-30 04:12:03 [launcher.py:46] Route: /v1/... + # 2026-06-30T03:55:12.7099980Z INFO 06-30 03:55:00 [importing.py:81] ... + raw_vllm_pattern = re.compile( + r'^(?:(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+)?' + r'(?P\([A-Za-z_][A-Za-z0-9_]*(?:\s+pid=\d+)?\)\s+)?' + r'(?PINFO|WARNING|ERROR|DEBUG|CRITICAL)\s+' + r'(?P\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}(?:\.\d+)?)\s+' + r'(?P\[[^\]]+\])(?:\s+(?P.*))?$' + ) + structured_logs = [] seen_timestamps = set() + # Raw vLLM engine timestamps have no year ("06-15 03:52:34"), which CLP + # cannot parse. Infer a year: prefer a leading ISO timestamp on the line, + # then a YYYY-MM-DD date in the filename, else the current year. The + # resulting timestamp is normalized to "YYYY-MM-DD HH:MM:SS,mmm". + fn_year_match = re.search(r'(\d{4})-\d{2}-\d{2}', os.path.basename(input_filepath)) + fallback_year = (fn_year_match.group(1) if fn_year_match + else str(datetime.datetime.now().year)) + + def _normalize_raw_ts(raw_ts, lead_ts): + year = lead_ts[:4] if lead_ts else fallback_year + ts = f"{year}-{raw_ts}" # raw_ts == "MM-DD HH:MM:SS" or "MM-DD HH:MM:SS.frac" + if '.' in ts: + ts = ts.replace('.', ',', 1) # -> "YYYY-MM-DD HH:MM:SS,frac" + else: + ts = ts + ',000' + return ts + with open(input_filepath, 'r', encoding='utf-8') as f: for line in f: line = line.strip() @@ -57,9 +91,23 @@ def convert_vllm_log_to_json(input_filepath, output_filepath): structured_logs.append(log_entry) else: - # Handle multi-line strings (like large JSON dumps) by appending to the previous message - if structured_logs: - structured_logs[-1]['message'] += '\n' + line + raw_match = raw_vllm_pattern.match(line) + if raw_match: + component = (raw_match.group('component') or '').strip() + message = raw_match.group('raw_message') or '' + if component: + message = f"{component} {message}".strip() + structured_logs.append({ + 'timestamp': _normalize_raw_ts( + raw_match.group('raw_ts'), raw_match.group('lead_ts')), + 'logger': raw_match.group('raw_logger').strip('[]'), + 'level': raw_match.group('raw_level'), + 'message': message, + }) + else: + # Handle multi-line strings (like large JSON dumps) by appending to the previous message + if structured_logs: + structured_logs[-1]['message'] += '\n' + line # Write out as JSON Lines (JSONL) with open(output_filepath, 'w', encoding='utf-8') as out_f: diff --git a/plugins/clp/skills-claude/vllm-insights/SKILL.md b/plugins/clp/skills-claude/vllm-insights/SKILL.md index 90c4ecf..d9ba5c4 100644 --- a/plugins/clp/skills-claude/vllm-insights/SKILL.md +++ b/plugins/clp/skills-claude/vllm-insights/SKILL.md @@ -20,7 +20,7 @@ For a single ad-hoc KQL query, use the `search` skill instead. ## Supported inputs - A CLP archive directory that was produced with `--structurize` (fields: - `timestamp`, `logger`, `level`, `worker`, `message`). + `timestamp`, `logger`, `level`, `message`). - A folder of raw vLLM wrapper text logs. The skill compresses it with `--structurize` first. - Already-structured JSONL/NDJSON vLLM logs (compress with @@ -48,9 +48,9 @@ For a single ad-hoc KQL query, use the `search` skill instead. - `Archive metadata` 3. **Spawn a subagent for the insight pass.** Use the Agent tool with model - `haiku` (fall back to `sonnet`). The subagent runs a focused KQL sequence - and returns only the compact Markdown report — keeping the parent context - clean. + `haiku` (fall back to `sonnet`). The subagent runs a focused KQL + + semantic-search sequence and returns only the compact Markdown report — + keeping the parent context clean. Subagent prompt template (fill in `ARCHIVE` and `GOAL`): @@ -64,7 +64,7 @@ For a single ad-hoc KQL query, use the `search` skill instead. - Use compound KQL instead of many separate queries: level:WARN AND message:*memory* - Count matches with: clp-s-search-kql ARCHIVE 'KQL' | grep -c '^{' - Project aggressively. Pass --project for each column you need - (timestamp, level, logger, worker, message). Omit --project only when you + (timestamp, level, logger, message). Omit --project only when you genuinely need the full record. - Records are stored in chronological order. If you need the first or last timestamp, project timestamp and use head/tail; do NOT sort. @@ -73,11 +73,9 @@ For a single ad-hoc KQL query, use the `search` skill instead. - Use message:*term* for substring search. Bare message:term matches whole tokens only. - Add --ignore-case when case is uncertain. - - Use semantic("query") for exploratory searches or when exact field values - are unknown. KQL syntax rules: - - Available fields: timestamp, logger, level, worker, message + - Available fields: timestamp, logger, level, message - String match: field:value - Substring: field:*value* (required for message substring) - Prefix: field:value* @@ -85,33 +83,77 @@ For a single ad-hoc KQL query, use the `search` skill instead. - Boolean: A AND B, A OR B, NOT A - No array fields exist in this archive; all fields are scalar. + Semantic search rules: + - Use semantic("natural language query") in KQL to find log events whose + logtype is semantically similar to the query, even when exact keywords differ. + - No extra flags needed — the wrapper auto-selects a working endpoint and + shares a local cache across sessions. + - Combine semantic() with regular KQL using AND, for example: + semantic("GPU memory issues") AND level:WARN + - Use semantic search for EXPLORATORY queries where you don't know the exact + field values or keywords. It finds conceptually related events that keyword + search would miss. + - Use keyword KQL for TARGETED queries where you know the exact field value or + substring (level:ERROR, message:*OOM*, etc.). + - When a semantic query returns zero results, try rephrasing or broadening the + query, or fall back to keyword search. + - Use --semantic-top-k N (default 5) to control how many nearest logtypes are + returned. Increase to 8–10 for broader recall; decrease to 2–3 for precision. + - Use --semantic-threshold T (0.0–1.0, default 0.3) to set the minimum + similarity floor. Lower values return more results; raise to 0.5+ for + stricter matching. + Required query sequence: 1. Total records: '*' 2. Level breakdown, one query per level: level:INFO, level:DEBUG, level:WARN, level:WARNING, level:ERROR - 3. Errors / exceptions: + 3. Errors / exceptions (keyword): level:ERROR OR level:WARN OR level:WARNING OR message:*Exception* OR message:*Traceback* - 4. Performance signals: + 4. Semantic: error patterns: + semantic("errors and failures") + 5. Performance signals (keyword): message:*ms* OR message:*latency* OR message:*throughput* OR message:*slow* OR message:*took* OR message:*second* - 5. Configuration / startup: + 6. Semantic: performance and latency: + semantic("slow operations or performance degradation") + 7. Configuration / startup (keyword): message:*engine* OR message:*model* OR message:*dtype* OR message:*quantization* OR message:*tp* OR message:*pp* OR message:*cuda* OR message:*GPU* - 6. Worker distribution: project logger and count distinct logger values. - 7. Connectivity / downloads: - message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection* OR message:*timeout* - 8. Memory / KV cache: - message:*memory* OR message:*KV* OR message:*cache* OR message:*OOM* OR message:*allocation* - 9. Dynamo / Pyxis (if relevant): - message:*dynamo* OR message:*pyxis* + 8. Semantic: startup and initialization: + semantic("startup configuration and model initialization") + 9. Worker distribution: project logger and count distinct logger values. + 10. Connectivity / downloads (keyword): + message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection* OR message:*timeout* + 11. Semantic: network and connectivity: + semantic("network connectivity and download issues") + 12. Memory / KV cache (keyword): + message:*memory* OR message:*KV* OR message:*cache* OR message:*OOM* OR message:*allocation* + 13. Semantic: GPU memory and caching: + semantic("GPU memory allocation and cache issues") + 14. Dynamo / Pyxis (if relevant): + message:*dynamo* OR message:*pyxis* + 15. Semantic: request lifecycle: + semantic("request prefill decode batching") + + After running the keyword + semantic queries above, compare results: + - If a semantic query found events that the keyword query missed, note them + in the report as "semantic-only findings". + - If both found the same events, report only the keyword result. + - Deduplicate across all queries to avoid double-counting. Return ONLY a Markdown vLLM Insights Report with these sections: 1. Summary — total records, level counts, time span, top logger 2. Issues & Warnings — error count, warning count, top 3 warning patterns, - any actionable problems - 3. Performance Signals — latencies, throughput, slow operations, counts - 4. Configuration & Startup — inferred model, dtype, TP/PP, GPU, engine args + any actionable problems; include semantic-only findings that keyword + searches missed + 3. Performance Signals — latencies, throughput, slow operations, counts; + include semantic-only performance findings + 4. Configuration & Startup — inferred model, dtype, TP/PP, GPU, engine args; + include any startup events found only via semantic search 5. Worker & Health Notes — logger/worker distribution, connectivity, - download issues - 6. Top 3 follow-up KQL queries worth running + download issues; include semantic-only connectivity findings + 6. Semantic Search Coverage — for each semantic query that found events the + keyword equivalent missed, list: the semantic query, what it found that + keywords didn't, and count of semantic-only events + 7. Top 3 follow-up KQL queries worth running (mix keyword and semantic) ``` 4. Present the subagent's report to the user. Offer to: @@ -123,8 +165,39 @@ For a single ad-hoc KQL query, use the `search` skill instead. /tmp/vllm-archive-decompressed ``` +## When to use semantic search vs keyword KQL + +| Situation | Use | Why | +| --- | --- | --- | +| You know the exact field value | `level:ERROR` | Keyword is precise and fast | +| You know a substring | `message:*OOM*` | Wildcard substring is direct | +| Exploring an unfamiliar archive | `semantic("…")` | Finds conceptually related events without knowing keywords | +| Keyword search returned nothing | `semantic("…")` | May find events phrased differently than expected | +| You want breadth of coverage | Both | Combine: run keyword first, then semantic to catch what keywords missed | +| Narrowing by severity | `semantic("…") AND level:WARN` | Semantic finds the concept, KQL narrows by field | + +The vLLM insight pass above runs **both** keyword and semantic queries for every +analysis category, then reports what semantic search found that keywords +missed. This maximizes coverage. + +## Semantic search flags + +These flags only take effect when the KQL query contains `semantic()`. The +wrapper auto-detects semantic queries and configures the endpoint and cache. + +| Flag | Default | Purpose | +| --- | --- | --- | +| `--semantic-top-k K` | 5 | Number of nearest logtypes to return. Raise (8–10) for broader recall; lower (2–3) for precision. | +| `--semantic-threshold T` | 0.3 | Minimum similarity floor (0.0–1.0). Raise to 0.5+ for stricter matching. | + +The wrapper auto-selects a working semantic endpoint (local first, then +remote) and auto-enables a local embedded cache so repeated queries hit +in-process (~sub-ms). No manual configuration is needed for normal use. + ## Query Starters +### Keyword queries + | Goal | KQL | | --- | --- | | All records | `*` | @@ -145,9 +218,30 @@ For a single ad-hoc KQL query, use the `search` skill instead. | Connectivity / downloads | `message:*download* OR message:*ModelExpress* OR message:*HF* OR message:*transport* OR message:*connection*` | | Dynamo / Pyxis | `message:*dynamo* OR message:*pyxis*` | | Failures / exceptions | `message:*error* OR message:*Error* OR message:*Exception* OR message:*Traceback* OR message:*failed*` | -| Semantic: slow operations | `semantic("slow operations")` | -| Semantic: download failures | `semantic("download or connection failures")` | -| Semantic: GPU memory | `semantic("GPU memory issues")` | + +### Semantic queries + +| Goal | KQL | +| --- | --- | +| Slow operations | `semantic("slow operations")` | +| Download failures | `semantic("download or connection failures")` | +| GPU memory issues | `semantic("GPU memory issues")` | +| Errors and failures | `semantic("errors and failures")` | +| Performance degradation | `semantic("performance degradation or bottlenecks")` | +| Startup / initialization | `semantic("startup configuration and model initialization")` | +| Network connectivity | `semantic("network connectivity and download issues")` | +| Request lifecycle | `semantic("request prefill decode batching")` | +| Crashes / fatal exits | `semantic("process crash or fatal error")` | +| Configuration drift | `semantic("unexpected configuration or misconfiguration")` | + +### Combined (semantic + keyword) + +| Goal | KQL | +| --- | --- | +| Warnings about memory | `semantic("GPU memory issues") AND level:WARN` | +| Errors during startup | `semantic("startup initialization") AND level:ERROR` | +| Slow operations (errors only) | `semantic("slow operations") AND level:ERROR` | +| Download issues (warnings+) | `semantic("download failures") AND (level:WARN OR level:ERROR)` | Combine any starter with a user-supplied term using `AND`, for example: `level:WARN AND message:*memory*`. @@ -191,8 +285,25 @@ Do not use `--tge` / `--tle`. To obtain the span, project `timestamp` and use **Use semantic search for exploratory questions:** ```bash +# Find events conceptually related to slow operations "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'semantic("slow operations")' +# Combine semantic with keyword filters for precision "${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" ARCHIVE 'semantic("download failures") AND level:WARN' +# Broaden recall with --semantic-top-k +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" --semantic-top-k 8 ARCHIVE 'semantic("GPU memory issues")' +# Tighten precision with --semantic-threshold +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" --semantic-threshold 0.5 ARCHIVE 'semantic("errors and failures")' +``` + +**Run both keyword and semantic queries, then diff the results:** +```bash +# Keyword search for memory issues +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" --project timestamp,level,message \ + ARCHIVE 'message:*memory* OR message:*OOM* OR message:*cache*' +# Semantic search for memory issues (may find differently-phrased events) +"${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" --project timestamp,level,message \ + ARCHIVE 'semantic("GPU memory allocation and cache issues")' +# Compare: events found by semantic but not keyword are "semantic-only findings" ``` ## Report format @@ -201,9 +312,16 @@ Present subagent results in this order: 1. **Summary** — total records, level counts, archive span, top logger. 2. **Issues & Warnings** — errors, warnings, top repeated messages, anything - that needs action. -3. **Performance Signals** — latencies, throughput, slow operations, counts. + that needs action; include semantic-only findings that keyword searches + missed. +3. **Performance Signals** — latencies, throughput, slow operations, counts; + include semantic-only performance findings. 4. **Configuration & Startup** — inferred model, dtype, TP/PP, GPUs, engine - args. -5. **Worker & Health Notes** — logger distribution, connectivity, downloads. -6. **Follow-up queries** — 2–3 concrete KQL queries the user can run next. \ No newline at end of file + args; include any startup events found only via semantic search. +5. **Worker & Health Notes** — logger distribution, connectivity, downloads; + include semantic-only connectivity findings. +6. **Semantic Search Coverage** — for each semantic query that found events the + keyword equivalent missed, list: the semantic query, what it found that + keywords didn't, and count of semantic-only events. +7. **Follow-up queries** — 2–3 concrete queries (mix keyword and semantic) the + user can run next. \ No newline at end of file From 3e867526cc3084cf9584083baab9779aae8926fa Mon Sep 17 00:00:00 2001 From: Robin Li Date: Fri, 7 Aug 2026 16:53:08 -0400 Subject: [PATCH 3/5] Add logtype-insights skill for app-agnostic log analysis This commit introduces the logtype-insights skill, which enables end-to-end analysis of any CLP archive using a logtype baseline method. The skill classifies message templates into categories, caches classifications for efficiency, and generates targeted KQL queries based on the identified logtypes. It supports various input formats, including structurized text logs and native JSON logs, and provides a comprehensive workflow for schema discovery, logtype dumping, classification, and insights reporting. --- .../skills-claude/logtype-insights/SKILL.md | 529 ++++++++++++++++++ .../skills-codex/logtype-insights/SKILL.md | 336 +++++++++++ 2 files changed, 865 insertions(+) create mode 100644 plugins/clp/skills-claude/logtype-insights/SKILL.md create mode 100644 plugins/clp/skills-codex/logtype-insights/SKILL.md diff --git a/plugins/clp/skills-claude/logtype-insights/SKILL.md b/plugins/clp/skills-claude/logtype-insights/SKILL.md new file mode 100644 index 0000000..f5f06c4 --- /dev/null +++ b/plugins/clp/skills-claude/logtype-insights/SKILL.md @@ -0,0 +1,529 @@ +--- +name: logtype-insights +description: App-agnostic logtype-baseline log analysis with CLP. Dump the archive's logtype dictionary first, classify the real templates into (generic + app-discovered) categories, and drive targeted KQL from them — no blind queries. Caches the classification so re-analyzing the same application skips it. Works on any structurized or native-JSON CLP archive (vLLM, MongoDB, nginx, …). +allowed-tools: + - "Agent" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-decompress:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache:*)" + - "Bash(jq:*)" + - "Bash(grep:*)" + - "Bash(sort:*)" + - "Bash(uniq:*)" + - "Bash(head:*)" + - "Bash(tail:*)" + - "Bash(cat:*)" + - "Bash(wc:*)" + - "Bash(sed:*)" +--- + +# Logtype Insights (App-Agnostic, Logtype-Baseline) + +End-to-end analysis of **any** CLP archive — structurized text logs (vLLM +wrapper logs → `timestamp/logger/level/message`), native JSON logs (MongoDB → +`t.$date/s/c/msg/attr`), or other JSON — using the **logtype baseline** method: +dump the archive's logtype dictionary first, classify those *real* message +templates into categories, and derive every later query from a template that is +guaranteed to exist. No blind keyword batteries, no queries wasted on keywords +that aren't there. + +This is the generalized successor to the old vLLM-only `vllm-insights-logtypes` +skill. The logtype method is **not application-specific**: the dictionary dump, +the generic category taxonomy, the classification cache, and the project+grep +retrieval pattern all work on any archive. The only thing that changes between +applications is the set of templates — which the skill reads from the archive +itself rather than guessing. + +For a single ad-hoc KQL query, use the `search` skill. For app-specific +batteries on vLLM logs use `vllm-insights`/`vllm-kql`; on MongoDB use +`mongodb-semantic`/`mongodb-kql`. + +## Why a logtype baseline beats blind search + +A CLP logtype is a message template with variables replaced by `<*>`, e.g. +`Triton not installed or not compatible; certain GPU-related functions ...` or +MongoDB's `Slow query`, `attr.durationMillis=<*>`. The logtype dictionary is the +**complete vocabulary** of distinct message shapes in the archive — for a +typical run, tens to a few hundred templates, no matter how many millions of +records. Dumping it gives you, in one cheap pass that reads the dictionary +rather than every record: + +- Every kind of event the run actually produced (no guessing keywords). +- The static tokens of each template, which you turn into queries that always + match — so counts are exact and zero queries return zero by surprise. +- A natural unit for "top repeated messages": frequency per template. + +The blind variants (`*-insights`, `*-kql`) run a fixed battery of hardcoded +queries; on an unfamiliar archive many return nothing. This skill runs **1 dump ++ schema discovery + a handful of targeted queries**, each grounded in a real +template. + +## Why the classification is cached + +Classifying the templates into categories and deriving a query plan is the one +expensive step, and it is a property of the **application**, not the individual +capture: the same app build emits the same message templates on every run, so +the same classification applies. This skill caches the classification keyed by a +fingerprint of the template set (`sha256` of the sorted logtypes). On a cache +hit (same app), classification is skipped entirely and the skill goes straight +to the insight pass with the pre-made plan — so re-analyzing the same +application costs only the cheap Haiku insight pass, not the classification. + +## Supported inputs + +- A CLP archive directory (any kind). Primary input. +- A folder of raw logs — compress first with the app-appropriate settings, since + compression is the one app-specific step: + - vLLM wrapper text logs: `--structurize` (produces `timestamp/logger/level/message`). + - MongoDB JSON: `--extensions '*' --timestamp-key t.$date` (native). + - Generic JSON with a known timestamp field: `--timestamp-key `. + - Then point this skill at the resulting archive. +- If nothing was provided, ask for an archive or folder path. + +## Workflow + +1. Determine the input: + - If the user provided an archive path, use it. + - If the user provided a folder, compress it with the app-appropriate + settings (above) and use the resulting archive. If the app is unknown, ask + the user how the logs should be compressed (structurize vs native + `--timestamp-key`), or have them compress first and pass the archive. + - If nothing was provided, ask for an archive or folder path. + +2. Report compression stats when you compressed the folder: + - `Raw input bytes`, `Archive bytes`, `Compression ratio`, + `File size reduction`, `Input files`, `Archives dir`, `Archive metadata`. + +3. **Discover the schema** (cheap; do this in the parent). A no-projection + search returns the full original record, so one sample line reveals the + field names: + + ```bash + ARCHIVE= + SEARCH="${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" + + # One full record (reveals the JSON keys / structurized fields): + "$SEARCH" "$ARCHIVE" '*' 2>/dev/null | grep '^{' | head -1 + ``` + + Identify and record, as `schema`, the field names for: + - **timestamp** — e.g. `timestamp` (vLLM structurized) or `t.$date` (Mongo). + If it is a real epoch (native JSON), `--tge`/`--tle` work; if it is a + structurized string, they do not. + - **severity** — e.g. `level` (vLLM) or `s` (Mongo). + - **logger/component** — e.g. `logger` (vLLM) or `c` (Mongo). + - **message** — the clp-string field whose logtypes appear in + `stats.logtypes` — e.g. `message` (vLLM) or `msg` (Mongo). + - **payload** (optional) — e.g. `attr` (Mongo); note the useful leaf paths + (e.g. `attr.durationMillis`, `attr.host`). + Also note the distinct values of the severity and logger fields (one count + query each) so the classifier and insight pass can use the real vocabularies: + ```bash + "$SEARCH" --projection "$ARCHIVE" '*' | grep '^{' | jq -r '.' | sort | uniq -c | sort -rn + "$SEARCH" --projection "$ARCHIVE" '*' | grep '^{' | jq -r '.' | sort | uniq -c | sort -rn + ``` + +4. **Dump the logtype baseline** (cheap; reads the dictionary, not every record): + + ```bash + # Canonical template dictionary — one JSON object per logtype: + # {"id":0,"logtype":"Triton not installed ... <*> functions ..."} + "$SEARCH" "$ARCHIVE" 'stats.logtypes' > /tmp/logtypes.ndjson 2>/tmp/logtypes.err + + # Summary: how many distinct templates, and the templates themselves. + jq -s 'length' /tmp/logtypes.ndjson + jq -r '.logtype' /tmp/logtypes.ndjson + ``` + + **Fallback if `stats.logtypes` emits no NDJSON** (some builds only print a + `[stats]` dictionary-size summary to stderr). If `/tmp/logtypes.ndjson` has + zero JSON lines, build an approximate baseline by projecting the message + field for all records and templatizing the variable runs (O(records), but + produces templates AND counts in one pass): + + ```bash + MSG= # e.g. message (vLLM) or msg (Mongo) + "$SEARCH" --projection "$MSG" "$ARCHIVE" '*' \ + | grep '^{' | jq -r --arg f "$MSG" '.[$f]' \ + | sed -E 's/\{[^}]+\}/<*>/g; s/0x[0-9a-fA-F]+/<*>/g; s/\b[0-9]+\b/<*>/g' \ + | sort | uniq -c | sort -rn > /tmp/logtype-freqs.txt + ``` + + (The wrapper prints archive-metadata header lines to stdout, so `grep '^{'` + filters to JSON records before `jq` — same idiom as `grep -c '^{'` for + counting.) + +5. **Classification cache lookup** (cheap; parent): + + ```bash + CACHE="${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache" + APP_KEY="$("$CACHE" key --logtypes-file /tmp/logtypes.ndjson)" + # If you used the templatize fallback instead, build the key from it: + # APP_KEY="$(sort /tmp/logtype-freqs.txt | ... )" # see logtype-cache `key` + + if CLASSIFICATION="$("$CACHE" get "$APP_KEY" 2>/dev/null)"; then + echo "CACHE HIT — reusing cached classification for $APP_KEY" + echo "$CLASSIFICATION" > /tmp/logtype-classification.json + else + echo "CACHE MISS — will classify and store for $APP_KEY" + fi + ``` + + On a **cache hit**, verify the cached `schema` field matches the schema you + discovered in step 3 (same field names). If it matches, skip to step 7 — the + classification is reused as-is and the expensive classification subagent is + skipped entirely. If the schema differs, treat it as a miss (reclassify). + +6. **(Cache miss only) Classify the templates.** Spawn a **classification + subagent** with the Agent tool, model `sonnet` (fall back to `haiku`). It + takes the baseline templates + the discovered schema + the generic taxonomy, + classifies each template, discovers any app-specific categories, builds a + query plan expressed in the discovered field names, and writes the result as + structured JSON to `/tmp/logtype-classification.json`. + + Classification subagent prompt template (fill in `ARCHIVE`, the `schema`, and + paste the baseline from step 4): + + ``` + You are classifying the logtype templates of a CLP archive so a later insight + pass can run targeted queries. Do NOT write the final report — only the + classification JSON. + + Archive: ARCHIVE + Discovered schema (field names in this archive): + timestamp: + severity: + logger: + message: (the clp-string field whose logtypes these are) + payload: (leaf paths if any, e.g. attr.durationMillis) + Severity values seen: + Logger values seen: + + LOGTYPE BASELINE (every distinct message template in the archive; <*> marks + variables). Classify each into a category and derive a query plan from its + static tokens. + + + Method: + 1. Classify each template into the best-fitting category. Use this GENERIC + default taxonomy, AND add any APP-SPECIFIC categories you discover from the + templates (e.g. for MongoDB: workload/operations (slow query, write-concern + waits), replication/election, sharding, indexing, WiredTiger/storage; for + vLLM: worker-health, kv-cache, model-loading). Name + app-specific categories descriptively. + Generic defaults: + - errors / exceptions / failures + - warnings + - performance (latency / throughput / timing / "took <*> ms") + - config / startup / initialization + - network / connectivity / timeout + - resource (memory / disk / file-descriptors / storage pressure) + - lifecycle / state-transitions (start/stop/election/stepdown/restart) + - security / auth / access + - other (note but don't deep-search) + 2. Build a QUERY PLAN: a list of targeted queries, each derived from one or + more real templates, expressed in the discovered field names. For each plan + entry give: label, the KQL filter (using the searchable scalar fields — + severity/logger/payload leaves; NOT message:term which is a clp-string and + returns 0), the columns to --projection, and the method: + - "count" -> count matches via `... | grep -c '^{'` + - "project+grep" -> project the message field and grep its static text + - "project+jq" -> project message/payload and jq-filter (e.g. a + numeric threshold on a payload leaf) + - "semantic" -> semantic("...") AND , ONLY for an + ambiguous template or to group similar ones + Example plan entry (Mongo schema): + {"label":"Slow queries","kql":"attr.durationMillis:*", + "project":"t.$date,attr.durationMillis,msg", + "jq":"select((.attr.durationMillis//0)>100)","method":"project+jq"} + Example plan entry (vLLM schema): + {"label":"Memory warnings","kql":"level:WARNING", + "project":"timestamp,level,message","grep":"memory|OOM|KV", + "method":"project+grep"} + 3. Total records: '*'. Severity breakdown: one count per severity value seen. + Logger breakdown: project logger + uniq -c. Time span: project the + timestamp field and use head/tail (records are chronological; do NOT sort). + 4. Remember: the message field is a clp-string. KQL `message:term` / + `msg:term` and `message:*term*` / `msg:*term*` return 0. Only the scalar + fields (severity, logger, payload leaves) are KQL-searchable. Retrieve + message content by projecting the message field and grepping. + + Write the result as valid JSON to /tmp/logtype-classification.json with EXACTLY + this shape, then print "DONE" and nothing else: + { + "app_key": "", + "schema": {"timestamp":"","severity":"","logger":"","message":"","payload":["",...]}, + "taxonomy": [{"category":"","description":""}], + "templates": [{"logtype":"