Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,18 @@

Notable changes to the Datum Cloud Claude Code plugins.

## [1.7.1] - 2026-05-11

### Changed

- **Operational review skill** (`operational-review`) — After opening the traffic report PR, now finds the current on-call issue in `datum-cloud/engineering` (open issue titled "on-call: week of …") and posts a comment linking the review.

## [1.7.0] - 2026-04-25

### Added

- **GCP spend analysis skill** (`analyze-gcp-spend`) — Weekly GCP cost report covering datum-cloud staging and production. Queries BigQuery billing exports and live cluster state, generates mermaid trend charts for the trailing 4 months, and files a PR to `datum-cloud/engineering`. Includes a dedicated `cost-analyst` agent and full report-format and query reference.

## [1.6.0] - 2026-04-12

### Added
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ A marketplace for Claude Code plugins providing platform engineering tools and a

| Plugin | Description | Version |
|--------|-------------|---------|
| [datum-platform](./plugins/datum-platform/) | Kubernetes platform engineering automation with aggregated API servers, controller patterns, and GitOps deployment | 1.6.0 |
| [datum-platform](./plugins/datum-platform/) | Kubernetes platform engineering automation with aggregated API servers, controller patterns, and GitOps deployment | 1.7.1 |
| [datum-gtm](./plugins/datum-gtm/) | Go-to-market automation with commercial strategy, product discovery, and customer support | 1.0.0 |
| [milo-activity](./plugins/milo-activity/) | Query audit logs, investigate incidents, and author ActivityPolicies using the Milo Activity service | 1.0.0 |

Expand Down Expand Up @@ -123,8 +123,8 @@ claude-code-plugins/
Kubernetes platform engineering automation with aggregated API servers, controller patterns, and GitOps deployment for the Datum Cloud platform.

**Features:**
- 6 specialized agents (api-dev, frontend-dev, sre, test-engineer, code-reviewer, tech-writer)
- 26 skill modules covering Kubernetes patterns, Go conventions, deployment workflows, and more
- 7 specialized agents (api-dev, frontend-dev, sre, test-engineer, code-reviewer, tech-writer, cost-analyst)
- 27 skill modules covering Kubernetes patterns, Go conventions, deployment workflows, and more
- Pipeline orchestration for structured feature development
- Automatic learning engine for pattern extraction

Expand Down
2 changes: 1 addition & 1 deletion plugins/datum-platform/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "datum-platform",
"description": "Kubernetes platform engineering automation with aggregated API servers, controller patterns, GitOps deployment, and platform capabilities",
"version": "1.6.0",
"version": "1.7.1",
"author": {
"name": "Datum Cloud",
"url": "https://github.com/datum-cloud"
Expand Down
36 changes: 36 additions & 0 deletions plugins/datum-platform/agents/cost-analyst.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
name: cost-analyst
description: >
Use when producing or updating a weekly GCP spend report for datum-cloud infrastructure.
Triggered by phrases like "gcp spend", "cost report", "analyze costs", "cloud spend",
"monthly spend", or "cost analysis". Covers both production and staging, queries BigQuery
billing exports and live cluster state, generates mermaid trend charts for the trailing
4 months, and files a PR to the engineering repo.
tools: Read, Write, Edit, Grep, Glob, Bash
model: sonnet
---

# Cost Analyst Agent

Senior SRE responsible for GCP cost visibility across datum-cloud infrastructure.
Produces weekly spend reports and identifies optimization opportunities.

## Workflow

Load `analyze-gcp-spend` skill, then follow its workflow:

1. **Determine cadence** — check today's date; select full-month (day ≤ 7) or MTD mode
2. **Query BigQuery** — billing data for both `datum-cloud-prod` and `datum-cloud-staging`
3. **Query live state** — node pools, PVCs, Cloud SQL from both clusters
4. **Build 4-month trend** — trailing-4-months BigQuery query → mermaid chart data
5. **Draft report** — compute section totals, write Top Issues, fill mermaid charts
6. **Open PR** — write to `/workspace/engineering`, commit, push, create PR

## Skills

- `analyze-gcp-spend` — billing queries, live state commands, report format, PR workflow

## Output

A pull request to `/workspace/engineering` at `reports/gcp-spend/YYYY-MM-DD-gcp-spend.md`.
Branch: `ops/gcp-spend-YYYY-MM-DD`. See `analyze-gcp-spend/report-format.md` for full conventions.
97 changes: 97 additions & 0 deletions plugins/datum-platform/skills/analyze-gcp-spend/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
name: analyze-gcp-spend
description: >
Produces a GCP spend report covering all datum-cloud infrastructure across staging and production.
On the first week of the month (run date ≤ day 7), performs a full retrospective for the prior
calendar month. Otherwise produces an MTD snapshot. Queries BigQuery billing exports and live
cluster state, generates mermaid trend charts for the trailing 4 months, and files a PR to
the engineering repo.
---

# GCP Spend Analysis

Produces a GCP spend report for all datum-cloud infrastructure (staging + production).
Run weekly on Mondays. On the first week of the month (run date ≤ day 7), produces a full
retrospective for the prior calendar month. All other weeks produce a month-to-date snapshot.

## Cadence Logic

| Run date | Reporting period | Report type |
|----------|-----------------|-------------|
| Day 1–7 of month | Previous calendar month (full) | Full monthly retrospective |
| Day 8+ of month | Current month-to-date | MTD snapshot |

## Scope

| Environment | GCP Project ID | Cluster | Region |
|-------------|---------------|---------|--------|
| Production | `datum-cloud-prod` | `infrastructure-control-plane-prod` | us-east4 |
| Staging | `datum-cloud-staging` | `infrastructure-control-plane-staging` | us-east4 |

Edge clusters (dfw, tyo, syd, etc.) run outside GCP — exclude from this analysis.

## Preflight Checks

Run these before doing any work. If BigQuery fails, stop and report — do not proceed
with estimates in place of real billing data.

```bash
# 1. Verify BigQuery billing export is accessible
bq ls --project_id datum-cloud-prod billing
# Expected: lists at least one table named gcp_billing_export_resource_v1_*
# Failure → stop. Fix: grant bigquery.jobs.create + bigquery.tables.getData
# on the billing dataset to the active service account or user.

# 2. Verify prod cluster is reachable
gcloud container clusters list --project datum-cloud-prod --format="value(name)"
# Expected: infrastructure-control-plane-prod
# Failure → note in report, use IaC config for prod compute section only

# 3. Verify staging cluster is reachable
gcloud container clusters list --project datum-cloud-staging --format="value(name)"
# Expected: infrastructure-control-plane-staging
# Failure → note in report, use IaC config for staging compute section only
```

**BigQuery is a hard requirement.** Storage and Cloud SQL are only visible in the billing
export — without it the report is missing $2,000–$3,000/month of spend. A report produced
without billing data will be materially wrong. Stop and surface the access error; do not
publish an estimate-based report as if it were authoritative.

## Workflow

1. **Preflight** — run the checks above; halt on BigQuery failure
2. **Determine period** — check today's date; select full-month (day ≤ 7) or MTD mode
3. **Query billing** — run BigQuery queries from `queries.md` for services, SKUs, and storage
4. **Query live state** — pull current node pools, PVC inventory, and Cloud SQL from both clusters
5. **Compute 4-month trend** — run the trailing-4-months query; populate mermaid chart data
6. **Identify issues** — compare billing data to live state; flag IaC drift, oversized resources,
generation-penalty instances (n1 vs n2), and unexpected on-demand pools
7. **Draft report** — follow the structure and mermaid templates in `report-format.md`
8. **File PR** — write to `/workspace/engineering` and open a PR via `gh pr create`

## What to Flag

- **IaC drift** — node pools or resources visible in the cluster but absent from Pulumi/IaC config
- **Generation penalty** — n1 instances cost ~15% more than equivalent n2; flag and quantify savings
- **Spot vs on-demand transitions** — note any pools that switched from spot to on-demand; compute delta
- **Oversized storage** — PVCs provisioned much larger than workload requirements (flag if detectable)
- **Single points of failure on premium storage** — hyperdisk-balanced or pd-ssd on unreplicated nodes
- **Month-over-month deltas** — any service that increased >10% MoM warrants a note in Top Issues

## Infrastructure

- **Billing source**: BigQuery billing export — see `queries.md` for table discovery
- **Live state**: `gcloud`, `kubectl` — commands in `queries.md`
- **Target repo**: `/workspace/engineering`
- **Output path**: `reports/gcp-spend/YYYY-MM-DD-gcp-spend.md`

## Sub-topics

- `queries.md` — BigQuery billing SQL, gcloud and kubectl commands, 4-month trend queries
- `report-format.md` — report sections, mermaid chart templates, PR workflow and git commands

## Related Skills

- `operational-review` — Weekly traffic and latency report for the Envoy edge ingress
- `capability-quota` — Quota system that can gate resource consumption and bound costs
178 changes: 178 additions & 0 deletions plugins/datum-platform/skills/analyze-gcp-spend/queries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# GCP Billing Queries

## Finding the Billing Export Table

```bash
bq ls --project_id datum-cloud-prod billing
```

The export table is typically named `gcp_billing_export_resource_v1_{ACCOUNT_ID}`. Replace
`{BILLING_TABLE}` throughout this file with the fully-qualified table reference, e.g.:
`datum-cloud-prod.billing.gcp_billing_export_resource_v1_XXXXXX_XXXXXX_XXXXXX`

## GCP Projects in Scope

| Environment | Project ID |
|-------------|-----------|
| Production | `datum-cloud-prod` |
| Staging | `datum-cloud-staging` |

## Period Parameters

Set `@period_start` and `@period_end` based on the cadence rule in `SKILL.md`:

| Mode | @period_start | @period_end |
|------|--------------|-------------|
| Full month (day ≤ 7) | First day of prior month | First day of current month |
| MTD (day ≥ 8) | First day of current month | Tomorrow |

## Cost by Service

```sql
SELECT
project.id AS project,
service.description AS service,
SUM(cost) AS total_cost
FROM `{BILLING_TABLE}`
WHERE
DATE(_PARTITIONTIME) >= @period_start
AND DATE(_PARTITIONTIME) < @period_end
AND project.id IN ('datum-cloud-prod', 'datum-cloud-staging')
GROUP BY project, service
ORDER BY total_cost DESC
```

## Cost by SKU (Compute + GKE Detail)

```sql
SELECT
project.id AS project,
service.description AS service,
sku.description AS sku,
SUM(cost) AS total_cost
FROM `{BILLING_TABLE}`
WHERE
DATE(_PARTITIONTIME) >= @period_start
AND DATE(_PARTITIONTIME) < @period_end
AND project.id IN ('datum-cloud-prod', 'datum-cloud-staging')
AND service.description IN ('Compute Engine', 'Kubernetes Engine')
GROUP BY project, service, sku
ORDER BY total_cost DESC
LIMIT 50
```

## Storage Breakdown

```sql
SELECT
project.id AS project,
sku.description AS sku,
SUM(cost) AS total_cost
FROM `{BILLING_TABLE}`
WHERE
DATE(_PARTITIONTIME) >= @period_start
AND DATE(_PARTITIONTIME) < @period_end
AND project.id IN ('datum-cloud-prod', 'datum-cloud-staging')
AND service.description = 'Compute Engine'
AND (sku.description LIKE '%Storage%' OR sku.description LIKE '%Disk%')
GROUP BY project, sku
ORDER BY total_cost DESC
```

## 4-Month Trend by Service

Returns monthly totals for the trailing 4 calendar months to power the mermaid trend charts.
Run this query regardless of cadence mode — trend always covers the trailing 4 full months.

```sql
SELECT
DATE_TRUNC(DATE(_PARTITIONTIME), MONTH) AS month,
project.id AS project,
service.description AS service,
SUM(cost) AS total_cost
FROM `{BILLING_TABLE}`
WHERE
DATE(_PARTITIONTIME) >= DATE_TRUNC(DATE_SUB(CURRENT_DATE(), INTERVAL 3 MONTH), MONTH)
AND project.id IN ('datum-cloud-prod', 'datum-cloud-staging')
GROUP BY month, project, service
ORDER BY month, total_cost DESC
```

Aggregate service rows into three buckets for charting:

| Chart bucket | Billing service rows |
|-------------|---------------------|
| Compute | Compute Engine, Kubernetes Engine |
| Storage | Cloud Storage, Filestore, plus storage/disk SKUs within Compute Engine |
| Other | Cloud SQL, Networking, and everything else |

## Cloud SQL Inventory

```bash
gcloud sql instances list \
--project datum-cloud-prod \
--format="table(name, databaseVersion, tier, settings.availabilityType, state, diskSizeGb)"

gcloud sql instances list \
--project datum-cloud-staging \
--format="table(name, databaseVersion, tier, settings.availabilityType, state, diskSizeGb)"
```

Cloud SQL cost from billing is often understated if disk size isn't exported. Use the
`diskSizeGb` field above to sanity-check storage costs.

## Live Node Pool Inventory

```bash
# Production
gcloud container node-pools list \
--cluster infrastructure-control-plane-prod \
--region us-east4 \
--project datum-cloud-prod \
--format="table(name, config.machineType, config.spot, autoscaling.minNodeCount, autoscaling.maxNodeCount, initialNodeCount)"

# Staging
gcloud container node-pools list \
--cluster infrastructure-control-plane-staging \
--region us-east4 \
--project datum-cloud-staging \
--format="table(name, config.machineType, config.spot, autoscaling.minNodeCount, autoscaling.maxNodeCount, initialNodeCount)"
```

Cross-reference the node pool list against the Pulumi/IaC config in the infrastructure repo.
Any pool not present in IaC should be flagged as drift in the Top Issues section.

## Live PVC Inventory

```bash
# Production
kubectl get pvc --all-namespaces --context infrastructure-control-plane-prod \
-o custom-columns="NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.storageClassName,CAPACITY:.status.capacity.storage"

# Staging
kubectl get pvc --all-namespaces --context infrastructure-control-plane-staging \
-o custom-columns="NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.storageClassName,CAPACITY:.status.capacity.storage"
```

Sum capacity by storage class; reconcile against the Storage section of the billing query output.
Significant gaps between provisioned and billed storage can indicate orphaned volumes.

## GCP List Prices (us-east4 Reference)

Use these for cost decomposition when billing data is aggregated and you need to back-calculate
per-node or per-GiB costs. These are on-demand list prices — actual bill may reflect CUDs or SUDs.

| Resource | Price |
|----------|-------|
| n1-standard-8 (on-demand) | ~$277/month |
| n2-standard-8 (on-demand) | ~$239/month |
| n2d-standard-8 (on-demand) | ~$228/month |
| n2d-standard-2 (on-demand) | ~$57/month |
| n2d-standard-2 (spot) | ~$17/month |
| c4d-standard-8 (on-demand) | ~$312/month |
| n2-standard-4 (on-demand) | ~$139/month |
| hyperdisk-balanced | $0.12/GiB/month |
| pd-ssd (`premium-rwo`) | $0.17/GiB/month |
| pd-balanced (`standard-rwo`) | $0.10/GiB/month |
| pd-standard | $0.04/GiB/month |
| GKE cluster fee | $72/month per cluster |
Loading