Research: where the Harness Knowledge Graph lands, and what structure fits instead

[JRichlen]

Adds three research deliverables under docs/research/:

• harness-knowledge-graph.md — a placement analysis for the Harness Software Delivery Knowledge Graph, diffed against every shipped skill and against this repo's own prior verdict on graph memory in docs/research/agentic-patterns-corpus.md.
• llm-wiki-patterns.md — the follow-on question that note raises and does not answer: if not a graph, then what structure? Surveys the LLM-generated-wiki family and everything else between flat grep and a knowledge graph. Details in its own section below.
• knowledge-management-corpus.md + .json — the broad survey behind both: 140 patterns across ten source clusters, seven verification dives, 57 corrections. Details in its own section below.

Plus two corrections to claims that verification found false, in harness-knowledge-graph.md and in AGENTS.md / evals/cheap/run.sh — see "Corrections to this repo's own claims" below.

Docs, plus two comment lines in evals/cheap/run.sh correcting a false timing claim. No plugins/** and no SKILL.md touched — so no behavioral tier, no demonstration comment owed. Cheap tier run on every push; 1291 passed, 0 failed on the current head.

The verdict

Not a new plugin. Harness's payoff comes from a condition a repo fleet does not meet: heterogeneous, non-git-native estate data (billing, K8s, CloudWatch, Jira, PagerDuty) with no single authoritative query surface. A repo fleet already has one — git, gh api, grep. Harness's own ROI rule ("start with one use case that cannot be solved by a single system") is the exact test this domain fails.

What it does change

The corpus rejected graph memory twice, and the two rejections age differently:

• Evidence quality — now dead. "rests on one 198-document academic system, not leader adoption." Harness retires this: 1K+ enterprise customers, 40+ integrations, a published cost model, a published eval methodology.
• Domain fit — holds, and is now load-bearing. "a repo already has git, grep and a type checker as a better graph." Harness's own canonical-identity section ("the same service is called something different in Git, Kubernetes, CloudWatch, and your runbook") names the problem a graph solves — and GitHub's opaque node_id, independent of the mutable full_name, means a repo fleet doesn't have it. (Deliberately opaque, not "stable" — see "Corrections to this repo's own claims" below.)

Same answer, better reason, now domain-scoped rather than universal. If this marketplace ever spans cost data, incident history, or runtime telemetry, the argument reopens on its merits. Both corpus entries now carry a back-link here, so the retired half is never read as live.

Where it lands

eval-ladder (strongest fit, real gap). Their eval post is a published production eval ladder for a retrieval system, stratified the way ours is. It names something we don't: "a registered relationship is not necessarily a usable relationship" — a query can be structurally valid, reference a schema-declared relationship, execute successfully, and return nothing because the relationship was never populated. That's rung 0's stated blind spot reappearing in the data layer: declared ≠ populated ≠ fresh. The rung for it already exists (rung 1, discriminating corpus); the vocabulary doesn't. Their product-backed validation is independent, dated, citable confirmation of "grade the surface closest to the harm."

fleet-playbook-curator (validated design, one gap). Four of Harness's theses were arrived at here independently — canonical identity on node_id over full_name, freshness as an independent staleness clock, index-not-runbook, drift-as-change-management. The gap is a fifth: the fleet manifest is a flat entity table ({node_id, name, full_name, default_branch, head_sha, pushed_at, archived, private}) with no edge field. Relationships are cited like any other claim and machine-checked for traceability by validate-citations.sh — they are simply not modeled, so (a) every key diff-fleet.sh has is per member, none per edge, and (b) traceable is not supported: for an edge the evidence is the join, and nothing deterministic checks it. The note sketches three candidate edges but explicitly does not recommend building yet; that goes through grill-me and clears eval-ladder's bar first.

agent-compiler / semver-gate / redgate. Their four-tier data-ownership ladder is the same construct as an effect ceiling: always take the most-deterministic rung available, never optimize for the least. "Measure determinism, not just capability" is a usable external citation for verify-before-claim.

docs-hygiene. Corroboration only. Our GitOps Ready: True framing already states the failure shape better.

Companion note — docs/research/llm-wiki-patterns.md (62c8b0e)

The Harness note names a gap in fleet-playbook-curator and never asks what structure
would close it. This companion asks that, surveying everything between flat grep and a
knowledge graph: DeepWiki (Cognition), Google Code Wiki, DeepWiki-Open, Auto Wiki,
aider's repo map, SCIP/precise code navigation, code embeddings, llms.txt,
AGENTS.md, graphify, and doctests.

Verdict: nothing in the family closes the gap. The verdict above is unchanged. But
the reason gets sharper, and the sharpening is the point.

A measured external failure, not an argument. The note fetched a live DeepWiki page
for Aider-AI/aider pinned to commit 5dc9490b — verified via git ls-remote to still
be that repo's HEAD, so zero drift, the best possible case for a generated wiki.
Three of the five rows in its architecture table are wrong, including one naming a
function rank_tags() that does not exist anywhere in the cited file (verified by
fetching the blob at that sha; the real symbols are get_ranked_tags,
get_ranked_tags_map, get_ranked_tags_map_uncached). Every citation resolves. Every
citation is commit-pinned with a line range — strictly finer-grained than this repo's
own contract requires.

That is the gap validate-citations.sh names in its own comments — traceability is
"necessary but not sufficient," semantic support explicitly out of scope
(validate-citations.sh:38-41) — reproduced at industrial scale by the best-funded team
in the category. The "traceable is not supported" bullet above is now backed by a
measurement rather than an argument, and it means no vendor is about to solve this for us.

The design constraint that falls out. Nothing in this survey machine-checks generated
prose, and the reason is not difficulty: a prose claim has no failure condition.
Everything trustworthy in the survey got there by giving the claim one — a parser that
either resolves the symbol or does not (SCIP, aider, graphify's EXTRACTED edges), a
doctest that either compiles or does not, a head_sha that either matches or does not.
So:

An edge is only worth adding if it comes with a command that re-derives it and a
comparison that can go red.

Applied to the three candidate edges sketched above, that test disqualifies them
unevenly, which is the useful part: repo --deploys-via--> workflow passes (parse
.github/workflows/*.yml, diff the derived set); repo --depends-on--> repo passes
(manifest/lockfile extraction is deterministic); repo --authenticates-as--> identity
fails as stated — nothing in the fleet can re-derive it without touching the identity
provider. It is exactly the class of claim that looks checkable and is not.

Freshness, measured. Indexed-commit-vs-live-HEAD on five repos, all 2026-09-14:
microsoft/vscode ~6 days behind, openai/codex ~10, langchain-ai/langchain ~23,
anthropics/claude-code ~32. Not a criticism — the stamp moves with the page, so a stale
page is legible as stale. What the stamp cannot say is which claims the intervening 32
days invalidated. Same two-clock distinction fleet-playbook-curator already draws.

Also recorded: AGENTS.md is by a distance the highest-adoption pattern in the survey
and the least structured one, which strengthens the case for docs-hygiene rather than
weakening it; llms.txt is a format worth copying and an adoption claim worth
discounting; and the embeddings retreat (Sourcegraph 2024-02-15, Anthropic 2025-09-29,
Cursor's retired indexing doc) is itself the finding.

Claims are tagged VERIFIED (artifact read first-hand) or CLAIMED (vendor repeated). What
the research could not establish is recorded in the note, including the load-bearing one:
three-of-five is a finding about one table on one page chosen because it was the best
case, not a measured error rate.

Knowledge-management corpus — docs/research/knowledge-management-corpus.md + .json (b246eec)

The two notes above ask whether a knowledge graph and an LLM-generated wiki belong here,
and both answer no. This asks the question they raise and neither answers: across every
field that has ever tried to keep written knowledge true, what actually works?

Ten scouts over ten source clusters → 140 unique patterns → seven dives re-verifying 64 of
them against primary sources, organised by cross-domain convergence rather than by scout →
57 corrections → 156 recorded could-not-establish items → one roadmap.

Verdict: nobody machine-checks a knowledge claim. The field's entire answer is to make
claims that do not need checking — derive them from a parser or compiler (SCIP, aider's
repo map, Bazel query, graphify's EXTRACTED edges), execute them (doctests), or accept
them as unchecked and label how they were produced (DataHub's matchType, OpenMetadata's
source enum defaulting to Manual). Where a semantic check is attempted the measured
ceiling is ~77% balanced accuracy on prose — 0.55 informedness against a 50% chance
baseline — falling to ~61 on the hardest prose split, ~58 on cases where existing
detectors disagree, and 0.17 span-F1 for a prose-trained checker on code as evidence.
The edge gap this PR names is the unsolved problem in every field that has tried it.

Three findings the survey did not expect:

Fail-open is the norm, and nobody says so. Every expiry and drift mechanism examined
degrades to green rather than red — an env var that skips every check, a swallowed network
error, mdBook silent (not merely non-blocking) on a missing include anchor for nearly
seven years, CODEOWNERS degrading open in exactly the case that matters. So the companion
note's rule needs a clause: the comparison must also go red when it cannot be made. The
cut that follows is internal-oracle vs external-oracle — a check whose answer lives inside
the repo can be honest offline; one needing an external oracle cannot, and its green means
"either fine, or I couldn't look."

Citation is not transcription, and the corpus proved it on itself. 57 of ~140 scout
claims needed correction — almost none for missing citations, nearly all for misread ones.
The clean external instance: "84% of KM programmes fail" traces through three hops of
valid, resolvable citations to a 1997 article stating the rate is one third, and that
the figure is an estimate from consulting engagements rather than a study. Every footnote
in the chain was correct. Pinning where is not pinning what — and repo@sha:path
pins only where. That is why the top roadmap item is a quoted span alongside the path,
which is deterministic, offline, and would have caught an 84%-class error.

This marketplace is ahead of where its own scouts placed it, and its best evidence is
buried. plugins/verify-before-claim ran a negative-control experiment three times
across six scenarios and got a null every time — the base model, given a gutted stub,
already produced the behavior the skill exists to require, including reproducing two
specific reference-file procedures unprompted. It ships without a calibration case, with
every scenario and verbatim grader quote recorded. That is an in-house reproduction of the
year's most-cited context-file null, on a different task class, reached before the paper it
matches was revised. It lives in a YAML comment referenced by no tier and no doc, and its
own pointer to git history resolves to nothing because the file it names was never
committed.

Two findings recorded but deliberately NOT fixed here

Both are plugins/** changes that trigger the behavioral tier and the
demonstration-comment review gate. This PR is docs-only and I am not routing around a human
review gate to save a round trip. Flagged for a maintainer's call:

1. SKILL.md:65 asserts GitHub's "stable node_id". GitHub declines to guarantee it —
   the global-node-ID guide promises only that "it's best practice to persist the global
   node ID so you can easily reference objects across API versions," and the migration guide
   states "The legacy format will be closing down and replaced with a new format." The
   plugin's behavior (join on node_id, never full_name) is correct and unaffected;
   the rationale overstates its warrant. harness-knowledge-graph.md carried the same
   overstatement and is fixed in da22822.
2. The claim ledger cannot survive a rename. index.schema.json requires repo
   ("owner/name") with additionalProperties: false — no node_id field and no way to add
   one without a schema change — while validate-citations.sh matches with an exact
   grep -qxF against full_name. After a rename every prior claim carries the dead name,
   and GitHub's own rename doc warns that reusing an old name silently retargets the
   redirect, so a name-keyed citation can resolve to the wrong repository with the
   validator unable to detect it. This is the same defect the dive found shipped in
   OpenMetadata and in Port's default GitHub mapping, so it is the category norm rather than
   an oversight — which is exactly why it is worth fixing.

Review findings, all accepted

Two from bots on the first pass, six from a full self-review of the note against this repo's own source (4417c61).

First pass

• Codex (P2): the first draft called relationship prose "uncited by construction." False — SKILL.md:98 names cross-repo interactions as playbook content, the per-claim repo@sha:path rule covers relationship claims, and validate-citations.sh enforces traceability. Fixed in c1d3d9a; the restated gap ("traceable is not supported — for an edge the evidence is the join") is sharper than what it replaced.
• Copilot: "no single strongly-consistent query surface" asserted a guarantee GitHub doesn't publish for REST list endpoints. Fixed in 7e27eec — both occurrences now say "authoritative," which is all the argument needs.

Full review (4417c61)

Every repo-internal claim in the note was checked against source. Most held — the two corpus quotes are verbatim, the validate-citations.sh quote matches its comment exactly, and the manifest field list, node_id join, rung-0 blind spot and docs-hygiene GitOps framing all check out. Six did not:

1. diff-fleet.sh does not key on pushed_at. It joins on node_id; its content-drift bucket keys on head_sha (diff-fleet.sh:28-30), so head_sha is the diff's own key, not a separate clock beside it. The "no edge is diffable" point survives — the mechanism given for it was wrong. Introduced by c1d3d9a, the commit that fixed the Codex finding.
2. eval-ladder already has the rung. Rung 1 (discriminating corpus) sits exactly between structural and code assertion. What's missing is the retrieval-layer vocabulary, not a rung.
3. Counts in the fleet-playbook section disagreed three ways ("two invariants", a four-row table, "three of four theses"). Now: four rows, four theses, gap named as a fifth item absent from the table.
4. Provenance said five sources; the list has six. The uncounted product page is load-bearing further down.
5. "CMDB" is Harness's vocabulary, not ours. SKILL.md:45 says "not a runbook" and says nothing about a CMDB.
6. The corpus now back-links here. agentic-patterns-corpus.md and its .json twin carried the retired evidence-quality rejection with no pointer, so a reader arriving via red-gate-protocol.md:745 read a dead verdict as live. Original verdicts left intact and annotated: evidence half marked superseded, domain-fit half upheld and now scoped to git-native repo data. This is the one finding that widened the PR to a second and third file.

The note's argument and verdict are unchanged by all six.

Corrections to this repo's own claims

Two claims were found false by verification and fixed in da22822.

node_id is not documented as stable. harness-knowledge-graph.md said GitHub hands you a "stable" node_id. GitHub does not say that; both docs pages were read directly. The verdict's domain-fit argument rests on this claim, so it now says what is supported — node_id is opaque, unambiguous, and independent of the mutable full_name within a pass. The correction has a consequence: it does not license joining a manifest captured before the format migration against one captured after, which is what diff-fleet.sh does across passes. A fleet straddling that boundary sees every member as removed plus added rather than renamed, and nothing in the plugin would say why.

The cheap tier is not "under a second." AGENTS.md:63 and evals/cheap/run.sh:3 both claimed it. Measured on this checkout: 18.63s for 1290 checks across 25 plugins — off by roughly 19x, in the file that tells every contributor and every agent what the tier costs. Both now carry the measured figure, its date, and the fact that it scales with plugin count; an instruction to re-measure outlives a number that will go stale again. This is the only change in the PR outside docs/research/, and it is two comment lines.

One further correction caught in self-review before pushing 62c8b0e, listed here because it is the same class as finding 5: the companion note's verdict glossed validate-citations.sh's disclaimer as "traceable is not supported" in a way that read like a quotation. It is a paraphrase; the script's actual words are "repo-was-read is necessary but not sufficient… Semantic support of the claim by the file is the behavioral/verifier layer's job." Rewritten to cite validate-citations.sh:38-41 and paraphrase in the open.

What the research could not establish

Recorded in the notes, not buried.

Harness note: the 15–25x token claim is an unreplicated vendor benchmark on one self-chosen question; the three failure modes read as hard-won but sit on a product page with no incident write-ups; no public HQL grammar was located; the argument for an edge comes from Harness's claim, not from a measurement on our own material; and the docs page was read through a search-index fetch because developer.harness.io is egress-blocked in this container.

Companion note: whether Google Code Wiki actually regenerates on merge (its landing page is a client-rendered SPA; the strongest freshness claim in the category is untested); Code Wiki's citation granularity; whether the DeepWiki three-of-five error rate is representative — it is one table on one page, chosen because it was the best case, by someone looking for that failure; whether Cursor removed embeddings or only the documentation of them; graphify's mechanism as run rather than as read, whose adoption evidence does not meet this repo's bar — measured 2026-09-14 it has 116,700 stars and 3 subscribers, a watcher-to-star ratio two to three orders of magnitude below every control repo checked; and any independent adoption census for the wiki category, since DeepWiki's "50,000+ repos" and AGENTS.md's "60k+ projects" are both self-reported and undated.

Corpus: 156 items are recorded in the JSON twin. The ones that would change a decision: any production system that detects staleness rather than resolving it on write (the largest hole in the field, and the one docs-hygiene sits in); any published accuracy figure for the three vendor grounding APIs (all three primary doc pages read, none has one); whether the ~77% ceiling still holds, since the backing leaderboard appears frozen at 2025-09-08 with no 2026 frontier model among its 39 entries; and whether the code-evidence measurement replicates — it uses synthetic error injection, has an author conflict of interest, and is a single paper, yet it is the only measurement of the thing this repo most needs measured.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-10T17:51:44.543096Z	edb4ce5	PR opened
🔒 Security Review	✅ Completed 2026-09-10T17:52:04.373799Z	edb4ce5	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: edb4ce58bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

🟡 Changes recommended

The new note makes “strong consistency” claims about GitHub API surfaces that are not documented as guarantees and should be softened to avoid overstating factual assertions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a research note (docs/research/harness-knowledge-graph.md) that evaluates Harness’s Software Delivery Knowledge Graph against this repository’s existing “graph memory” verdict and maps any actionable takeaways to existing plugins/skills (notably eval-ladder and fleet-playbook-curator). This fits the codebase as part of the ongoing research corpus that informs which plugin ideas belong in the marketplace vs. remain out-of-scope.

Changes:

• Introduces a placement analysis concluding this is not a new plugin fit for a GitHub repo fleet domain, while updating the reasoning behind prior “no graph memory” conclusions.
• Extracts and summarizes Harness’s published eval ladder as external support for eval-ladder, including a rung distinction the note argues is missing in our current ladder framing.
• Identifies a specific, narrowly-scoped “edges missing” gap in fleet-playbook-curator while explicitly not recommending implementation yet.

File summaries

File	Description
docs/research/harness-knowledge-graph.md	New research note analyzing Harness KG claims and mapping implications to existing marketplace skills/plugins.

Review details

Suppressed comments (1)

docs/research/harness-knowledge-graph.md:104

• “gh api orgs/<owner>/repos is a strongly-consistent read” is a fairly specific consistency guarantee that GitHub doesn’t document for REST listing endpoints. To avoid overstating the claim, rephrase this as an “authoritative inventory” (or similar) without asserting strong consistency.

problem: GitHub hands you a stable `node_id`, `gh api orgs/<owner>/repos` is a
strongly-consistent read, and the join key is not ambiguous. The graph is buying
normalization nobody in this domain needs to buy.

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.