feat: make the docs index build deterministic and change-aware

[ChiragAgg5k]

Why

Every daily "refresh documentation search index" commit since Aug 27 changed only docs_index.npz, never the metadata. OpenAI embeddings are not bit-for-bit reproducible, so re-embedding unchanged text produced a different binary, which the cron turned into a commit, a patch release, and a production deploy with no documentation change behind it.

What

• Single artifact. docs_index.npz now holds the vectors, the chunk to page map, the chunk hashes, and the page metadata as an embedded meta.json member. docs_index_meta.json is gone. One file is published with one rename, so vectors and metadata can never disagree. An exclusive lock serializes overlapping manual builds.
• Hashes. Each page stores a hash of title, description, and body. Each chunk vector stores the hash of its text.
• Embedding cache. Chunks whose hash already exists in the committed artifact reuse the stored vector. Only new or edited chunks are embedded. The cache is dropped if the model or dimension changes. Identical chunk texts (273 of 5849 are shared boilerplate) embed once and share a vector.
• Fixed zip timestamps. np.savez_compressed stamps entries with the current time, which alone made two identical builds differ.
• Change report. The build prints added, changed, and removed pages and writes the same as JSON when DOCS_REPORT_FILE is set. This is the input for the follow-up that turns the report into release notes.
• New src/mcp_server_appwrite/docs_index.py owns all of this. docs_search.py reads metadata from the archive and rejects an artifact whose chunk map does not fit its page list. The daily workflow now stages only the one artifact.

What the artifact looks like

src/mcp_server_appwrite/data/docs_index.npz is a plain zip that np.load opens. Every entry carries the same fixed 1980-01-01 timestamp.

vectors.npy        35,936,384 bytes   float32 (5849, 1536)  L2-normalised chunk embeddings
chunk_page.npy         23,524 bytes   int32   (5849,)       row i -> index into meta.pages
chunk_hash.npy      1,497,472 bytes   <U64    (5849,)       sha256 of chunk i's text (embedding cache key)
meta.json           5,115,353 bytes   page metadata, read by the server and by the next build

meta.json (one page shown, content truncated):

{
  "model": "text-embedding-3-small",
  "dimension": 1536,
  "pages": [
    {
      "path": "docs/advanced/billing/abuse",
      "title": "Abuse policy",
      "description": "Guidelines on abusive behavior, prohibited activities, and reporting mechanisms under our Fair Use Policy.",
      "content": "Appwrite is committed to providing a fair, secure, and high-quality experience for all users. This Abuse Policy, as part…",
      "hash": "b3015ebf07947ec377b44bb25156f132947203d01115df2306bc82630c293b14"
    }
  ]
}

hash is sha256 over title, description, and content, so a front-matter edit counts as a change without re-embedding anything.

Build report written to DOCS_REPORT_FILE (this is a quiet day; on a real change the lists hold {path, title} entries):

{
  "pages": 586,
  "chunks": 5849,
  "chunks_embedded": 0,
  "chunks_reused": 5849,
  "changes": { "added": [], "removed": [], "changed": [] }
}

Console output of the same run:

Fetched 586 pages, skipped 73 unpublished
Wrote 5849 vectors (0 embedded, 5849 reused) across 586 pages to .../data
Changes: 0 added, 0 changed, 0 removed

Verification

• A fresh build from an empty data directory and a cached rebuild against appwrite.io produce byte-identical docs_index.npz (sha256 compared). The rebuild embedded 0 of 5849 chunks and reported 0 changes.
• Server loads the new artifact and search returns expected pages.
• tests/integration/test_docs_index.py runs end to end against appwrite.io with a local stand-in embedder (no OpenAI credentials): fetches four real pages, builds, rebuilds byte-identically with zero embedding calls, re-embeds only an edited page's chunks, reports metadata-only edits and removed pages, checks the artifact layout above, and round-trips a search through DocsSearch.
• ruff, black, pyright clean. Greptile 5/5 with no open threads after three review rounds.

Effect on the daily workflow

No further workflow change is needed for the skip: update-docs-index.yml already exits when git diff --cached --quiet, which now actually holds on quiet days. Release-note summaries from the JSON report are the next PR.

[greptile-apps]

Greptile Summary

The PR consolidates documentation vectors and metadata into one deterministic, atomically published index artifact.

• Reuses embeddings by chunk hash and invalidates the cache when the model or dimension changes.
• Reports page changes using hashes that include title, description, and content.
• Serializes publication with an exclusive lock and unique temporary file.
• Updates the runtime loader, refresh workflow, documentation, and tests for the single-artifact format.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/mcp_server_appwrite/docs_index.py	Adds deterministic hashing, compatible embedding reuse, change reporting, exclusive locking, and atomic single-file publication; the previously reported metadata, cache-dimension, and publication issues are addressed.
src/mcp_server_appwrite/docs_search.py	Loads vectors and page metadata from the unified archive and rejects invalid chunk-to-page mappings.
scripts/build_docs_index.py	Delegates index construction to the new deterministic builder and optionally writes a JSON change report.
.github/workflows/update-docs-index.yml	Stages only the unified index artifact, allowing unchanged deterministic rebuilds to skip releases.
tests/integration/test_docs_index.py	Exercises deterministic rebuilds, selective re-embedding, change reporting, artifact loading, and search behavior.
tests/unit/test_docs_search.py	Updates loader coverage for embedded metadata and malformed chunk mappings.

_{Reviews (6): Last reviewed commit: "test: replace docs index unit tests with..." | Re-trigger Greptile}