Replace the search index with Pagefind by enf0rc3 · Pull Request #3371 · OctopusDeploy/docs

enf0rc3 · 2026-08-17T04:35:01Z

Replaces the hand-rolled search.json index with Pagefind, behind the SearchEngine seam the overlay already talks to. Pagefind indexes the HTML the build emits, so article body text is searchable for the first time — previously only titles, headings and extracted keywords were.

Build

src/integrations/pagefind-index.ts runs on astro:build:done and indexes dist/docs/**/*.html into dist/docs/pagefind/.

Registered before pruneDist() in astro.config.mjs, which deletes every top-level entry outside its allowlist.
Indexes from dist/docs, so stored URLs are relative to the /docs/ prefix the site is proxied under.
includeCharacters: '.#{}<>+$_' keeps technical punctuation in the index, so <head>, .nupkg and #{Octopus.Environment.Name} are findable as written. Pagefind strips punctuation from both index and query by default, and <head> matched nothing at all.
Logs how many pages were indexed and warns below 1,000. A broken body scope is a silent failure otherwise: search still works, it is just full of the wrong pages.

Pagefind's Node API is imported statically, because astro:build:done fires after Vite's module runner has closed and a dynamic import from inside the hook cannot resolve.

What gets indexed

Default.astro decides:

data-pagefind-body on the <article>, which keeps sidebar and footer text out of every page's terms and excludes the 1,409 redirect stubs for free, since they never reach this layout.
data-pagefind-filter supplies the facet tabs, matching classify() in search-engine.ts.
data-pagefind-meta carries the breadcrumb the result row prints.
data-pagefind-sort carries a date key, so a recency sort is available in the index without a rebuild. Nothing in the overlay offers that control yet.
data-pagefind-ignore="all" on the page-action and markdown-link blocks, which are identical on every page.
ArticleHeader.astro takes a searchWeight prop, which lifts an h1 to 10 on pages three segments deep or shallower.

src/plugins/pagefind-image-attrs.js puts data-pagefind-index-attrs on every <img> carrying alt or title text. A screenshot's alt text is often the only place a UI label appears in prose. Image.astro carries the attribute itself, because raw <img> tags are passed through as HTML and never become hast nodes.

Query side

src/scripts/search-engine-pagefind.ts:

metaWeights: { title: 8, trail: 0.5 }. The breadcrumb is derived from the URL, so every page under /docs/projects/ matches a search for "projects" as well as the Projects page does. Demoting it is worth seven points of Success@5.
A reorder that puts a page the query names — by title or last URL segment — ahead of the pages inside it, with a per-segment depth discount behind that. data-pagefind-weight alone measured as a no-op; this is worth about twenty points of Success@5.
A score floor of 8 on the top result, so a query with no answer returns empty. Pagefind ranks whatever shares a few letters and returns it as confidently as a real hit. Calibrated against every term in the search log: the weakest genuine query (cli) scores 9.5, keyboard mash 6.0. It suppresses 17 of the 33 terms with no right answer and costs none of the 57 that have one. Asked of the unfiltered scores, because narrowing to a tab only removes documents.
Up to two matched headings under each of the leading three results, each its own role="option", so a reader can arrow onto a section and land there.
excerptLength: 20, sized to the single line the result row gives it.
Facet counts come from the unfiltered match set, so the tab strip keeps showing what the other tabs hold while one is selected.

SearchEngine gains eager and preload. Pagefind sets eager: true — 118KB of runtime and WASM, warmed on page load — and preloads the chunks a query needs on each keystroke, ahead of the debounce.

Overlay

docs-search.ts:

Two character minimum query length, applied in the debounce, in the preload, and in the ?q= deep-link path. A single character matches a large share of the index, which for a chunked engine means fetching a large share of it: 975ms against 837ms at two characters and under 200ms beyond that.
A role="status" live region reports the result count and the empty state. The combobox roles said a listbox existed and which row was active; nothing said what came back. It sits outside the search body, which is hidden until there is a query — a region placed inside would leave and re-enter the accessibility tree on every open and drop the first announcement.

Removed

search.json.ts, search-engine-legacy.ts, the Porter stemmer, the string helpers and the synonym map: 645 lines. Drops the html-to-text and keyword-extractor dependencies and adds pagefind. It is a regular dependency rather than a dev one, because astro.config.mjs imports it statically and a production-only install would fail to load the config at all.

Tests

tests/api-page.spec.ts read /docs/search.json to check the under-construction API reference stays out of search. It now asks through the overlay, searching accounts — a word naming pages both inside and outside the reference, so an empty list cannot pass the test by accident.

Notes

Azure Blob and Front Door serve .pf_fragment, .pf_index, .pf_meta and .pf_filter correctly. Pagefind reads them as bytes, so the empty Content-Type is fine, and there is no .wasm file to configure — the wasm ships as wasm.en.pagefind.
Pagefind's options() merges rather than replaces, so anything recalibrating the score floor has to start from a clean page or every score comes back depressed.

Known gaps

Thirty results, with no paging and nothing saying the list is capped.
Cold first search is slower than the old engine's. Warming on page load covers most of it.
The Integrations facet has no content: all 153 pages under /docs/api-and-integration/ are redirect stubs. The tab hides itself, so the mapping needs a real home or the tab should go.
Plausible.astro listens for a searched event that nothing fires, so there is still no search analytics.

team-marketing-branch-protections · 2026-08-17T22:47:25Z

Pull request environment is available at https://stoctodocspr3371.z22.web.core.windows.net.

You can view the ephemeral environment status in Octopus Deploy.

This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity.

enf0rc3 · 2026-08-17T22:53:55Z

Verified on the ephemeral environment

https://stoctodocspr3371.z22.web.core.windows.net/docs — this closes the one open risk from the plan.

Every Pagefind file type serves. pagefind.js as application/javascript, pagefind-entry.json as application/json, the wasm as application/x-gzip (Azure content-sniffs it, which is harmless because Pagefind reads it as bytes rather than streaming-compiling). The index chunks and fragments load too. Nothing was skipped by the upload step and no MIME allowlist was needed.

Search is correct on real hosting. tentacle → 248 results, All 248 / Docs 183 / API 14 / CLI 51 — identical to local. Deployed pagefind-entry.json reports page_count: 1251, matching the build. Cold first search 2.31s over the internet.

Result links stay on the host. Zero absolute hrefs; clicking a result landed on stoctodocspr3371.../docs/octopus-rest-api/tentacle.exe-command-line/.

One thing to know about the hosting

Azure Blob static website does no compression — Accept-Encoding: gzip returns everything with no Content-Encoding. It barely matters here, because Pagefind's .pf_* chunks are already compressed binary and pagefind.js is 45 KB. It matters a great deal for #3372, which pays 11.4 MB uncompressed rather than the 2.4 MB I estimated. Details in that PR.

Bake-off result

	Legacy	Pagefind	Orama
First search, over the wire	1,860 KB	278 KB	11,989 KB
Each later search	0 KB	40–160 KB	0 KB
Cold on staging	—	2.31s	2.16s
At 1.6 Mbps (arithmetic)	—	~1.4s	~60s
Typo tolerance	no	no	no (did not work)
Hosting risk	—	disproven	none

Both feel the same on a fast connection. The difference is what happens below that, and payload was the defect this work set out to fix.

Recommend merging this and closing #3372.

Indexes the built HTML at the end of the build and puts Pagefind behind the SearchEngine seam. Body text is indexed for the first time, so a phrase that appears in an article but not its title or headings is now findable. The index goes to dist/docs/pagefind, before pruneDist would delete it, and is built from dist/docs so stored URLs match the /docs/ prefix the client sets as its basePath. data-pagefind-body on the article is what scopes indexing and, as a side effect, excludes all 1,409 redirect stubs: they render through a minimal layout that has no article at all. 1,250 pages are indexed, matching the eligible count the markdown emitter reports. Removes the search.json endpoint and the client scoring it fed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three configuration corrections from the Pagefind audit. Its indexing and ranking were already idiomatic, so none of this touches either. preload() now runs on each keystroke ahead of the debounced search, which is what Pagefind documents as the way to fetch the chunks a query needs while the reader is still typing. Warming moves to page load for Pagefind only. That is 118KB of runtime and WASM against a cold first result measured at 2239ms. Which side of that trade is right differs by engine, so `SearchEngine` carries an `eager` flag and the engine states its own answer rather than the overlay assuming one — Orama pays for its whole index on every navigation and must stay lazy. excerptLength drops from its 30-word default to 20, to fit the single line the result row gives it. Sub-results are deliberately not used. They render heading-scoped matches with their own anchors, which is real, but the corpus fights it: the CLI tree has 511 headings across only 49 distinct titles, 213 of them "Learn more" and 176 "Examples", and the API tree 291 across 100. Only documentation pages are distinctive, at 4160 titles across 5302 headings. Suppressing the boilerplate needs a threshold nobody has justified yet, and the audit is clear that sub-results move neither Success@1 nor Success@5 — so this stays out of a comparison it cannot decide, and can be added later against whichever engine wins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Six capabilities that were available and unused. Indexing. A depth-derived data-pagefind-weight on the page header, so a bare section name ranks the section above the pages inside it — the largest single source of missed traffic on the search terms readers actually type. Image alt text becomes searchable via data-pagefind-index-attrs, which has no inheritance and so needs a hast plugin to reach every markdown image. A date sort key is indexed, and a frontmatter title is registered as fallback metadata for any page whose heading comes back empty. Querying. highlightParam so result links carry the query for the destination page to highlight, and sub_results rendered as their own options beneath their page, so a match inside a long page can be arrowed onto and lands on that heading. The weight is the one worth measuring: applied as a client-side reorder it took Pagefind from 42% to 62% weighted Success@5 on real search terms, and from 73% to 95% on top-visited pages. At index time it has no top-30 reach limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Built and measured locally, which corrected two things. data-pagefind-weight on its own did nothing. Scaling it with depth scored 42% weighted Success@5 on real search terms against the deployed 42%, because plus or minus a point around Pagefind's default h1 weight of 7 is far too weak a lever. It now boosts shallow pages to 10 and leaves deep pages at the default, and the reordering that does the work lives in the engine: the same name-then-depth rule the Orama worker uses, with the same constant. Three matched headings across thirty results made two thirds of the list headings, which buried the pages a reader is choosing between. Only the leading three results carry them now, two each — 15% of rows rather than 67%. Real search terms go 42% to 62% Success@5 and 19% to 43% at rank one. Top-visited pages go 73% to 95% and 37% to 82%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three gaps, all measured on a local build. Pagefind strips punctuation from index and query alike, so `<head>` matched nothing and `Octopus.Action.Package` was indistinguishable from prose using those words. includeCharacters keeps `.#{}<>+$_`, chosen from what appears in this corpus and in the search logs. Indexing is additive — both `head` and `<head>` are stored — so ordinary searches are untouched and the index grew 1%, 8.8MB to 8.9MB. `<head>` went from 0 results to 55, and `octopus.server.exe` now leads with the octopus.server.exe reference page. metaWeights sets title to 8 and trail to 0.5. The breadcrumb is derived from the URL, so every page under /docs/projects/ carries "Projects" and matched a search for it as well as the Projects page did. It belongs in the result row, not the ranking. Both ignore blocks move to `all`. Bare ignore still lets a title or metadata be read out of the block, and neither holds anything worth reading. Real search terms: 62% to 81% Success@5, 43% to 52% at rank one. Top-visited pages: 95% to 98%, and 82% to 89%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The test read /docs/search.json, which this branch deletes along with the rest of the legacy engine, so it was parsing a 404 page as JSON. It now drives the search overlay instead. The two spikes ship indexes of different shapes — one JSON document, one directory of compressed chunks — and neither is readable the way the old one was, but what has to hold is the same either way: a reader searching a word the API reference is full of must not be sent into it. `accounts` names pages both inside the API reference and outside it, so the result list is never empty. A query that matched nothing would pass without proving anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two changes, both measured on a local build. Searches now need three characters. Almost the whole of Pagefind's keystroke lag lives in the two keystrokes nobody could act on anyway: one character costs 975ms and two cost 837ms, against 342ms at three and under 200ms beyond. On a desktop this takes p95 from 927ms to 347ms, and the median from 346ms to 200ms. A search whose best match scores under 8 now returns nothing. Pagefind has no notion of a query it cannot answer — asked for `sssieddqxsx`, the most-typed term in the log, it offered three security articles. Calibrated against every logged term: the weakest genuine query scores 9.5, the mash 6.0. The floor suppresses 17 of the 33 terms with no right answer and costs none of the 57 that have one. Relevance is unchanged either way: 81% Success@5 on real terms and 98% on top-visited pages, the same as before. Nonsense returning nothing went from 0% to 76%. The floor was first set at 6, which did nothing, because it was calibrated while earlier sweep parameters were still active — `options()` merges rather than replaces, and every score came back depressed. calibrate-score-floor.mjs starts from a clean page for that reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The overlay carried the combobox roles without the one part that speaks. A screen reader was told a listbox existed and which row was active, and never how many results arrived or that none had. The live region sits outside the body, which is hidden until there is a query. Inside it, the region would leave and re-enter the accessibility tree on every open, and a region that has only just appeared does not reliably announce the first thing written to it. It is clipped rather than hidden for the same reason: `display: none` would silence it. It speaks from `render`, which runs once per settled query, so it keeps pace with the results rather than with the keyboard. Below the three character minimum it says nothing, matching the panel's own waiting state. Pagefind's own Component UI does this and ours did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

They pointed at the bake-off and at the other candidate engine, and both are gone from the tree. The rationale each comment carried is kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

- preload ignored MIN_QUERY_LENGTH, so the first keystrokes still fetched the chunks the guard exists to avoid. - A failed fragment fetch rejected the whole search and left the previous query's rows under the new text. - The score floor was applied to the filtered results while the tab counts came from the unfiltered ones, so a tab could advertise results and then show none. It is now asked of the unfiltered scores. - highlightParam was set and then stripped off every link, so it is dropped. - The minimum query length goes to two characters. - A missing fragment directory threw instead of reaching the page-count warning that exists to report it. - Future-dated pages are indexed, and the comment claiming otherwise is fixed. - The live region counted pages while the listbox held pages and sections. - Removed an unreachable filter, logged the swallowed index-load error, and corrected the deleted call site named in underConstruction.ts. pagefind moves to a real dependency: astro.config.mjs imports it statically, so a production-only install could not load the config at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Removed the running commentary on approach, the measurements that only justified a decision at the time, and a reference to a calibration script that is not in the repo. What is left explains Pagefind behaviour and coupling that the code does not show on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The previous pass cut too far in four places. Restored: what each character in `includeCharacters` is for, the evidence behind the score floor and the `options()` merge that skews a recalibration run, and which of the two section constants is which. The comment on the API reference test claimed the search term was what stopped an empty list passing; the visibility assertion is. `hashOf` is gone. `URL.hash` already returned the same string, so the anchor is read off the URL that is being parsed anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

- One failed fragment fetch took the whole result set with it. allSettled drops the row it belongs to and keeps the rest. - Asking the score floor of the unfiltered corpus fixed a false negative and opened a false positive: a tab could hold nothing but accidents scoring far below the query and still render them. The floor now answers both questions — nothing anywhere returns empty, nothing under this tab returns no rows while keeping the counts that say where the answers are. - The two searches run together, so the second no longer sits in front of every fragment fetch. - An index with no pages in it fails the build instead of warning. Every query returning nothing must not ship green. - The live region counts pages the way the tab strip does, and names the matched sections separately. - components.mdx still told readers production used legacyEngine over search.json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

enf0rc3 mentioned this pull request Aug 17, 2026

Spike: Orama #3372

Closed

enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch from 5ac3f67 to 6cbd53f Compare August 17, 2026 21:23

enf0rc3 requested a review from steve-fenton-octopus as a code owner August 17, 2026 21:23

enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch 3 times, most recently from 0d065ca to 1c011db Compare August 17, 2026 22:38

enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch 5 times, most recently from d9d2279 to 49885ce Compare August 18, 2026 20:59

enf0rc3 changed the title ~~Spike: Pagefind~~ Replace the search index with Pagefind Aug 19, 2026

enf0rc3 and others added 12 commits August 19, 2026 17:21

Add pagefind to the project dictionary

8278e5a

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rewrite the search comments that referred to the spike

f6e0c0f

They pointed at the bake-off and at the other candidate engine, and both are gone from the tree. The rationale each comment carried is kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch from 1993ecf to f47ccfc Compare August 19, 2026 05:27

enf0rc3 and others added 2 commits August 19, 2026 17:51

Navigation Menu

Uh oh!

Replace the search index with Pagefind - #3371

Replace the search index with Pagefind#3371
enf0rc3 wants to merge 14 commits into
mainfrom
willlaugesen/docs-search-pagefind

enf0rc3 commented Aug 17, 2026 •

edited

Loading

Uh oh!

team-marketing-branch-protections commented Aug 17, 2026

Uh oh!

enf0rc3 commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

enf0rc3 commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build

What gets indexed

Query side

Overlay

Removed

Tests

Notes

Known gaps

Uh oh!

team-marketing-branch-protections commented Aug 17, 2026

Uh oh!

enf0rc3 commented Aug 17, 2026

Verified on the ephemeral environment

One thing to know about the hosting

Bake-off result

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

enf0rc3 commented Aug 17, 2026 •

edited

Loading