WIP: Make API-only the default build and exclude site pages from the API bundle

architecture.md

@@ -0,0 +1,210 @@
+# Architecture
+
+This document describes how `@patternfly/patternfly-doc-core` is structured, how
+its build works, and how the **documentation API** and the **documentation site
+UI** are decoupled so the API can be built by itself.
+
+## Overview
+
+The project is a single [Astro](https://astro.build/) application driven by a
+thin CLI (`cli/cli.ts`). From the same source it produces two distinct things:
+
+1. **The documentation API** — machine-readable JSON/text endpoints under
+   `/api/**`, intended for LLM agents, MCP servers, and tooling.
+2. **The documentation site UI** — the human-facing HTML pages (home page and
+   the rendered component/pattern documentation).
+
+The API is cheap to build. The site UI is expensive because it renders
+PatternFly React components and MDX content. Therefore:
+
+> **The API is built by default. The full site UI is opt-in (`--site`).**
+
+## Directory layout
+
+```
+cli/                     # Build CLI (compiled to dist/cli, exposed as the package bin)
+  cli.ts                 # Commands: setup, init, start, build, deploy, ...
+
+src/
+  pages/                 # Astro file-based routing — ALWAYS built
+    api/**               # The documentation API (SSR + a few static routes)
+    apiIndex.json.ts     # Prerendered /apiIndex.json   (API depends on this)
+    iconsIndex.json.ts   # Prerendered /iconsIndex.json (icons API depends on this)
+    props.json.ts        # Prerendered /props.json
+    props.ts             # SSR /props endpoint
+    404.astro            # Lightweight; shows API-only messaging when PF_API_ONLY=true
+
+  site-pages/            # Site UI pages — NOT in Astro routing by default.
+    index.astro          # Full home page (injected only for full-site builds)
+    api-landing.astro    # Lightweight "/" stub (injected for API-only builds)
+    [section]/[...page].astro     # Docs pages (heavy: React + MDX)
+    [section]/[page]/[tab].astro  # Docs tab pages (heavy: React + MDX)
+
+  components/            # React + Astro components used by site-pages
+  layouts/              # Astro layouts (Main.astro pulls in PatternFly CSS + nav)
+  utils/                # Shared helpers (used by both API and site)
+    apiIndex/           # API index generate/get/fetch
+    apiRoutes/          # Content matching, example parsing, collections
+    icons/              # Icon metadata + SVG helpers
+    propsData/          # Props fetch helper
+  content.ts            # Generated list of content collections
+
+astro.config.mjs        # Astro config + the `optional-site-pages` integration
+wrangler.jsonc          # Cloudflare Pages config (deploy target)
+pf-docs.config.mjs      # Consumer-provided docs config (content sources, outputDir)
+```
+
+## The API / site split
+
+### Why they can be separated
+
+The API side is self-contained. Its route handlers read only from **source**
+inputs, never from the built site output:
+
+- SSR API routes (`prerender = false`) fetch the prerendered `/apiIndex.json` at
+  runtime (`src/utils/apiIndex/fetch.ts`) — keeping the Cloudflare Worker bundle
+  small instead of embedding the ~500 KB index.
+- Text/example routes read the original Markdown/MDX from the content source via
+  `src/utils/apiRoutes/*`.
+- Props routes read `props.json`; icon routes read `@patternfly/react-icons`
+  static output plus the prerendered `/iconsIndex.json`.
+
+The heavy, memory-intensive work lives exclusively in the site-UI pages under
+`src/site-pages/` — the only files that import `@patternfly/react-core`,
+`LiveExample`, `SectionGallery`, the layouts, and render MDX. Nothing in
+`src/pages/api/**` depends on those pages being built.
+
+### How the separation is implemented
+
+Astro builds everything it finds under `src/pages/`. To keep the site pages out
+of the default build **without breaking their relative imports**, they live in
+`src/site-pages/` — a sibling directory at the *same depth* as `src/pages/`.
+Because both are direct children of `src/`, every relative import inside the
+moved files (`../layouts/...`, `../../components/...`, `../../content`, etc.)
+resolves unchanged.
+
+Being outside `src/pages/`, those files are no longer file-routed, so their
+modules are **not part of the SSR bundle** for an API-only build.
+
+They are added back to routing on demand by the `optional-site-pages`
+integration in `astro.config.mjs`, using the Astro `command` plus an env var:
+
+```js
+'astro:config:setup': ({ command, injectRoute }) => {
+  const buildFullSite = command === 'dev' || process.env.BUILD_SITE === 'true'
+  if (buildFullSite) {
+    injectRoute({ pattern: '/',                       entrypoint: '.../src/site-pages/index.astro' })
+    injectRoute({ pattern: '/[section]/[...page]',    entrypoint: '.../src/site-pages/[section]/[...page].astro' })
+    injectRoute({ pattern: '/[section]/[page]/[tab]', entrypoint: '.../src/site-pages/[section]/[page]/[tab].astro' })
+  } else {
+    injectRoute({ pattern: '/',                       entrypoint: '.../src/site-pages/api-landing.astro' })
+  }
+}
+```
+
+Rules:
+
+- **Dev server** (`command === 'dev'`): the full site is always injected, so
+  local development is unchanged.
+- **Build** (`command === 'build'`): the full site is injected only when
+  `BUILD_SITE=true`; otherwise a lightweight `api-landing.astro` stub is injected
+  at `/` so the route still resolves without pulling in any heavy modules.
+
+Entrypoints are resolved with
+`fileURLToPath(new URL('./src/site-pages/...', import.meta.url))` so they resolve
+correctly whether the package runs from this repo or from a consumer's
+`node_modules`.
+
+### Two env vars, two jobs
+
+| Var            | Set when            | Effect                                                        |
+| -------------- | ------------------- | ------------------------------------------------------------ |
+| `BUILD_SITE`   | `build --site`      | Integration injects the full site routes.                    |
+| `PF_API_ONLY`  | `build` (default)   | Remaining stub pages (e.g. `404.astro`) show API-only copy.  |
+
+## CLI and scripts
+
+`cli.ts build` is **API-only by default**; pass `--site` for the full site:
+
+```ts
+if (site) {
+  process.env.BUILD_SITE = 'true'   // inject full site routes
+} else {
+  process.env.PF_API_ONLY = 'true'  // API-only messaging on stub pages
+}
+```
+
+| Script              | Builds          | Command                                    |
+| ------------------- | --------------- | ------------------------------------------ |
+| `npm run build`     | API only        | `... cli.js build`                         |
+| `npm run build:all` | API + site UI   | `... cli.js build --site`                  |
+| `npm run start`     | Full site (dev) | `... cli.js start` → `astro dev`           |
+
+Both build scripts run under `--max-old-space-size=8192`; SSR bundling of the API
+routes plus the content collections is memory-heavy on its own, so the elevated
+heap is required even for the API-only build.
+
+## Build pipeline (`cli.ts build`)
+
+`buildProject()` runs these steps before invoking Astro:
+
+1. `updateContent()` — regenerate `src/content.ts` from the content sources in
+   `pf-docs.config.mjs`.
+2. `generateProps()` — generate `props.json` (component prop metadata).
+3. `initializeApiIndex()` — seed `apiIndex.json` from a template if absent.
+4. `transformMDContentToMDX()` — convert configured `.md` content to `.mdx`.
+5. `build({ root, outDir })` — run Astro. Prerendered endpoints
+   (`apiIndex.json.ts`, `iconsIndex.json.ts`, `props.json.ts`) are generated;
+   site pages are included only when `BUILD_SITE=true`.
+6. Copy `apiIndex.json` into the docs output so it can be fetched by SSR API
+   routes at runtime.
+
+Output goes to `<outputDir>/docs` (default `dist/docs`), which `wrangler.jsonc`
+deploys to Cloudflare Pages.
+
+### What the output contains
+
+| Path in `dist/docs`         | API-only (default) | `--site`      |
+| --------------------------- | :----------------: | :-----------: |
+| `api/`                      |         ✓          |       ✓       |
+| `apiIndex.json`, `iconsIndex.json`, `props.json` | ✓ | ✓        |
+| `_worker.js`, `_routes.json`|         ✓          |       ✓       |
+| `index.html`                | ✓ (stub landing)   | ✓ (full home) |
+| `404.html`                  |         ✓          |       ✓       |
+| `components/`, `patterns/`, `foundations-and-styles/`, `extensions/` (HTML) | — | ✓ |
+
+### Measured build cost (this repo)
+
+| Build            | Wall time | Peak RSS |
+| ---------------- | --------- | -------- |
+| API-only (`build`)   | ~44 s | ~6.7 GB |
+| Full (`build:all`)   | ~57 s | ~7.2 GB |
+
+The API-only build is faster mainly because it skips prerendering the many
+component/pattern HTML pages and the client-side JS build for them. A large
+portion of peak memory comes from SSR-bundling the API routes together with the
+content collections, which happens in both builds.
+
+## API index data flow
+
+The API index (`src/utils/apiIndex/`) exists in three forms for three contexts:
+
+- **generate** (`generate.ts`) — builds the index from Astro content collections
+  at build time (`getCollection`); writes `apiIndex.json`.
+- **get** (`get.ts`) — reads `apiIndex.json` from the filesystem; used in
+  build-time contexts (e.g. the prerendered `apiIndex.json.ts` endpoint).
+- **fetch** (`fetch.ts`) — fetches `/apiIndex.json` over HTTP at runtime; used by
+  the SSR API handlers so the large index is never bundled into the Worker.
+
+## Deployment
+
+`cli.ts deploy` shells out to `wrangler pages deploy`, publishing `dist/docs`
+(per `wrangler.jsonc`). Because the API build is self-sufficient, a deploy can
+ship the API alone or the API plus the site UI, depending on which build script
+produced `dist/docs`.
+
+## Local development
+
+`npm run start` → `cli.js start` runs `astro dev`. In dev the
+`optional-site-pages` integration always injects the full site routes, so the
+site and the API are both available locally.

astro.config.mjs

@@ -2,12 +2,64 @@
 import { defineConfig } from 'astro/config';
 import react from '@astrojs/react';
 import mdx from '@astrojs/mdx';
+import { fileURLToPath } from 'node:url';
 
 import cloudflare from '@astrojs/cloudflare';
 
+/**
+ * The documentation site UI pages live in `src/site-pages` rather than
+ * `src/pages`, so they are NOT part of Astro's default file-based routing and
+ * are never bundled unless explicitly requested. This makes the API-only build
+ * (the default) cheap: the memory-heavy site pages are excluded entirely.
+ *
+ * The API routes and their supporting static endpoints stay in `src/pages` and
+ * are always built.
+ *
+ * Routing at `/`:
+ *   - Full site build (dev, or `BUILD_SITE=true`): the real home + docs pages
+ *     are injected.
+ *   - API-only build (the default): a lightweight `api-landing.astro` stub is
+ *     injected at `/` instead, so `/` still resolves without pulling in any
+ *     heavy modules.
+ *
+ * See architecture.md for the full rationale.
+ */
+const optionalSitePages = {
+  name: 'optional-site-pages',
+  hooks: {
+    'astro:config:setup': ({ command, injectRoute }) => {
+      const entry = (relativePath) =>
+        fileURLToPath(new URL(relativePath, import.meta.url));
+
+      // Always build the full site in dev; in build, only when BUILD_SITE=true.
+      const buildFullSite = command === 'dev' || process.env.BUILD_SITE === 'true';
+
+      if (buildFullSite) {
+        injectRoute({
+          pattern: '/',
+          entrypoint: entry('./src/site-pages/index.astro'),
+        });
+        injectRoute({
+          pattern: '/[section]/[...page]',
+          entrypoint: entry('./src/site-pages/[section]/[...page].astro'),
+        });
+        injectRoute({
+          pattern: '/[section]/[page]/[tab]',
+          entrypoint: entry('./src/site-pages/[section]/[page]/[tab].astro'),
+        });
+      } else {
+        injectRoute({
+          pattern: '/',
+          entrypoint: entry('./src/site-pages/api-landing.astro'),
+        });
+      }
+    },
+  },
+};
+
 // https://astro.build/config
 export default defineConfig({
-  integrations: [react(), mdx()],
+  integrations: [react(), mdx(), optionalSitePages],
   vite: {
     ssr: {
       noExternal: ["@patternfly/*", "react-dropzone"],

cli/cli.ts

@@ -117,10 +117,22 @@ async function initializeApiIndex(program: Command) {
 }
 
 async function buildProject(program: Command): Promise<DocsConfig | undefined> {
-  const { verbose, apiOnly } = program.opts()
-
-  if (apiOnly) {
+  const { verbose, site } = program.opts()
+
+  // API-only is the default. The full documentation site UI is opt-in via
+  // `--site`, which tells the astro config to inject the site routes
+  // (BUILD_SITE). When not building the site, PF_API_ONLY switches the
+  // remaining stub pages (e.g. 404) to their API-only messaging.
+  if (site) {
+    process.env.BUILD_SITE = 'true'
+    if (verbose) {
+      console.log('Building API and documentation site pages')
+    }
+  } else {
     process.env.PF_API_ONLY = 'true'
+    if (verbose) {
+      console.log('Building API only (pass --site to also build the site pages)')
+    }
   }
 
   if (!config) {
@@ -199,7 +211,7 @@ program.name('pf-doc-core')
 program.option('--verbose', 'verbose mode', false)
 program.option('--props', 'generate props data', false)
 program.option('--dry-run', 'dry run mode', false)
-program.option('--api-only', 'only build API and component pages, skip standalone content pages', false)
+program.option('--site', 'also build the documentation site UI pages (API only by default)', false)
 
 program.command('setup').action(async () => {
   await Promise.all([
@@ -221,10 +233,8 @@ program.command('init').action(async () => {
 })
 
 program.command('start').action(async () => {
-  const { apiOnly } = program.opts()
-  if (apiOnly) {
-    process.env.PF_API_ONLY = 'true'
-  }
+  // The dev server always serves the full site (see the astro config), so no
+  // API-only handling is needed here.
   await updateContent(program)
   await initializeApiIndex(program)
 

package.json

@@ -12,6 +12,7 @@
     "start:cli": "npm run build:cli && node ./dist/cli/cli.js start",
     "start:astro": "astro dev",
     "build": "npm run build:cli && node --max-old-space-size=8192 ./dist/cli/cli.js build",
+    "build:all": "npm run build:cli && node --max-old-space-size=8192 ./dist/cli/cli.js build --site",
     "build:astro": "astro check && astro build",
     "build:cli": "tsc --build ./cli/tsconfig.json",
     "build:cli:watch": "tsc --build --watch ./cli/tsconfig.json",

src/pages/[section]/[...page].astro → src/site-pages/[section]/[...page].astro

@@ -30,12 +30,6 @@ import {
 import DocsTables from '../../components/DocsTables.astro'
 
 export async function getStaticPaths() {
-  const apiOnly = process.env.PF_API_ONLY === 'true'
-
-  if (apiOnly) {
-    return []
-  }
-
   const collections = await Promise.all(
     content.map(
       async (entry) => await getCollection(entry.name as 'textContent'),

src/pages/[section]/[page]/[tab].astro → src/site-pages/[section]/[page]/[tab].astro

@@ -31,12 +31,6 @@ import DocsTables from '../../../components/DocsTables.astro'
 import { addDemosOrDeprecated, getDefaultTab } from '../../../utils'
 
 export async function getStaticPaths() {
-  const apiOnly = process.env.PF_API_ONLY === 'true'
-
-  if (apiOnly) {
-    return []
-  }
-
   const collections = await Promise.all(
     content.map(
       async (entry) => await getCollection(entry.name as 'textContent'),