Doc: Add Core Internals handbook under docs/ - #1971
Conversation
|
That's awesome! I'll try to read it, cannot say could read the whole document within a day, but try my best ) |
Thanks a lot! Some oversights are inevitable. Really appreciate your time — no pressure to finish it in one day, just take your time. |
Add a developer-oriented handbook for the Cloudberry kernel: 21
chapters and 104 sections covering storage, indexing, the catalog,
the executor (single-node and distributed), the dual optimizer
(PostgreSQL planner and ORCA), transactions, HA/recovery, and
resource management. It is organised into five parts after the
structure of Rogov's "PostgreSQL 14 Internals".
The content is written for Docusaurus so that apache/cloudberry-site
can pull it in as a submodule with a sparse checkout limited to
docs/core-internals, served by its own docs plugin instance with an
autogenerated sidebar -- the same arrangement already used for the
PXF documentation. It renders on GitHub as-is too.
What the chapters contain:
- 372 code excerpts, each fenced with title="file:line" so a
reader can go straight to the source it was read from
- 487 psql sessions captured from a running cluster, split into
an input block and an output block
- 113 mermaid diagrams and 64 SVG figures
- internal cross-references that resolve to section anchors and
are validated at build time
Host names, addresses and install paths belonging to the machine
the sessions were captured on are replaced with documentation-safe
equivalents of identical width, so the column alignment of captured
output is preserved.
Every file carries the ASF license header, so no pom.xml RAT
exclusions are needed. The tree was verified against Docusaurus 3
with MDX v3: no compilation errors, no broken links, no broken
anchors.
See docs/core-internals/README.md for the layout, the conventions
and a note on maintenance -- file and line references describe a
point in the tree's history and will drift, which the
title="file:line" convention makes machine-checkable.
fa603a4 to
4208df9
Compare
Review on apache#1971 flagged two Chinese strings in ch01. Both came from the generator, not the handbook, and tracing them turned up a third problem and a fourth string: - "本节要点" (87 sections). This printed each section's `map` keyword array. The handbook's own renderer never displays that array -- it only uses `mapSkip` to decide which sections appear in the "In this chapter" list. Confirmed with jsdom: the string "cdbhash -> segment" exists only inside the data <script>, in no rendered element. So these 87 lines were content the handbook does not have. Dropped rather than translated. - "| 部分 | 含义 |" (10 field diagrams). Invented table headers. The handbook renders these as label-plus-description cells with no header row, so they are now a definition list, which needs no header text at all. - ":::info 参见" (1 cross-reference). The handbook marks these with a CSS "<->" prefix and no word, so the admonition is untitled now. - The index page's own prose. Separately, the handbook itself cited the Chinese edition of Rogov's "PostgreSQL 14 Internals" in three places. Two already carried English glosses, making the Chinese redundant; one figure reference and one term did not. All three are now English. Chinese characters remaining in docs/core-internals: zero. Rebuilt against Docusaurus 3 / MDX v3: no compilation errors, no broken links, no broken anchors.
|
Fixed in ddaedff — thank you, both were real bugs and tracing them turned up two more. Neither string came from the handbook; both came from the generator that converts it to
Separately, the handbook itself cited the Chinese edition of Rogov's PostgreSQL 14 Internals Chinese characters under The PDF linked in the description is from before this fix; I will refresh it. |
Three problems found by auditing the generated output against the handbook and against the tree it cites. 1. The pages referenced an MDX component that the PR did not ship. 37 figures are laid out with HTML and CSS rather than drawn as SVG, and are handed to a component as a string because their markup (1,062 inline style attributes, 46 void tags, 2 blocks with unbalanced tags) is not valid JSX. The component and its stylesheet lived only in my local preview site, so anyone building from this branch would have hit an undefined component and a failed build. They now ship in docs/core-internals/_components/, and each page that needs the component imports it, so a consuming site registers nothing. Verified by building against a site with an empty customCss that registers no components: 23 pages, no MDX errors, no broken links, no broken anchors, and all 101 figures present -- 64 images and 37 HTML figures. 2. One code citation pointed at the wrong directory: src/backend/access/transam.h should be src/include/access. The line number was right; the content at transam.h:33 is the FrozenTransactionId block the excerpt quotes. Found by checking every title="file:line" citation against git ls-tree: 361 of 370 path fragments resolve exactly, and this was the only one wrong rather than abbreviated or generated. 3. 68 highlight markers leaked into the output. The handbook wraps a key token in a marker pair and its renderer turns that into a highlight span. The generator stripped the markers line by line, which misses a pair that opens on one line and closes on the next, and it did not touch the markers inside diagram sources or inside the HTML figures at all. Markers are now stripped across the whole block; in diagram labels they reduce to plain text, and in the HTML figures they become the same highlight span the handbook uses, with the same CSS. Markers remaining in docs/core-internals: zero.
|
Self-review before asking anyone to spend more time on this. Three problems, all mine, all 1. The PR referenced a component it did not ship — any build from this branch failed37 figures are laid out with HTML and CSS rather than drawn as SVG, and are handed to an MDX They now ship in 2. One code citation pointed at the wrong directory
I found it by checking every 3. 68 highlight markers leaked into the outputThe handbook wraps a key token in a marker pair, and its renderer turns that into a highlight Markers remaining under Still open, and I would value an opinionThe handbook carries no incubation notice — no mention of incubation, of the ASF, or of |
3 participants
Summary
Adds a developer-oriented handbook for the Cloudberry kernel under
docs/core-internals/— 21 chapters, 104 sections, organised into fiveparts after the structure of Rogov's PostgreSQL 14 Internals.
Coverage: storage and data layout (heap, AO/AOCS, PAX, indexes, the
catalog), transactions and MVCC (isolation, snapshots, vacuum, locks and
the global deadlock detector, distributed transactions and 2PC), query
processing (stages, statistics, the PostgreSQL planner, distributed
planning, ORCA, the executor, Motion and the interconnect), high
availability and recovery, and resource management.
Read the handbook as a PDF (652 pages, 20 MB) so you can
review the content without building the site. All 113 diagrams are rendered
as vector SVG and the text is selectable and searchable. It was produced by
printing the handbook's HTML edition from a headless browser, so layout and
typography match the source. The file is a release asset, deliberately not
part of this diff — a 20 MB binary does not belong in the repository's
history.
What the chapters contain
title="file:line"pointing at the source it was read frompsqlsessionsoutputblock — transcripts from a running cluster, not illustrations§N.Mlinks that resolve to section anchors and are checked at build timeWhy here, and how the site uses it
The directory is written for Docusaurus so that
apache/cloudberry-sitecan pull it in as a git submodule with a sparse checkout limited to
docs/core-internals, served by its own docs plugin instance with anautogenerated sidebar — the same arrangement already used for the PXF
documentation (apache/cloudberry-site#372). Keeping the source next to the
code it describes means a change to the engine and the change to its
description can travel in one commit.
A follow-up PR against
cloudberry-sitewill add the submodule and thethree config touch points. It has to come second, since the submodule
needs a commit on this side to point at.
Nothing here depends on site-level configuration except one small MDX
component: 37 figures are CSS-laid-out HTML rather than SVG, and are handed
to that component as a string so MDX does not try to parse them as JSX. The
component and its CSS are a few lines and are described in the README.
Verification
broken links, no broken anchors, 23 pages, all 64 figures resolved.
so no
pom.xmlexclusions are needed.paths from the machine the sessions were captured on are replaced with
documentation-safe equivalents of identical width, so the column
alignment of captured output is preserved. Verified across the markdown,
the HTML and the PDF.
Testing
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance: None. No code is touched.
User-facing changes: New documentation. Nothing changes for existing
users until
cloudberry-sitelinks it in.Dependencies: None added to this repository. A site that renders the
diagrams needs
@docusaurus/theme-mermaid; without it the mermaid blocksdegrade to code blocks and everything else renders unchanged.
Checklist
Additional Context
On maintenance. The chapters describe a specific point in the tree's
history. Prose is the durable part; line numbers are the perishable part.
The
title="file:line"convention is machine-readable on purpose — a CIjob can verify that every cited path still exists, which catches the worst
class of drift (a file that moved or was deleted) cheaply. I would rather
agree on that check in review than add it unilaterally here.
On the size of the review. This is a large diff and I do not expect
anyone to read 29k lines in a PR. Reading order that costs the least: the
attached PDF for the content,
docs/core-internals/README.mdfor thelayout and conventions, and one chapter of your choice for the texture —
ch02 is the shortest, ch18 the most demanding. Chapter-level corrections
are welcome as ordinary follow-up PRs.
On the branch base. This branch is cut from an April snapshot of
mainrather than today's tip, so GitHub will report it as behind. That is an
artifact of how it was pushed, not of the content: all 93 files are new
paths under
docs/core-internals/, nothing else is touched, and the diffGitHub computes is exactly those 93 additions. Happy to rebase onto current
mainif you would rather review it that way.On AI assistance. Disclosed above per
AI_GUIDELINE.md. The generationwas AI-assisted; the technical claims, the code citations and the captured
sessions are mine to answer for, and I will answer review questions
directly rather than through a tool.