From 437ebf67ac0c2b26d32d0118b003f2b215ebdf97 Mon Sep 17 00:00:00 2001 From: Mark Riechers <6778715+mriechers@users.noreply.github.com> Date: Fri, 4 Sep 2026 13:40:15 -0500 Subject: [PATCH 1/5] fix(formatter): restructure Tidy to match prettyhtml.com's real pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HTML copied out of a Google Doc came through the Tidy tool mangled. Root cause: Docs autocorrects straight quotes to curly ones even when the text is HTML source, so `class="hero lede"` arrives curly-quoted. parseAttributes read the value as unquoted, stopped at the space, and invented a bare attribute from the tail that no cleaner matched — so it survived into the output. Fixing that meant auditing the whole pipeline against the live site. Read String(convertText) in the page and replayed each option's replacement sequence against window.text; the dispatch turned out to differ from ours in order and in several option boundaries. Pipeline (new exported runTidyPipeline, so it is testable): - adds the unconditional whitespace pre-pass, looped post-pass and final cleanup that were missing entirely - runs to-plain-text first, as theirs does - option 5 is nbsp-only; literal whitespace collapse moves to the passes - option 3 reclaims the "> <" join, "> \n" join and newline-only-tag removal that had been sitting behind option 4 - toPlainText substitutes a space per tag instead of deleting it - replaceUntilStable mirrors their helyettesit replace-to-idempotence, and refuses to loop on a replacement that re-creates its own pattern Their engine is two layers — a TinyMCE DOM round-trip, then the string cleaners. We have no round-trip, so what it did for them now rides default-ON Extras: block elements on their own lines (the largest visible gap), nested empty-tag removal, Google Docs residue, script/style strip. Trailing
before a block close is dropped too. Curly quotes are now accepted as attribute delimiters at the tokenizer level, unconditionally — it is robustness, not an option. Prose typography is left alone; a default-OFF Extra straightens it for anyone who wants ASCII. Closes #22: role="presentation", aria-level and the docs-internal-guid wrapper are stripped (layer-1 compensation), as is dir="ltr" (which survives on their site — deliberately better than parity). Verification: 128 tests pass. parity.test.mjs replays 21 black-box fixtures captured from the live site; the three where we deliberately differ carry an `ours` field recording what and why. Raw third-party capture stays gitignored, and eslint now ignores it too. Co-Authored-By: Claude Opus 5 --- .gitignore | 4 + CLAUDE.md | 8 +- README.md | 3 +- eslint.config.js | 13 +- formatter/app.mjs | 449 +++++++++++++++--- formatter/index.html | 20 +- formatter/tests/curly-quotes.test.mjs | 115 +++++ formatter/tests/docs-residue.test.mjs | 68 +++ .../tests/fixtures/prettyhtml-golden.json | 194 ++++++++ formatter/tests/parity.test.mjs | 113 +++++ formatter/tests/plain-text.test.mjs | 43 +- formatter/tests/stray-breaks.test.mjs | 14 + planning/2026-05-27-prettyhtml-parity.md | 4 +- .../2026-09-03-prettyhtml-complete-capture.md | 285 +++++++++++ 14 files changed, 1261 insertions(+), 72 deletions(-) create mode 100644 formatter/tests/curly-quotes.test.mjs create mode 100644 formatter/tests/docs-residue.test.mjs create mode 100644 formatter/tests/fixtures/prettyhtml-golden.json create mode 100644 formatter/tests/parity.test.mjs create mode 100644 planning/2026-09-03-prettyhtml-complete-capture.md diff --git a/.gitignore b/.gitignore index defcf74..6a0b1d5 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,7 @@ package-lock.json # Non-Claude agent worktrees (created per CLAUDE.md isolation rules) .worktrees/ + +# Raw prettyhtml.com capture — third-party copyrighted JS/CSS/HTML, local reference only. +# Clean-room behavior specs live in planning/*.md; the raw files never get committed. +planning/captures/ diff --git a/CLAUDE.md b/CLAUDE.md index 72d227a..585bbd1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,7 +63,13 @@ Each tool lives in its own directory with a consistent pattern: ### Current Tools -- **`/formatter/`** — HTML Formatter & Tidy. Pipeline: tokenize → indent (two-stage) → tidy → compress. Tidy dropdown is organized into three groups: **Formatting** (lowercase/sort/quote — our extras), **Cleaning (prettyhtml.com)** (the 10 options 1-for-1 with prettyhtml.com — Inline styles, Classes & IDs, Empty tags, Tags with 1 space, Successive spaces, Comments, Tag attributes, To plain text, AI Watermarks, Smart  s; first 6 ON by default), and **Extras** (data-attrs, span-unwrap, strip-stray-line-breaks, etc.). Literal ports of `removeTagAttributes`, `aiWatermarkFixer`, and `smartNbsps` live in `formatter/app.mjs` (ES module). `normalizeStrayBreaks` (same module) runs as a pre-pass before `tidy()` to strip `
` residue between/around block elements and inside empty blocks (default-ON Extras option `opt-stray-breaks`), replicating prettyhtml.com's TinyMCE normalization layer without a DOM round-trip; tests in `formatter/tests/stray-breaks.test.mjs`. Tests in `formatter/tests/*.test.mjs` run via `npm test` (uses `linkedom` as DOMParser shim in Node). Options in `localStorage` key `htmlTidy_options`. See `planning/2026-05-27-prettyhtml-parity.md` for the algorithm snapshot / insurance documentation, and `planning/2026-06-24-stray-line-break-normalization-{design,plan}.md` for the stray-break feature. +- **`/formatter/`** — HTML Formatter & Tidy. All logic lives in `formatter/app.mjs` (an ES module; there is no `app.js` here). Buttons: Indent (two-stage), Tidy, Compress. + - **Tidy runs through `runTidyPipeline(html, opts)`**, ordered to match prettyhtml.com's `convertText()`: stray-break normalization → whitespace pre-pass → script/style strip → to-plain-text (**first**, as theirs is) → nbsp collapse (option 5) → inter-tag gap joins → `tidy()` → nested-empty fixpoint → block-newline separation → looped whitespace post-pass → tag-attributes → AI-watermarks → smart-punctuation straightening → smart-nbsps → final cleanup. `replaceUntilStable()` mirrors their `helyettesit()` replace-to-idempotence semantics. + - **prettyhtml.com is two layers**: a TinyMCE DOM round-trip, then the string cleaners behind the ten checkboxes. We have no round-trip, so several default-ON **Extras** stand in for layer 1 — `opt-stray-breaks`, `opt-block-newlines`, `opt-nested-empties`, `opt-docs-residue`. Reasoning about their cleaners in isolation gives the wrong answer about what their site outputs; always check end-to-end. + - Dropdown groups: **Formatting** (lowercase/sort/quote), **Cleaning (prettyhtml.com)** (the 10 options 1-for-1; first 6 ON by default), **Extras** (block newlines, nested empties, Google Docs residue, script/style strip, straighten smart punctuation, stray line breaks, data-attrs, span-unwrap). + - **Deliberate divergences** (documented in `app.mjs` above the pipeline): E — options 1/2 parse attributes structurally rather than doing double-quote-only string surgery; G — empty-tag removal exempts `td/th/script/style/media` and requires matching tag names; H — one-space-tag removal accepts ` `; N — curly quotes are accepted as attribute delimiters, which is what makes HTML pasted out of Google Docs survive. + - Tests in `formatter/tests/*.test.mjs` via `npm test` (`linkedom` as DOMParser shim). `parity.test.mjs` runs the pipeline against `tests/fixtures/prettyhtml-golden.json` — black-box input/output pairs captured from the live site; a fixture with an `ours` field is a recorded deliberate divergence. Options persist in `localStorage` key `htmlTidy_options`. + - Docs: `planning/2026-09-03-prettyhtml-complete-capture.md` is the current clean-room spec (supersedes most of `planning/2026-05-27-prettyhtml-parity.md`); `planning/2026-06-24-stray-line-break-normalization-{design,plan}.md` covers stray breaks. The raw third-party capture lives in gitignored `planning/captures/` and must never be committed. - **`/og-image/`** — OG Image Preview. Platform specs in `platforms.json`. Optional Cloudflare Worker CORS proxy in `functions/fetch-meta.js`. Fallback proxies for CORS. ### Suggested Tools System diff --git a/README.md b/README.md index 89dd27c..0d0c110 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,8 @@ Beautify, format, and clean up messy or minified HTML instantly. └── formatter/ # HTML Formatter & Tidy ├── index.html ├── styles.css - └── app.js + ├── app.mjs # ES module — pipeline, cleaners, DOM wiring + └── tests/ # node --test, linkedom DOMParser shim ``` Each tool is self-contained in its own directory with its own `index.html`, making it easy to develop, test, and deploy independently. diff --git a/eslint.config.js b/eslint.config.js index 859875d..e09b808 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -41,6 +41,15 @@ module.exports = [ }, }, }, + { + files: ['formatter/tests/**/*.mjs'], + languageOptions: { + sourceType: 'module', + globals: { + ...globals.node, + }, + }, + }, { files: ['og-image/functions/**/*.js'], languageOptions: { @@ -51,6 +60,8 @@ module.exports = [ }, }, { - ignores: ['node_modules/'], + // planning/captures/ holds the raw third-party prettyhtml.com capture. It is + // gitignored reference material, not our source — never lint or ship it. + ignores: ['node_modules/', 'planning/captures/'], }, ]; diff --git a/formatter/app.mjs b/formatter/app.mjs index a930794..b9a914a 100644 --- a/formatter/app.mjs +++ b/formatter/app.mjs @@ -67,25 +67,55 @@ const BLOCK_ELEMENTS = new Set([ 'td', 'th', 'dl', 'dt', 'dd', 'form', 'fieldset', 'address', 'pre', ]); +/** + * Curly quotes accepted as attribute delimiters. + * + * Word processors (Google Docs especially) autocorrect straight quotes to curly + * ones even when the text is HTML *source*, so pasted markup routinely arrives as + * `class=“hero lede”`. Treating those as delimiters is a robustness fix, not a + * parity concern: prettyhtml.com's TinyMCE mis-parses the same input (it reads the + * value as unquoted and truncates at the first space) and only looks clean because + * its classes/IDs option then deletes the wreckage. Ours has no such backstop, so + * without this the invented attribute survives into the output. + * + * Each opener maps to the set of characters that may close it — a pair is accepted + * in either orientation, since autocorrect sometimes emits two openers or two + * closers when it guesses the word boundary wrong. + */ +const QUOTE_CLOSERS = { + '"': '"', + "'": "'", + '\u201C': '\u201D\u201C', + '\u201D': '\u201D\u201C', + '\u2018': '\u2019\u2018', + '\u2019': '\u2019\u2018', +}; + /** * Parse an HTML attribute string into an array of {name, value, quote} objects. */ -function parseAttributes(attrString) { +export function parseAttributes(attrString) { const attrs = []; if (!attrString || !attrString.trim()) return attrs; - // Regex matches: name="value", name='value', name=value, or bare name - const re = /([^\s=]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|(\S+)))?/g; + // Regex matches: name="value", name='value', name=“value”, name=‘value’, + // name=value, or bare name. Curly-delimited values are normalized to straight + // quotes here so every downstream rebuild path emits valid HTML. + const re = /([^\s=]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|[\u201C\u201D]([^\u201C\u201D]*)[\u201C\u201D]|[\u2018\u2019]([^\u2018\u2019]*)[\u2018\u2019]|(\S+)))?/g; let match; while ((match = re.exec(attrString)) !== null) { const name = match[1]; const value = match[2] !== undefined ? match[2] : match[3] !== undefined ? match[3] : match[4] !== undefined ? match[4] + : match[5] !== undefined ? match[5] + : match[6] !== undefined ? match[6] : null; const quote = match[2] !== undefined ? '"' : match[3] !== undefined ? "'" - : match[4] !== undefined ? '' + : match[4] !== undefined ? '"' + : match[5] !== undefined ? "'" + : match[6] !== undefined ? '' : null; attrs.push({ name, value, quote }); } @@ -96,7 +126,7 @@ function parseAttributes(attrString) { * Tokenize an HTML string into an array of tokens. * Each token has: { type, raw, tagName?, attributes?, content? } */ -function tokenize(html) { +export function tokenize(html) { const tokens = []; let pos = 0; const len = html.length; @@ -224,9 +254,14 @@ function findTagEnd(html, pos) { const len = html.length; while (i < len) { const ch = html[i]; - if (ch === '"' || ch === "'") { - // Skip quoted attribute value - const closeQuote = html.indexOf(ch, i + 1); + const closers = QUOTE_CLOSERS[ch]; + if (closers) { + // Skip a quoted attribute value so a '>' inside it doesn't truncate the tag. + // Curly delimiters close on either member of their pair (see QUOTE_CLOSERS). + let closeQuote = -1; + for (let j = i + 1; j < len; j++) { + if (closers.includes(html[j])) { closeQuote = j; break; } + } if (closeQuote === -1) return -1; i = closeQuote + 1; } else if (ch === '>') { @@ -284,7 +319,7 @@ function rebuildTag(token) { * @param {number} stage - 1 = block elements only, 2 = all elements * @returns {string} Indented HTML */ -function indent(html, opts, stage) { +export function indent(html, opts, stage) { const tokens = tokenize(html); const lines = []; let indentLevel = 0; @@ -421,6 +456,14 @@ function buildTidyTag(tagName, attrs, selfClose, opts) { if (opts.removeStyles && (lowerAttrName === 'style' || lowerAttrName === 'valign' || lowerAttrName === 'align')) return null; if (opts.removeClassesIds && (lowerAttrName === 'class' || lowerAttrName === 'id')) return null; if (opts.removeDataAttrs && lowerAttrName.startsWith('data-')) return null; + // Google Docs residue. TinyMCE drops role/aria-level for prettyhtml.com at + // layer 1; we have no layer 1, so the option covers them explicitly. dir="ltr" + // survives on their site — stripping it is deliberately better than parity. + if (opts.docsResidue) { + if (lowerAttrName === 'role' && value === 'presentation') return null; + if (lowerAttrName === 'aria-level') return null; + if (lowerAttrName === 'dir' && (value || '').toLowerCase() === 'ltr') return null; + } if (opts.removeEmptyAttrs && value === '' && !isBooleanAttr(attrName)) return null; if (opts.quoteAttrs && value !== null && quote !== '"' && quote !== "'") { @@ -453,12 +496,16 @@ function buildTidyTag(tagName, attrs, selfClose, opts) { * Tidy tokenized HTML — applies cleaning without changing whitespace. * Returns { output, fixCount, tagCount }. */ -function tidy(html, opts) { +export function tidy(html, opts) { const tokens = tokenize(html); const parts = []; let fixCount = 0; let tagCount = 0; let unwrappedSpanDepth = 0; + // One entry per open : true if it was unwrapped as a Google Docs container. + // A plain stack rather than a depth counter, so a real nested keeps its + // own closing tag instead of consuming the wrapper's. + const boldStack = []; for (let i = 0; i < tokens.length; i++) { const token = tokens[i]; @@ -495,7 +542,7 @@ function tidy(html, opts) { const canRemove = !VOID_ELEMENTS.has(lowerName) && !['script', 'style', 'iframe', 'canvas', 'video', 'audio', 'td', 'th'].includes(lowerName); - // Case 1: — directly empty + // Case 1: — directly empty. prettyhtml option 3 (uresTagotTorul). if (canRemove && opts.removeEmptyTags && nextToken && nextToken.type === TokenType.CLOSE_TAG && nextToken.tagName.toLowerCase() === lowerName) { @@ -504,12 +551,15 @@ function tidy(html, opts) { break; } - // Case 2:   — contains only   / whitespace - if (canRemove && opts.removeOneSpaceTags && nextToken && nextToken.type === TokenType.TEXT) { - const stripped = nextToken.content.replace(/ /g, '').trim(); + // Case 2: \n — a single newline. Also option 3, via + // csakEnteresTagotTorul; it used to be lumped in with the one-space + // option below, which put it behind the wrong checkbox (divergence F). + // Whitespace runs have already been collapsed by the pre-pass, and a + // space-only tag reaches Case 1 through option 3's "> <" join. + if (canRemove && opts.removeEmptyTags && nextToken && + nextToken.type === TokenType.TEXT && nextToken.content === '\n') { const closeToken = tokens[i + 2]; - if (!stripped && - closeToken && closeToken.type === TokenType.CLOSE_TAG && + if (closeToken && closeToken.type === TokenType.CLOSE_TAG && closeToken.tagName.toLowerCase() === lowerName) { fixCount++; i += 2; @@ -517,6 +567,33 @@ function tidy(html, opts) { } } + // Case 3:   — option 4 (csakEgyNbspTagotTorul). Theirs + // matches the named entity only; we accept the numeric spelling too + // (divergence H, a deliberate improvement). + if (canRemove && opts.removeOneSpaceTags && nextToken && nextToken.type === TokenType.TEXT && + NBSP_SPELLINGS.includes(nextToken.content)) { + const closeToken = tokens[i + 2]; + if (closeToken && closeToken.type === TokenType.CLOSE_TAG && + closeToken.tagName.toLowerCase() === lowerName) { + fixCount++; + i += 2; + break; + } + } + + // Google Docs wraps a whole paste in with + // font-weight:normal. It is a transparent container, not bold — unwrap it. + if (lowerName === 'b') { + const idAttr = (token.attributes || []).find(a => a.name.toLowerCase() === 'id'); + const isDocsWrapper = Boolean(opts.docsResidue && idAttr && idAttr.value && + idAttr.value.startsWith('docs-internal-guid')); + boldStack.push(isDocsWrapper); + if (isDocsWrapper) { + fixCount++; + break; + } + } + // Unwrap empty spans if (opts.unwrapSpans && lowerName === 'span') { const remainingAttrs = (token.attributes || []).filter(attr => { @@ -552,6 +629,10 @@ function tidy(html, opts) { break; } + if (lowerName === 'b' && boldStack.length > 0 && boldStack.pop()) { + break; + } + parts.push(``); break; } @@ -563,22 +644,9 @@ function tidy(html, opts) { } case TokenType.TEXT: { - if (token.preserveWhitespace) { - parts.push(token.content); - break; - } - - let text = token.content; - if (opts.trimWhitespace) { - text = text.replace(/\s+/g, ' '); - // Only fully trim if the result is all whitespace - if (!text.trim()) { - // Preserve a single space between inline elements - parts.push(' '); - break; - } - } - parts.push(text); + // Pass through untouched. Whitespace normalization belongs to the + // pipeline's pre/post-passes, not to any option (divergence C). + parts.push(token.content); break; } } @@ -594,7 +662,7 @@ function tidy(html, opts) { // COMPRESS — Strip all unnecessary whitespace (formerly minify) // ============================================================ -function compress(html) { +export function compress(html) { const tokens = tokenize(html); const parts = []; @@ -719,13 +787,18 @@ export function removeAllTagAttributes(text) { export function toPlainText(text) { const SENTINEL = '\x00COMMENT\x00'; // NUL-bracketed sentinel — won't collide with real content const comments = []; - // Save comments, replace each with the sentinel + // Save comments, replace each with the sentinel. Theirs sentinels only the + // "/g, m => { comments.push(m); return SENTINEL; }); - // Strip all remaining tags - t = t.replace(/<[^>]*>/g, ''); + // Replace each remaining tag with a single space. Theirs collapses every tag + // to "<>" and then substitutes " ", so "a
b" becomes "a b", not "ab" + // (divergence D). The doubled spaces this leaves are collapsed by the + // pipeline's post-pass. + t = t.replace(/<[^>]*>/g, ' '); // Restore comments in order t = t.replace(new RegExp(SENTINEL, 'g'), () => comments.shift()); return t; @@ -939,6 +1012,18 @@ export function normalizeStrayBreaks(html) { continue; } if (!hasDirectText[pIdx]) { i++; continue; } // empty block -> drop the
+ + // A
with nothing but whitespace between it and its block's closing + // tag is filler, not a line break — it renders nothing. TinyMCE drops + // these for prettyhtml.com at layer 1; without that layer we do it here. + let k = i + 1; + while (k < n && tokens[k].type === TokenType.TEXT && + parentBlock[k] === pIdx && tokens[k].content.trim() === '') k++; + if (k < n && isBlockClose(tokens[k]) && + tokens[k].tagName.toLowerCase() === tokens[pIdx].tagName.toLowerCase()) { + i++; + continue; + } // else: real line break inside text — keep it } out.push(t.raw); @@ -947,6 +1032,265 @@ export function normalizeStrayBreaks(html) { return out.join(''); } +// ============================================================ +// TIDY PIPELINE — the order and scaffolding of prettyhtml.com convertText() +// +// Their engine is two layers. Layer 1 is a TinyMCE round-trip that normalizes +// the DOM before any cleaner runs; layer 2 is the string cleaners behind the +// ten checkboxes. We have no layer 1, so a few things TinyMCE does for them are +// handled here by default-ON Extras options instead (normalizeStrayBreaks, +// opt-nested-empties, opt-docs-residue). Reasoning about their cleaners in +// isolation gives the wrong answer about what the site actually outputs — see +// planning/2026-09-03-prettyhtml-complete-capture.md. +// +// Deliberate divergences, kept because ours is better: +// E — options 1/2 parse attributes structurally; theirs does double-quote-only +// string surgery, so style='x' or class=“x” slip past it. +// G — canRemove exempts td/th/script/style/media; theirs deletes and +// does not even check that tag names match ( gets removed). +// H — one-space-tag removal accepts   as well as  . +// N — curly quotes are accepted as attribute delimiters (see QUOTE_CLOSERS). +// +// Known cost of parity: the whitespace passes are string-level and run outside +// the tokenizer, so like theirs they do not preserve 
/