feat(content-drive): Title / All Fields search scope + literal-text search terms (#37479, #37532)

[KevinDavilaDotCMS]

Spec-Kit PR 2 for #37479 / #37532. Implementation on top of the approved spec (PR #37518 @ 1ce8cdd1bf).

What this closes

#37532 (High, customer ticket 39185) — Content Drive's search box now matches a term as literal text instead of reading it as Lucene query syntax. The reported headline is findable verbatim, the full reserved character set (\ + - ! ( ) : ^ [ ] " { } ~ * ? | & /) matches literally in both scopes, and consecutive separators no longer emit empty clauses. Verified field filters were already correct; added the test that proves it (ContentDriveFieldFilterTest#textFieldFilter_matchesEveryReservedCharacterLiterally).

#37479 — Content Drive's search box gets a Title / All Fields scope selector, defaulting to All Fields (no regression). Title matches the contentlet title only, via a prefix-seek clause with no catchall and no leading wildcard — the fast path the issue asked for. The scope lives in the URL alongside the other filters, survives reload/Back/Forward/sharing, and counts as filter state only while non-default (so it never falsely lights up "Clear all").

Scope split — verified structurally, not assumed

Both fixes live entirely inside the ES text branch of BrowserAPIImpl, reachable only when useElasticsearchFiltering is set — and ContentDriveHelper is the only place in the codebase that sets it (grepped against main). The other three doors into the same listing — WebAssetHelper, BrowserAjax, DotCMSMacroWebAPI — build their BrowserQuery without it and are unaffected by construction, not by care. New isolation test in BrowserAPITest pins this. The Asset Picker (the other caller of POST /drive/search) gets no control and its 394 existing tests pass unmodified.

⚠️ Two spec amendments need re-approval before merge

The spec was approved with stronger wording than what shipped, for reasons found during implementation — both are struck through in place in spec.md with the date and rationale, not silently edited:

• FR-029 (narrowed 2026-09-14): "every query failure surfaces as an error" was implemented, then reverted — it broke ContentDriveFieldFilterTest#testMalformedDateBoundIsSafe's security guarantee (a Lucene-injection attempt must match nothing and produce no 500; raising query failures turned that into an error response instead, telling an attacker their probe landed). What ships: the front end surfaces failures it can itself observe (network/server errors); BrowserAPIImpl's internal execution failures stay logged-only, as before this feature.
• FR-003 (narrowed 2026-09-15): direct instruction during UI review — the placeholder no longer describes the active scope. It's fixed to the shared search box's default in both Title and All Fields. Not a defect; FR-003 as written had been fully implemented and tested.

@zJaaal @ihoffmann-dot — both of you approved PR #37518 after these amendments landed in the spec, so the sign-off already covers this wording. Flagging here so it's explicit rather than assumed.

SC-003 — not delivered as a completed benchmark

Measured Title vs. All Fields first-page latency on two independently-designed datasets (~3000 docs each), 12 interleaved rounds per dataset. Title was consistently slower, the opposite of what FR-010's rationale predicts. Ruled out one hypothesis with real numbers (the ES boolean-clause chunking math is identical for both clause shapes at this candidate-set size). Could not isolate the remaining candidates — OpenSearch query-cache state, segment-merge state on a just-bulk-indexed index, whether the leading-wildcard cost the feature avoids is simply too cheap to matter at 3000 distinct titles — within this session, on a shared dev instance that had absorbed a full day of builds and test runs.

This does not block this PR: FR-010's actual requirements (no catchall, no leading wildcard in the mandatory gate) are met and unit-tested regardless of what wall-clock timing shows on this instance. The performance claim is left honestly unproven rather than asserted. Full write-up with both datasets' numbers and the ruled-out hypothesis is in tasks.md's Phase 7 note (gitignored, so linking the finding here): a dedicated, unloaded OpenSearch instance at materially larger scale is needed before SC-003 can be claimed satisfied.

Convergence

/speckit-converge ran twice. First pass found 3 findings — two stale acceptance-scenario texts left over from the FR-003/FR-029 amendments, one under-tested requirement (field-filter reserved-character coverage existed as one fixed string, not the exhaustive sweep SC-012 promises). All three closed; second pass reports converged with zero findings.

Testing

• Backend unit: 13 assertions (GlobalSearchAttributeStrategyTest + baseline), 0.015s, no DB.
• Backend integration: 88 assertions across ContentDriveSearchScopeTest, ContentDriveLiteralTextSearchTest, ContentDriveKeywordSearchTest, ContentDriveFieldFilterTest, BrowserAPITest — all registered in MainSuite3a.
• Postman: 100/101 assertions on the full ContentDriveResource collection (the one failure is a pre-existing missing test-image fixture, unrelated to this change).
• Frontend: 1398 tests / 30 files (portlet) + 394 tests / 13 files (ui, Asset Picker included, unmodified).
• SC-002 carve-out enumerated: one pre-existing test outside the four Content Drive classes changed expectations (BrowserAPITest#test_buildBaseESQuery_withDifferentFilterCombinations, Test Case 5 — an unescaped & was pinning the exact defect Content Search: escape Lucene query-syntax characters in content type field filters #37532 reports).

No openapi.yaml regeneration — /v1/drive/search is @Hidden and absent from the committed spec.

Refs #37479, #37532

🤖 Generated with Claude Code

[KevinDavilaDotCMS]

⚠️ SC-003 (Title scope performance) — measured slower than All Fields, not faster. Flagging for backend input.

Tagging this separately from the PR description because it needs eyes from whoever owns query performance here, not just a merge-blocking checkbox.

What SC-003 asks

The spec's success criterion: a Title search should return its first page faster than the same term in All Fields scope, on a dataset large enough for the difference to exceed noise. It's framed as a recorded measurement, not an automated test — the deliverable is the comparison itself.

What I measured

Two independently-designed datasets, seeded on a local dev dotCMS + single-node OpenSearch, 12 interleaved rounds each (24 comparisons total), first-page timing via curl -w %{time_total} against POST /v1/drive/search:

1. First dataset (3000 docs, term in every body, in no title): All Fields 22–50ms, Title 52–98ms.
2. Second dataset, redesigned to match the actual use case — one document whose title carries the term, 3000 others where it's only a casual body mention (All Fields has hasMoreContent: true, the target isn't even on page 1; Title finds exactly the one document): All Fields 22–47ms, Title 55–77ms.

Same direction, same order of magnitude, in both designs. Title is consistently slower — the opposite of what FR-010's rationale predicts (no catchall aggregate, no leading wildcard in the mandatory gate — both prefix seeks, which should be cheaper).

What I ruled out with numbers, not guesswork

Suspected BrowserAPIImpl#calculateMaxInodesPerESQuery's clause-count estimate might chunk the two clause shapes differently (All Fields's query string is longer — more OR/:). Computed it directly for both actual clause strings at 3001 candidates: identical — 4 batches of 876 for both. Chunking is not the cause.

What I could not verify in this session

• OpenSearch query-cache state (repeated identical queries per scope, but caches are typically per-index and this was a fresh index each time)
• Segment-merge state on a just-bulk-indexed corpus
• Whether the leading-wildcard cost the Title clause exists specifically to avoid (title_dotraw:*term*, a full term-dictionary scan) is simply too cheap to matter at ~3000 distinct titles — the scale might need to be an order of magnitude+ larger for the win to show
• This dev instance ran a full day of builds, integration suites and Postman runs during this same session — a shared, contended environment is a real confound I didn't rule out

Where this leaves things

Not blocking this PR. FR-010's actual, testable requirements (no catchall, no leading wildcard in the mandatory gate) are met and unit-tested regardless of what this wall-clock measurement shows. The behavioral correctness of Title scope doesn't depend on this outcome — only the performance claim does, and I'm leaving it honestly unproven rather than asserting a win I can't back up.

What I'd want from someone who knows this index's real-world behavior:

• Does this match a known characteristic of the ES/OpenSearch query planner at this candidate-set scale, or does it point at something worth profiling (_profile API, hot-thread capture)?
• Is there a dedicated, unloaded OpenSearch instance where this is worth re-measuring at 10x–100x the scale before treating SC-003 as settled either way?

Full write-up (exact clause strings, the chunking math, both raw timing tables) is in specs/37479-content-drive-search-scope/tasks.md's Phase 7 note — that file is gitignored per this repo's Spec-Kit convention (process artifact, not committed), so it's not in the diff here. Happy to paste it inline if useful, or pair with whoever picks this up.

[github-actions]

🐳 PR Docker test image

Latest build for commit 07f0da9 pushed to dotcms/dotcms-test:

docker pull dotcms/dotcms-test:pr-37554-issue-37479-content-drive-search-scope-impl
docker pull dotcms/dotcms-test:pr-37554-issue-37479-content-drive-search-scope-impl_07f0da9

[nollymar]

do we have plans to support this kind of search for other searchable fields?

[KevinDavilaDotCMS]

Good question — out of scope for this PR. Title scope is deliberately limited to the contentlet title field only (spec.md Decision 2: "Contentlet title only — not fileName, not metadata.name", to keep the query to a single field so the performance argument holds). Scoping to other searchable fields would need its own spec — nothing planned yet, but it's a reasonable follow-up once this lands.

[zJaaal]

This field is never used — the addon's background isn't actually being repointed.

The template renders <p-inputgroup-addon> with no [style] binding, so ADDON_STYLE is read by nothing. It looks like it survived the styling rework that landed later in the branch (the trigger ended up using TRIGGER_PT for its border instead).

Worth resolving one way or the other, because the 15-line comment above it reads as a description of shipped behaviour. A reviewer coming to this file in six months will believe the addon background is pinned to --p-inputtext-background when nothing does that.

Either bind it on the addon, or delete the field and its comment.

---

🤖 Review comment written by Claude Code, posted from @zJaaal's account.

[KevinDavilaDotCMS]

Fixed in 7f39e67 — removed the dead ADDON_STYLE field and its stale 15-line comment.

[zJaaal]

Also check claude comments, remember to add some e2e for this feature

[KevinDavilaDotCMS]

🔬 Performance follow-up on SC-003 — from a suggestion by @nollymar

@nollymar pointed out that title is a text field while title_dotraw is its keyword sibling, and asked whether using title instead of title_dotraw could help Title scope's timing — worth writing up separately since it settles part of the SC-003 finding from the PR description (Title scope measuring slower than All Fields, the opposite of what FR-010's rationale predicts).

The idea, and where it actually applies

title_dotraw stores the whole raw title as one keyword term, so its cardinality is close to one distinct term per document. title is tokenized into words, so its term dictionary is far smaller (titles share vocabulary). A prefix query against a smaller term dictionary is cheaper — that part of the suggestion is correct.

But it's not "replace title_dotraw with title everywhere" — buildTitleScopedQuery's per-word clause is +(title:word* title_dotraw:word*), and title_dotraw:word* can only ever match when word is a prefix of the entire raw title. That's true, at best, for the first word of a multi-word search term — never for the ones after it. So dropping title_dotraw outright would risk losing a real match on the first word; the actual fix is to drop it from every word except the first, where it was structurally guaranteed to never contribute a match anyway.

Verified against real OpenSearch, not just query-shape unit tests

The local dev index only has 681 docs — too small to show a timing difference at this scale (same wall the original SC-003 investigation hit). So I stood up a temporary index on the same local OpenSearch instance, with the exact os-content-mapping.json / os-content-settings.json analyzer config, and a synthetic 150k-doc dataset:

• 140 interleaved paired comparisons (7 multi-word terms × 20 rounds, alternating query order each round to cancel drift): median took 2ms → 1ms, mean 2.11ms → 1.18ms (~44% reduction). Zero result-count mismatches across all 140 pairs — identical result sets, just cheaper to compute.
• _profile breakdown on one representative 5-word query: the four title_dotraw:<word>* clauses removed each cost 0.26–0.41ms; total profiled query time went 5.36ms → 1.44ms (~73% less) for that term.

Applied

07f0da97f8 — perf(content-drive): drop title_dotraw from non-leading Title-scope words. BrowserAPIImplTest updated to pin the new clause shape, plus a dedicated test (buildTitleScopedQuery_multiWordTerm_onlyFirstWordCarriesTitleDotraw) documenting why.

Doesn't fully settle SC-003 on its own — the original measurement compared Title vs. All Fields end-to-end, and this only cuts the cost of one piece of Title's query, not the whole picture (OpenSearch query-cache state, segment-merge state, etc. are still unmeasured, as the original write-up notes). But it's a real, measured win, correctness-neutral, and it came directly from @nollymar's suggestion — thanks for the pointer.