Skip to content

Part a trailing suffix part at a declared suffix delimiter as a comma would (#549) - #600

Merged
derek73 merged 4 commits into
masterfrom
fix/issue-549-delimiter-as-comma
Oct 4, 2026
Merged

derek73 merged 4 commits into
masterfrom
fix/issue-549-delimiter-as-comma

Conversation

@derek73

@derek73 derek73 commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Closes #549.

What changes

A delimiter declared through Policy(extra_suffix_delimiters=...) or HumanName(suffix_delimiter=...) now splits a trailing suffix part exactly where a typed comma would. group() cuts a tail segment at each delimiter before any grouping, groups each piece the way the comma version's own segment would be grouped, and drops the delimiter. No join, maiden walk or link search can reach across a delimiter, because none of them ever sees one.

This reverses #538's "skip" reading, where the connective search stepped over a delimiter as if it weren't written. The skip reading was the source of #549: a delimiter standing alone was dropped like a comma, while one next to a connective was merged into the joined phrase like a word.

Because delimiters never reach grouping, the machinery that stepped over them is deleted:

The result is one call per parse fewer (369 vs 370, tools/perf/call_count.py, py3.11).

Behavior, with suffix_delimiter=" - " (each checked against the released wheels)

input 1.4.0 2.0–2.3 this PR
Smith, John, PhD - and MD suffix PhD, and MD PhD - and MD PhD, and MD
Smith, John, Puig - y Soler Puig, y Soler Puig - y Soler Puig, y Soler
Smith, John, PhD née Puig - i Soler (no maiden markers) maiden Puig (2.3.0) maiden Puig, suffix PhD, i Soler
Smith, John, MD - née Jones Smith suffix MD - née Jones Smith maiden Jones Smith suffix MD, née Jones Smith, no maiden

The last row is an accepted cost (decided 2026-10-03). A maiden marker with the delimiter straight before or after it takes no clause, as it takes none after a typed comma. By count it is the larger move: every one of the 240 maiden movers in the generated set loses the maiden. Recorded as Accepted in rules.md#C1, in bold in the release note, and pinned by the case row suffix_delimiter_core_after_a_marker_takes_no_clause.

At the default policy nothing changes, since no delimiter is declared. Outside a trailing suffix part (before the first comma, or in the given part of Family, Given), the delimiter is still a word, as in 1.4.0.

Measurements (2026-10-03, branch vs 061f02d)

The full figures and the recompute recipe are in the new decisions.md#C1 entry.

  • Generated set (19,972 texts):
    • - delimiter: 2,624 parses move (suffix only 2,384; suffix + maiden 240). No other field moves and no ambiguity report moves.
    • / delimiter: the same 2,624 parses.
    • and delimiter: 2,528 parses.
  • Typed-comma comparison, role fields only. Before → after:
    • Smith, John, and Smith, John, Jr., heads: 752 of 2,100 disagree each → 0.
    • John Smith, head: 1,956 of 2,100, unchanged. That head is out of scope, because there a typed comma changes how the name is split into parts.
  • Differential gate: exit 0 at all five baselines.

Tests and docs

Review

Four passes: design docs, code, a second round on the first fix commit, then /pr-review-toolkit:review-pr (code, tests, comments). None found a correctness bug. Over 517k parses on both trees, every field that moved equals what the typed-comma version gives, and every recorded negative control reproduced.

🤖 Generated with Claude Code

derek73 and others added 3 commits October 3, 2026 17:10
…549)

A delimiter declared through extra_suffix_delimiters / suffix_delimiter
now separates a tail segment exactly as a comma typed in its place:
group cuts the segment at its cores before grouping, groups each part
as the comma twin's segment would be grouped, and drops the cores.
No join, maiden walk or link search can reach across a core, so the
#538 stepping machinery goes: the cores parameter on four functions,
_maiden_take's seen index list, the skip arguments of peel_walk and
trailing_start, and the post-join #206 drop block.

'Smith, John, PhD - and MD' reads suffix 'PhD, and MD' (1.4.0's
answer; 2.0-2.3 gave 'PhD - and MD'), and 'Smith, John, PhD née
Puig - i Soler' reads maiden 'Puig', suffix 'PhD, i Soler' again, as
2.3.0 did -- the boundary reading #538 declined, taken now.

The default policy declares no delimiter; the differential gate exits
0 at all five baselines. One call per parse fewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- group's tail split accumulated a tuple per token, quadratic in a
  part's length; build a list and freeze it once per part (4.7x for
  4x the input, against 8.0x).
- rules.md#M2's statement still described #538's stepping; it now
  defers to C1. P3's separator sentence is scoped to trailing parts,
  and both rules list C1 in interacts:. C1 states the lone-core
  exception and records, as Accepted, that a maiden marker beside a
  declared delimiter takes no clause -- the comma twin's reading, and
  the larger cost by count.
- decisions.md#C1's #549 entry: the maiden-loss class in WHAT MOVES
  and in the weighing; the Jr., head figure (752 of 2,100, not 973 of
  3,310); the ' / ' policy moving the same 2,624; the twin compared on
  role fields, comma-structure counts differing; the #418 repair
  added to SUPERSEDES.
- release_log: the maiden-loss consequence, and the scope stated as
  the part read wholly as suffixes.
- Stale prose in test_post_rules and a renamed test_group test.
- The C1 example enters the rules corpus and the 1.4.0 ledger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- release_log: the scope sentence names the comma forms it covers and
  drops an example whose reading moved between 2.3 and 2.4 for other
  reasons.
- rules.md#C1: the Accepted entry's reason is the marker's position in
  its part (opening it, or ending it), not a name word ahead of it;
  M2 joins C1's interacts:.
- expected_since_1.4.0.toml: drop a stale count of the alternation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 2.4 milestone Oct 4, 2026
@derek73 derek73 added bug docs Documentation fixes and updates labels Oct 4, 2026
@derek73 derek73 self-assigned this Oct 4, 2026
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.98%. Comparing base (061f02d) to head (d180516).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #600   +/-   ##
=======================================
  Coverage   98.98%   98.98%           
=======================================
  Files          45       45           
  Lines        4235     4238    +3     
=======================================
+ Hits         4192     4195    +3     
  Misses         43       43           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- _maiden_take's walk loses the scaffolding the `seen` list left:
  `j < len(pieces)` was implied by `j < trailing`, and `k` aliased `j`.
- Tests: a case row pins that a tail part holding only its core keeps
  it (the guard no test pinned before or after #549; recorded negative
  control 'PhD'); a case row pins 'Smith, John, PhD née - Jones' at
  parse level; the comma-twin invariant compares the reported kinds
  too, less comma-structure, with its negative control unchanged at
  102 of 129.
- Prose: the M2 delimiter examples are no longer called live examples
  of the #538 fix in the five ledgers and the guard test; a misquoted
  C1 excerpt; the lone-core comment states the several-cores case; a
  standing table count; an example fragment that read wrong beside
  the tail form; _marker_run_pieces' docstring after the rewrite; a
  renamed test whose mutant site #549 deleted; reflowed lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit aa401d8 into master Oct 4, 2026
11 checks passed
@derek73
derek73 deleted the fix/issue-549-delimiter-as-comma branch October 4, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug docs Documentation fixes and updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Should a declared suffix delimiter inside a joined connective run be dropped? Smith, John, Puig - i Soler keeps the - in the suffix

1 participant