Skip to content

fix(wikipedia): drop the site suffix from italicised titles - #2458

Open
Machen John (macjayz) wants to merge 1 commit into
microsoft:mainfrom
macjayz:macjayz/fix-wikipedia-markup-titles
Open

fix(wikipedia): drop the site suffix from italicised titles#2458
Machen John (macjayz) wants to merge 1 commit into
microsoft:mainfrom
macjayz:macjayz/fix-wikipedia-markup-titles

Conversation

@macjayz

Copy link
Copy Markdown
Contributor

Summary

WikipediaConverter looks for the article title in a span.mw-page-title-main. Wikipedia only emits that span when the title is plain text — titles rendered with markup (italicised species, film, album and journal names) are written straight into h1#firstHeading instead.

With the span absent, the converter silently fell back to the document <title>, which carries the site suffix. Both the returned title and the # heading then read … - Wikipedia.

Article Before After
Paris Paris Paris (unchanged)
Escherichia coli Escherichia coli - Wikipedia Escherichia coli
Titanic (1997 film) Titanic (1997 film) - Wikipedia Titanic (1997 film)

Cause

Two things had to line up:

  1. Wikipedia omits mw-page-title-main for titles containing markup, so title_elm was None and the <title> fallback won.
  2. Tag.string returns None as soon as an element has more than one child — the exact shape of <i>Titanic</i> (1997 film) — so even when the element is found, a mixed markup/plain-text title yields nothing.

Fix

Fall back to h1#firstHeading when the span is absent, and read get_text() rather than .string so titles that mix markup with plain text survive. Plain titles keep taking the existing path and are unaffected.

Verification

Checked against HTML fetched live from the three articles above.

Added packages/markitdown/tests/test_wikipedia_titles.py with three cases: a fully italicised title, a title mixing markup with plain text, and a plain title as a regression guard. Reverting only the converter change fails the first two and leaves the third passing, confirming the tests pin the bug and the plain-title path is unchanged.

Full suite: 648 passed (the one unrelated failure locally is test_speech_transcription, which needs ffprobe on PATH). black 23.7.0 clean.

Titles rendered with markup have no mw-page-title-main span, so the
converter fell through to the document <title>, suffix and all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant