fix(wikipedia): drop the site suffix from italicised titles - #2458
Open
Machen John (macjayz) wants to merge 1 commit into
Open
fix(wikipedia): drop the site suffix from italicised titles#2458Machen John (macjayz) wants to merge 1 commit into
Machen John (macjayz) wants to merge 1 commit into
Conversation
Titles rendered with markup have no mw-page-title-main span, so the converter fell through to the document <title>, suffix and all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WikipediaConverterlooks for the article title in aspan.mw-page-title-main. Wikipedia only emits that span when the title is plain text — titles rendered with markup (italicised species, film, album and journal names) are written straight intoh1#firstHeadinginstead.With the span absent, the converter silently fell back to the document
<title>, which carries the site suffix. Both the returnedtitleand the#heading then read… - Wikipedia.ParisParis(unchanged)Escherichia coli - WikipediaEscherichia coliTitanic (1997 film) - WikipediaTitanic (1997 film)Cause
Two things had to line up:
mw-page-title-mainfor titles containing markup, sotitle_elmwasNoneand the<title>fallback won.Tag.stringreturnsNoneas soon as an element has more than one child — the exact shape of<i>Titanic</i> (1997 film)— so even when the element is found, a mixed markup/plain-text title yields nothing.Fix
Fall back to
h1#firstHeadingwhen the span is absent, and readget_text()rather than.stringso titles that mix markup with plain text survive. Plain titles keep taking the existing path and are unaffected.Verification
Checked against HTML fetched live from the three articles above.
Added
packages/markitdown/tests/test_wikipedia_titles.pywith three cases: a fully italicised title, a title mixing markup with plain text, and a plain title as a regression guard. Reverting only the converter change fails the first two and leaves the third passing, confirming the tests pin the bug and the plain-title path is unchanged.Full suite: 648 passed (the one unrelated failure locally is
test_speech_transcription, which needsffprobeon PATH).black23.7.0 clean.