Fix notes with duplicate names in the same subfolder silently destroying each other's content - #3726
Conversation
…ing each other's content Note::handleNoteTextFileName()'s duplicate-name check calls Note::fetchByFileName(fileName) with no subfolder id, which defaults to NoteSubFolder::activeNoteSubFolderId() (whatever subfolder is selected in the UI) instead of the note's own subfolder. Two notes with an identical name in the same non-active subfolder aren't detected as colliding, and Note::canWriteToNoteFile()'s writability probe then truncates the colliding file as a side effect of merely checking it's writable -- before the second note's own write ever happens. Also fixes the same wrong-scope pattern in Note::renameNoteFile(), a suffix-escalation bug found while fixing the first issue, and (in JoplinImportDialog) duplicate resource copies when the same Joplin attachment/image is referenced more than once during import. Found while investigating a minor cosmetic follow-up noted during pbek#3672's validation (stale-looking Trash entries and orphaned duplicate attachment files) -- the "stale Trash entries" turned out to be the visible aftermath of this bug, which isn't Joplin-import-specific at all.
|
Since you want to change core functionality for an importer, I will need to find time first to test that properly. |
For sure, this is definitely more invasive and deserves thorough testing. Also worth noting that I've only tested this on Linux. When I circled back to this today, I was actually leaning towards not submitting a PR at all for the remaining import cleanup I had flagged for later (one time action, with no assumed impact to my data). When I started poking around at it more I realized my data impact assumption was incorrect. When i was testing the fix for #3672, i was doing that without the "Show subfolders" option enabled. So all of my Joplin data imported flat and my note counts between Joplin and QOwnNotes lined up perfectly. Today when I imported with "Show subfolders" enabled like I really would to preserve my folder hierarchy from Joplin, I discovered I actually lost 33 notes. If I import flat, i can see the collision detection working its magic and writing stuff to stderr like... Info: "Renamed note-file was removed: /home/foo/...Promox configuration overview - Imgur 1.md" ((null):0, (null)) When I import preserving the hierarchy, nothing gets logged to stderr about collisions. The notes are just silently gone. As always the most important thing is the reproducer data set. I don't care if the code in this PR gets merged, I just want to provide good issue reports. |
Description
Follow-up to #3672 and #3715. While validating #3672 against a real
~7,000-resource Joplin export, I'd noted two small, seemingly cosmetic
follow-ups worth a look someday: a handful of orphaned duplicate attachment
files, and some stale-looking Trash entries tied to duplicate note titles.
Going back to actually fix those turned into something more useful than
expected — the "stale Trash entries" turned out to be the visible aftermath
of a real content-loss bug that isn't specific to Joplin import at all. I
had Claude Code dig into it; it traced the whole thing down to two small,
unrelated-looking issues in
Notethat combine destructively, wroteregression tests for it, and put this PR together.
What this changes
The core bug:
Note::handleNoteTextFileName()'s duplicate-name check looks at the wrong subfolderIts collision-avoidance loop calls
Note::fetchByFileName(fileName)with nosubfolder argument. That defaults to
NoteSubFolder::activeNoteSubFolderId()— whatever subfolder happens to be selected in the UI — rather than the
subfolder the note actually being saved/renamed into. During a Joplin import
into a specific target notebook (or really, any time the active subfolder
differs from the note's own), two notes with an identical name in the same
real subfolder aren't detected as colliding, because the check is looking
in the wrong place.
Fix: pass the note's own
_noteSubFolderIdexplicitly, matching thepattern already used elsewhere in the same file (e.g.
fetchByRelativeFilePath()).Compounding it:
Note::canWriteToNoteFile()'s writability probe destroys the file it's checkingOnce the wrong-scope check lets a collision through,
handleNoteTextFileName()calls
canWriteToNoteFile()on the (colliding) target path to confirm it'swritable. That function opens the file with
QIODevice::WriteOnly, whichfor
QFileimpliesTruncateunless combined withAppend— so merelychecking whether an existing file can be written to empties it right there,
before the second note's own write ever happens. The existing
"clean up if it didn't exist before" logic never caught this either, since
it checked
file.exists()after theopen()call had already created thefile.
Fix: capture existence before opening (fixing the ordering above, so the
cleanup finally runs), and open
ReadWriteinstead ofWriteOnly. When thetarget doesn't exist yet, behavior is unchanged: the probe creates a
temporary file to verify writability and then removes it, leaving nothing
behind. When the target does already exist, opening
ReadWriteno longerwipes its content just to check.
Same wrong-scope pattern in
Note::renameNoteFile()The dedicated "rename note" action (and the scripting API's rename) has the
identical unscoped
Note::fetchByName(newName)call. It doesn't cause dataloss the same way — I checked directly, and Qt's
QFile::rename()safelyrefuses when the destination already exists — but the wrong-scope check lets
_fileName/_name/store()run before that refusal, leaving the note's DBrow briefly claiming a filename that isn't actually what's on disk. Fixed
the same way, scoped to the note's own subfolder.
One more found while fixing the first: suffix escalation on repeated calls
handleNoteTextFileName()gets called again from everystoreNoteTextFileToDisk()(up to a few times per note during import, onceper content/image/attachment pass). Since a note's title line is
deliberately never rewritten with its resolved suffix, "name" recomputed
from the text keeps not matching the already-suffixed
_nameon every latercall, re-entering the collision loop — and once correctly scoped, that loop
would find the note's own just-chosen name as a "collision" and escalate
past it (
Title 1→Title 2→ ...) every time. Fixed by excluding thenote's own id from the match.
Also fixed while in the neighborhood: duplicate resource copies during Joplin import
The other half of the original cosmetic report: a Joplin resource
referenced more than once (the same attachment linked twice in a note, or
shared across notes) was copied into the attachments/media folder once per
reference, leaving byte-identical
<id>-1.extduplicates. For images,Note::getInsertMediaMarkdown()already has a same-file-reuse mechanismgated behind a "use existing file?" dialog — the import now sets that
override for its duration (restored afterward) so it reuses automatically
instead of prompting once per repeat. For attachments (no such mechanism
exists), the import dialog now caches the destination filename per resource
id and reuses it directly.
Why this isn't just a Joplin-import quirk
The wrong-scope check lives in general
Notecode, reachable without anyimport: simply editing an existing note's title (QOwnNotes derives a
note's name from its own first line) to match another note that already
lives in the same subfolder goes through the same
storeNoteTextFileToDisk()→handleNoteTextFileName()→canWriteToNoteFile()path and destroys that other note's content the sameway — as long as that shared subfolder isn't the one currently active in the
sidebar at the moment you save. If it is the active one, the check happens
to look in the right place anyway and correctly refuses the collision.
Added a regression test for exactly that (no import involved at all)
alongside the import-specific one.
Validation
Added 4 new tests to the project's own
tests/unit_testssuite (the samesuite
build-test.ymlalready runs in CI): the import-shaped collisionfix itself, a regression check confirming collision detection still works
correctly in the case that already worked (same subfolder and active),
the plain-title-edit case below, and the
renameNoteFile()fix. All pass;the 45 pre-existing tests in the suite are unaffected.
Manual side-by-side comparison: built patched and unpatched binaries from
the identical base commit, imported the attached repro into both under
otherwise-identical fresh sessions.
Team Standupends up with only 2 of its 3 notes — the"Meeting Notes" note with Alice/Bob's action items is gone entirely, not
even recoverable from Trash (the destructive step happens before the
app's own delete-to-trash path ever runs).
attachments/gets twocopies of the one linked resource.
Team Standupnotes present (Meeting Notes.mdandMeeting Notes 1.mdcorrectly separated, each with its own actionitems, plus
Offsite Planning.md), the separate-notebookPersonal Journalnote untouched, and exactly one copy of the resource inattachments/.Manual non-import reproduction, to confirm this really isn't a Joplin-import
quirk: two subfolders, FolderA and FolderB. In FolderA, create "1st" with
body "this is the first note" and "2nd" with body "this is the second
note". With "2nd" still open in the editor, select FolderB (so it's now
the active subfolder, while both notes still live in FolderA), then
retitle "2nd" to "1st". On unpatched
main, that plays outdestructively in stages:
editor — blank. The wrong-scope check let "2nd" claim "1st"'s exact
name/filename,
canWriteToNoteFile()truncates the real1st.mdtoempty as a side effect of merely checking it's writable, and the
following disk write lands in a state briefly inconsistent with what
the UI has open.
this exact moment, each still shows its own correct, distinct content.
Nothing has actually been lost yet: "2nd"'s own DB record is
self-consistent (new name, its own real content), and "1st"'s original
DB record was never touched — only the file backing it got silently
swapped out from under it.
exact trigger wasn't pinned down — folder/note navigation seemed to be
enough), the two duplicate-named notes converge: both now show "2nd"'s
content. Two DB rows are claiming the identical filename in the
identical subfolder, which nothing prevents, so whichever one the
rescan binds to the one real file on disk gets its cached content
refreshed to match what's actually there, and the other just keeps
whatever it last held.
second note." The first note's real content is gone — not in Trash,
just gone, since the destructive step (stage 1) happens before the
app's own delete-to-trash path ever runs.
On the patched build, none of this happens: the collision is caught up
front, "2nd" is silently suffixed to
1st 1.md(filename only — its owntitle text stays "1st", which is QOwnNotes' existing, deliberate behavior
of never rewriting a note's authored first line to match its
disambiguated filename), and "1st" is untouched throughout.
Repro data
Small synthetic Joplin RAW export, no real data — 2 notebooks, 4 notes, 1
resource. Attached as
repro-duplicate-titles.zip. Import via "Import notesfrom Joplin" with "Import folders" checked:
Team Standuphas two notes titled "Meeting Notes" (recurring-meetingnotes sharing a generic title, plus one note linking the same attachment
twice).
Personal Journalhas a third note also titled "Meeting Notes" — acontrol to confirm the fix doesn't over-trigger across notebooks that
aren't actually colliding.
On unpatched
main, one of the twoTeam Standup"Meeting Notes" is lostoutright, and the attachments folder gets a duplicate file. With this patch,
all three notes come through as their own file, and there's exactly one
copy of the attachment. This is really the most useful part of the PR if
you'd rather approach the fix a different way.
repro-duplicate-titles.zip