Skip to content

Fix DMRL save() failing when log_metrics=True - #724

Merged
qtuantruong merged 2 commits into
PreferredAI:masterfrom
Shxiao101:fix/dmrl-save-tensorboard-writer
Sep 13, 2026
Merged

qtuantruong merged 2 commits into
PreferredAI:masterfrom
Shxiao101:fix/dmrl-save-tensorboard-writer

Conversation

@Shxiao101

@Shxiao101 Shxiao101 commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Description

DMRL(log_metrics=True) cannot be saved, as reported in #656.

DMRL.__init__ created a tensorboard SummaryWriter and stored it as self.tb_writer. Recommender.save() calls copy.deepcopy(self), and Recommender.__deepcopy__ deep-copies every attribute except ignored_attrs. The writer holds thread locks and an open event file, so saving fails:

A side effect: just constructing DMRL(log_metrics=True) created temp/tb_data/run_1 in the working directory, even if the model was never trained.

The writer is only used inside _fit_dmrl. This PR creates it there as a local variable, right before logging the hyperparameters, and closes it when training finishes. The log directory and the logged scalars/histograms are unchanged.

I considered adding tb_writer to ignored_attrs. It would fix save(), but a loaded model trained again with log_metrics=True would then have no writer, and __init__ would still create the directory.

How I tested

Per review, the save/load test that was in the first version of this PR has been removed, so the PR now only changes cornac/models/dmrl/recom_dmrl.py.

I verified the fix with a local script (Windows, Python 3.12, torch 2.14.0+cpu, sentence-transformers 6.0.1, cornac built from source). It trains a tiny DMRL on random feedback plus pre-encoded image features, with log_metrics=True and no model download, then calls save() and Recommender.load():

check master this PR
temp/ created by DMRL(log_metrics=True) before fit yes no (created during fit)
save() after fit TypeError: cannot pickle '_thread.lock' object succeeds
parameters after Recommender.load() identical to the trained model
pytest tests/cornac/models 10 passed, 3 failed 10 passed, 3 failed

The 3 failures are the tests/cornac/models/dmrl/test_transformertext.py tests. They need to download a Hugging Face model and I ran offline, so they fail the same way on master.

check master this PR
flake8 . --select=E9,F63,F7,F82 (as in CI) 0

Related Issues

Fixes #656

Checklist:

  • I have added tests. (a save/load test was dropped from this PR per review)
  • I have updated the documentation accordingly. (N/A, no API change)
  • I have updated README.md (if you are adding a new model). (N/A)
  • I have updated examples/README.md (if you are adding a new example). (N/A)
  • I have updated datasets/README.md (if you are adding a new dataset). (N/A)

🤖 Generated with Claude Code

DMRL created a tensorboard SummaryWriter in __init__ and kept it as
self.tb_writer. Recommender.save() deep-copies the model, and the writer
holds thread locks and an open event file, so saving raised
"TypeError: cannot pickle '_thread.lock' object" (PreferredAI#656). Constructing the
model also created temp/tb_data/run_1 even if it was never trained.

Create the writer locally in _fit_dmrl, where all logging happens, and
close it when training finishes. Add a save/load test that is skipped
when the DMRL requirements or tensorboard are missing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@qtuantruong qtuantruong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good to me. Though we should remove the test file from this PR. Thanks

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Shxiao101

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Done. I pushed a commit that removes tests/cornac/models/dmrl/test_recom_dmrl.py, so the PR now only changes cornac/models/dmrl/recom_dmrl.py.

I updated the description. The fix is still verified by a local script (fit with log_metrics=True, then save() and Recommender.load()): it fails on master with cannot pickle '_thread.lock' object and passes with this change.

@qtuantruong
qtuantruong merged commit 9a0fb78 into PreferredAI:master Sep 13, 2026
4 of 19 checks passed
@Shxiao101
Shxiao101 deleted the fix/dmrl-save-tensorboard-writer branch September 14, 2026 00:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] i am not able to do save dmrl.save() in Disentangled Multimodal Representation Learning for Recommendation (DMRL)

2 participants