Skip to content

[Bug] esolver_type=dp (MDCell): message-passing models (DPA-2/repformers) abort at MD step 1 or give wrong energies — missing InputNlist communication setup #8108

Closed

Description

@MarkYoungIV

Summary

esolver_type = dp with a message-passing model (DPA-2 / repformers) fails on the MD (MDCell) path in v3.11.0-beta10:

  • with MPI enabled (the normal build), the run aborts at MD step 1:
    MPI_Comm_rank on an invalid communicator (MPI_ERR_COMM), and
  • with the abort worked around (no-op MPI interposer / non-MPI setup), the run completes but the energy and forces deviate massively from the model's own prediction (here: energy off by ~ +36.6 eV on a 50-atom cell, forces off by ~0.5 eV/Å rms). se_e2_a models are unaffected.

Root cause

The MDCell branch of ESolver_DP::runner() constructs deepmd::InputNlist without any communication metadata (nswap = 0, no world). Message-passing descriptors (repformers) call torch.ops.deepmd.border_op(...) in every descriptor layer whenever a comm_dict is present, and DeepPotPT::compute builds that comm_dict from the InputNlist (lmp_list.world, sendlist, sendnum, sendproc, recvproc, nswap). Consequences:

  1. comm_dict["communicator"] is 0 -> border_op calls MPI_Comm_rank(NULL) when MPI is initialized -> abort.
  2. Without MPI the op silently no-ops -> the forward ghost-embedding exchange never happens; the message-passing layers aggregate zero embeddings for ghost neighbors -> wrong energies/forces.

Related detail found while debugging: the reverse force accumulation is not the esolver's job — MD_func::force_virial() already calls DomainDecomposition::accumulate_ghost_forces() after runner() returns. The esolver must return the raw model output (partial forces on owned atoms + the model's ghost force entries); doing accumulation inside the esolver double-counts.

For reference, se_e2_a works because it has no message passing, and cells with nghost == 0 never hit this path. This is independent of the neighbor-ordering bug fixed in #8071 / #8104.

Proposed fix (single-rank)

In source/source/source_esolver/esolver_dp.cpp (runner(), MDCell branch), build the ghost -> owned source mapping (atom.type, atom.type_index) -> owned index and pass the comm-aware InputNlist constructor:

nswap = (nghost > 0) ? 1 : 0
sendnum = recvnum = {nghost}
sendproc = recvproc = {0}          // self-exchange
sendlist = {ghost -> owned sources}
world = &dp_world_comm             // address of an MPI_Comm variable

i.e. the same self-exchange topology the LAMMPS deepmd pair style provides in the single-rank case (world must point to an MPI_Comm variable — passing the value itself crashes inside build_comm_dict).

This implements the single-rank case only; multi-rank DP-MD would additionally need a real cross-rank exchange topology (cf. DomainDecomposition::ghost_slots_).

Verification (with the fix)

Same job, 10 MD steps, every frame of OUT.ABACUS/MD_dump vs deepmd.infer.DeepPot.eval:

frame 0..9: rmse = 2e-05 eV/A,  max|dF| <= 8e-05 eV/A,  energy within ~0.01 eV

Reproduction package

https://gist.github.com/MarkYoungIV/b692ca7288ebbef0d332f271b4414445 (parts because gists don't take binaries)

cat part_aa part_ab part_ac > dpa2_repro.zip.b64
base64 -d dpa2_repro.zip.b64 > dpa2_repro.zip
unzip dpa2_repro.zip

Contains: INPUT (10-step MD, esolver_type dp), STRU (50-atom LiGePS cell), frozen_sse_abacus.pth (a DPA-2/repformers single-task model with public weights, built from the public DPA-2.4-7M release — SSE_ABACUS branch), esolver_dp_comm_fix.patch, and verify_after_fix.py (frame-by-frame comparison). On a pristine beta10 build the job aborts at MD step 1 as described; after the patch it completes and matches the Python API to ~2e-05 eV/Å.

Related: #8071 (neighbor ordering, fixed by #8104 — the patch here does not overlap that hunk).

Activity

  1. added
    MD & LAMMD and Larege Atomic Models
    RefactorRefactor ABACUS codes
    BugsBugs that only solvable with sufficient knowledge of DFT
    and removed
    RefactorRefactor ABACUS codes
    on Oct 11, 2026
  2. 19hello commented on Oct 14, 2026

    @19hello
    Collaborator

    This issue has been addressed in the current implementation #8143 .

    A unified GhostExchangePlan is now generated by DomainDecomposition, stored in MDCell, and passed to ESolver_DP to construct the DeePMD communication-capable InputNlist. The original 4-argument API remains available as a fallback for older DeePMD-kit versions.

    Single-rank runs use mapping for periodic ghost atoms, while multi-rank runs use the communication metadata directly.

    The implementation was validated with:

    • 1-rank and 4-rank DomainDecompositionTest.*;
    • 1-rank and 4-rank DeePMD MD runs for 5 steps;
    • matching first-step energies between 1-rank and 4-rank runs.

    A dedicated DPA-2/repformers model test is still pending because no such model was available in the current test environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugsBugs that only solvable with sufficient knowledge of DFTMD & LAMMD and Larege Atomic ModelsMachine Learning & AIIssues related to AI

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions