Tests: scf_out_wf fails on GPU with phase-sensitive per-band coefficient references #8153

Describe the Testing Issue

The scheduled GPU validation fails in tests/11_PW_GPU/scf_out_wf when comparing the Max_wfc_1–Max_wfc_6 properties. What is the appropriate way to retain useful wavefunction-output coverage without requiring a particular phase or basis choice within degenerate eigenspaces?

Failing run and output

• Workflow run at commit 8b546bdf369f8ff9c25a8aea580dd3560a1e3931.
• Download test results. The failure log is results/cases/11_PW_GPU__scf_out_wf/attempt-1.log inside the archive (GitHub sign-in may be required; artifact retention applies).

Relevant output:

[      OK  ] etotref
[      OK  ] etotperatomref
[WARNING   ] Max_wfc_1 cal=0.75350000 ref=0.95270000 deviation=0.19920000
[WARNING   ] Max_wfc_2 cal=0.33310000 ref=0.36750000 deviation=0.03440000
[WARNING   ] Max_wfc_3 cal=0.35840000 ref=0.33510000 deviation=-0.02330000
[WARNING   ] Max_wfc_4 cal=0.34430000 ref=0.38250000 deviation=0.03820000
[WARNING   ] Max_wfc_5 cal=0.45310000 ref=0.44090000 deviation=-0.01220000
[WARNING   ] Max_wfc_6 cal=0.46110000 ref=0.47520000 deviation=0.01410000

This run reports 116 passed cases, 1 failed case and 0 infrastructure failures. The reference was not stale: the workflow copies the case and its references from the tested source commit and uses that commit's shared integration-test scripts.

What the test compares

The collector reads the first k-point's wfk1_pw.txt and emits one maximum for each numbered band. It scans real and imaginary fields separately, so the current quantity is max_G(max(|Re c_n(G)|, |Im c_n(G)|)), rather than the maximum complex modulus. It is sensitive to an arbitrary overall phase.

Even replacing it with max_G |c_n(G)| leaves a second question: this case contains degenerate bands, whose individual eigenvectors need not be uniquely defined.

Additional Context

CPU/GPU reproduction

The tested commit was rebuilt with the GPU CI build options. CPU/GPU comparisons used the same executable, one MPI rank, OMP_NUM_THREADS=1, and identical inputs apart from device. The GPU was an NVIDIA Tesla V100. The case uses init_wfc random and pw_seed 1.

• CPU output matches all six current reference values.
• GPU output reproduces all six CI values; an independent repeat gives the same values.
• CPU/GPU total energies differ by about 1.2e-11 eV.
• After matching G-vector coordinates and normalizing each vector, corresponding first-k-point band overlaps have magnitudes greater than 0.9999999992 at the original scf_thr=1e-9.
• Aligning each GPU vector's overall phase to its CPU counterpart reproduces all six reference values at the collector's four-decimal precision.

Thus, the original failure is primarily explained by overall phases, not evidence of different physical states.

To check whether tighter convergence makes a phase-invariant, per-band maximum reliable, I also tightened SCF convergence. The largest CPU/GPU difference across the six max_G |c_n(G)| values was:

All eight calculations in this table reported SCF convergence. Tightening the threshold does not stabilize the individual-band quantity.

For example, the CPU OUT.autotest/eig_occ.txt at scf_thr=1e-13 contains:

spin=1 k-point=1/3 Cartesian=0.0000000 0.0000000 0.0000000 (411 plane wave)
1 -5.438151623432725 0.2500000000000000
2  6.727204110149499 0.2411498411155524
3  6.727204110149511 0.2411498411155507
4  6.727204110149645 0.2411498411155326
5  9.179147962382407 0.000000000000000
6  9.179147962385155 0.000000000000000

Bands 2–4 are degenerate, as are the displayed bands 5–6. Across the successful convergence comparisons, the CPU/GPU overlap singular values for these displayed groups remain within about 1e-7 of unity, despite changes in their individual-band maxima. This supports equivalent subspaces with different internal eigenvector choices. We have not yet checked whether band 7 extends the latter degenerate group.

Possible cause and question

The CPU subspace diagonalization uses LAPACK hegvd, while the GPU path uses cuSOLVER. Different eigensolver implementations may choose different phases and bases within a degenerate eigenspace. The shared CG reorder logic orders energies when an inversion exceeds its threshold, but does not impose a unique convention for equal-energy states.

Energy ordering alone does not uniquely identify individual orbitals within a degenerate group. Equal-energy states may be permuted or mixed by a unitary transformation, so stabilizing their order alone would not guarantee matching coefficients. The exact solver operation at which the CPU/GPU vectors first diverge has not been isolated.

The test currently checks a particular numbered orbital's maximum coefficient even within these degenerate groups. How should this case be handled so it stops reporting false failures while still detecting broken wavefunction output?

Would it be preferable to compare a phase- and degenerate-subspace-invariant quantity (for example, a sum of squared coefficient magnitudes over a complete degenerate group), or to choose a nondegenerate fixture for this particular output test? Is there an existing ABACUS comparison convention or helper that should be reused? Simply regenerating backend-specific maxima or increasing tolerances seems unlikely to address the underlying issue.

Task list for Issue attackers (only for developers)

• Review the failing case and CPU/GPU comparison results.
• Agree on a phase- and degeneracy-aware comparison criterion for this output test.
• Implement and validate the test change on CPU and GPU.