Support pulse-height tallies in shared secondary mode - #4089
Open
GuySten wants to merge 6 commits into
Open
Conversation
GuySten
marked this pull request as ready for review
August 31, 2026 18:02
GuySten
marked this pull request as draft
September 8, 2026 15:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support pulse-height tallies with the shared secondary bank
Removes the restriction that forced
shared_secondary_bankoff whenever apulse-height tally was present, and implements the deferred per-history scoring
described in the TODO that guard carried.
Problem
A pulse-height tally scores once per source history, in the bin containing the
total energy that history's entire particle tree deposited in a cell. In the
default transport modes the whole tree is carried by a single
Particle, sopht_storage_already holds the per-history total whenevent_death()runs.Under the shared secondary bank each generation is transported as a fresh set of
Particleobjects, load balanced across MPI ranks between generations. Threethings broke:
pht_storage_is a per-Particlemember reset ininitialize_particle_track, so every descendant's deposition was dropped from its root's total.event_death()scored unconditionally, so each fragment of one physical pulse was scored as a separate pulse, shredding the spectrum toward low energy.event_revive_from_secondary()skipped it on the grounds thatcreate_secondary()had already done it;create_secondary()contained no pulse-height code. The compensating pre-add insample_secondary_photons()still ran unconditionally.Approach
Every particle carries a root index: the dense global index of its primary
in the source bank, in
[0, n_particles). It has to survive bank sorting andthe MPI migration between generations, so it travels on
SourceSite.Each
Particlestill accumulates its own fragment as before. At death thefragment is staged in a per-thread buffer keyed by root index rather than
scored. Once the generation loop drains, contributions are routed by
MPI_Alltoallvto the rank owning each root, summed, and scored once perhistory.
Carrying the root without growing SourceSite
SourceSiteis the element type of the fission bank and the source bank, bothsized by particles per batch, so an added field would be paid for in every run
including those that never touch the shared secondary bank. The existing
parent_idfield is already free by the time the root is wanted: it andprogeny_idserve only to give a site its place in the collected bank, viaprogeny_per_particle[parent_id] + progeny_id, and that key is consumed themoment the site is placed.
The field is therefore renamed to
ancestor_indexand given two accessors overthe same storage:
parent_slot()— the immediate parent's slot in the current generation's work, valid from creation until collection places the site.root_index()— the primary at the root of the history, valid afterwards.resolve_root_indices()performs the handover, once per generation, aftercollection and before the migration. A site's parent is a primary on the first
pass, so its root comes from the phase-1 partition; on later passes the parent
already carries its own resolved root, so the value propagates down the tree.
Net effect on
SourceSiteis zero fields and zero bytes, and the MPI datatypeis unchanged in size.
progeny_idstays: fission sites are appended concurrently, so their positionin the bank is nondeterministic and the ordinal is the only thing
sort_bank()has to work from.
add_surf_source_to_bank()wrote a particle ID into one fieldand a progeny count into the other, neither matching what those fields mean
anywhere else and neither read by anything, so both assignments are removed.
The phase-1 partition is computed, not stored
calculate_work()is a pure function of the primary count and the rank count —a block partition with the remainder given to the first
remainderranks — sothe phase-1 partition can be recomputed wherever it is needed rather than
snapshotted before
calculate_work()starts rewritingwork_indexfor eachsecondary generation.
phase1_first_root()andphase1_owner_of_root()givethe range owned by a rank and the owner of a root in closed form. When
min_workis zero the boundary equalsn_particles, so the branch that woulddivide by zero is unreachable.
Reproducibility
Arrival order at a history's accumulator depends on which thread transported
each descendant and on which rank it landed, so summing fragments as they are
found would make a history's total vary in its last bits with the thread and
rank count. For most tallies that is invisible, but a pulse height is binned, so
an ulp can move a whole count across a bin edge. The regression test here is
exactly the exposed case: a
delta_function(1e6)source with bin edges atlinspace(0, 1e6, 101), so every fully absorbed history lands on the top edgeand a one-ulp difference decides whether it is counted at all.
Contributions are therefore collected, not summed, and folded in track id order
at the end. A track id is the particle's global slot within its generation plus
the tracks completed in earlier generations, both global quantities, so the
order is the same for any thread or rank count, and it is unique within a batch
so the ordering needs no tie-break. This relies on the global ordering of the
secondary bank being independent of the rank count, which is the property
sort_bank()already exists to provide.The cost is one sort per batch over the fragments that actually deposited,
which in a detector problem is a small fraction of tracks.
Changes
particle_data.h—SourceSite::parent_idrenamed toancestor_index, withparent_slot()androot_index()accessors;ParticleData::root_index_and accessorsparticle.cpp— propagate the root index; implement the parent-side subtraction increate_secondary(); routeevent_death()to staging in shared mode; drop the two dead bank-ordering writes inadd_surf_source_to_bank()tallies/pulse_height.{h,cpp}(new) — staging buffers, owner exchange, canonical fold order, deferred scoringsimulation.{h,cpp}—phase1_first_root(),phase1_owner_of_root(),resolve_root_indices(); hooks in both shared driverstally_scoring.cpp—score_pulse_height_tally()takes the energy vector as an argument instead of readingp.pht_storage()initialize.cpp— removecheck_pulse_height_compatibility();mpi::source_siteloses a fieldlib/core.py—_SourceSitemirrors the renamed memberTesting
New tests in
tests/unit_tests/test_pulse_height.py:compute_particle_id()andcompute_transport_seed()both branch on the shared bank and the two modes therefore sample different random number streams. Mean deposited energy and per-bin shape are compared against combined standard errors. The thick case is the sensitive one: nothing escapes and each history spawns roughly 1.7 secondaries.Regression references for
shared_photonandshared_neutronare regenerated;the
local_*references are unchanged.Notes for reviewers
SourceSite::parent_idis renamed. It is a public member of an installed header, so out-of-tree code touching it — most plausibly a compiled custom source — needs the new name.CollisionTrackSite::parent_idis a different struct and is untouched.root_index()on an uncollected site compiles and returns the placement key. The window is about ten lines in each driver, between collection andresolve_root_indices(). If reviewers would rather have it enforced, storing the placement key negated makes the sign a free discriminant and lets each accessor assert; that is a small follow-up.int64track id each; in a typical detector problem most histories never reach a pulse-height cell. The per-rank totals array is8 * n_owned_roots * n_pulse_height_cellsbytes.Checklist
I have followed the style guidelines for Python source files (if applicable)I have made corresponding changes to the documentation (if applicable)