Native array ops table - #67
Open
saidctb wants to merge 47 commits into
Open
Conversation
Fixed-shape module arrays without `target`, and array fields of derived-type module variables, are now live NumPy views like their `target` counterparts. The address is captured in C through a new `prik_capture_address` in prik_binding.h, which does the job of `c_loc` where the standard forbids it. An earlier attempt captured through a generated Fortran `target` dummy; that is invalid under F2018 15.5.2.4, where such a pointer becomes undefined on return. Target and non-target allocatable module arrays now share the one Fortran descriptor path. Logical arrays wider than one byte report the integer dtype matching their element width instead of being rejected, so `logical :: flags(3)` and its allocatable, pointer and derived-field forms are live writable views. `logical(c_bool)` stays `numpy.bool_` — one byte holding zero or one is exactly that dtype — and scalars remain Python `bool` in every kind. With the widths agreeing, no conversion remains on any path. PRIK now also requests the option that makes a Fortran logical interoperable with C: `-standard-semantics` on Intel, `-Munixlogical` on PGI/NVIDIA. Without it those compilers store all bits set for `.true.`, so a `logical(c_bool)` reaching C holds 255 where `_Bool` is defined to hold 1. It is on by default and can be turned off with `--no-standard-logicals` for the one case that needs it: linking prebuilt Intel objects compiled without it, which cannot be linked against objects compiled with it because the option also changes Intel module symbol mangling. A `character` array handle is accepted wherever a numeric one is. Verified on gfortran 11 and ifx 2026.1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
An allocatable actual argument cannot be built from C. F2018 18.5.5.6 requires a null `base_addr` when the attribute is `CFI_attribute_allocatable`, because an allocatable established from C must start unallocated; pairing that attribute with a real address describes an already-allocated allocatable, which the standard does not permit C to construct. PRIK did exactly that, so the same wrapper worked on gfortran -- which does not enforce the constraint -- and failed on ifx with `CFI_ERROR_BASE_ADDR_NOT_NULL`, surfacing as `Unable to establish native descriptor for argument a: 2`. The binding now keeps the descriptor the Fortran runtime already hands to its callback instead of rebuilding one from facts. Only the descriptor record is copied, not the array data, so the cost does not grow with array size, and the copy is remade on every call because reallocating the native entity invalidates the previous one. Module variables, derived-type fields and returned results all reach a read-only allocatable dummy on every compiler now, including the bounds a shifted allocatable carries. That also removes a per-call Python facts mapping, which makes such a call about twice as fast. A borrowed copy cannot stand in where the callee changes the allocation: an `intent(inout)` allocatable may deallocate and reallocate its dummy, and through a copy that lands in the copy and leaves the caller's entity pointing at released storage. The descriptor handoff therefore records whether it owns its descriptor, and a projected argument refuses a borrowed one rather than accepting it. Optional allocatable dummies keep the fact-packed form, whose absent branch establishes the unallocated placeholder that pairs with the present flag. Verified on gfortran 11 and ifx 2026.1 across module, derived-field, result and pointer handles, allocated and unallocated, against read-only and writable dummies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Reaching the entity behind an array handle costs a Python operation lookup today: the binding imports the runtime module, fetches a helper, calls it, and that helper dispatches through a PyCFunction wrapper to reach a generated C function that finally calls the bridge. The work at the end of that chain is a single indirect call; measured against a generated extension, the descriptor round trip is 0.06 us out of roughly 7 us for passing one handle argument. A module array handle now publishes `prik_native_array_ops`, a versioned cross-extension record naming the bridge entry points for its entity. It carries the identity a consumer must agree on -- descriptor kind, rank, CFI type, element size -- and `owner`, the address the entity needs, which is resolved once when the handle is built rather than looked up per operation. `scoped_descriptor` invokes a consumer while the runtime's descriptor is valid, so the consumer decides whether to copy the record out or make its call in place. The descriptor stays a compiler-owned representation in this record, as everywhere else in the header, so it does not depend on the Fortran interop header and remains usable from a C-only extension. A generated forwarder bridges the bridge's own consumer signature to the table's, which avoids casting between function pointer types. Nothing consumes the table yet, so this changes no behavior. Reaching it from C and calling through it runs the same call about 28 times faster than the current path, which is what the following change will use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A read-only allocatable argument no longer reaches its entity through the Python runtime. When the handle publishes a native entry-point table the binding validates it in C -- descriptor kind, rank, CFI type and element size must agree with the dummy -- fills a call-local descriptor through `scoped_descriptor`, and checks the extents before use. Passing a module allocatable to such a dummy costs 0.31 us rather than 7.26 us, close to the 0.21 us an ordinary array argument costs. The table is taken only where a copy of the runtime's descriptor is a valid actual. A projected argument keeps the general path, because a callee that reallocates through a copy would change the copy alone; that is the same rule the borrowed handoff follows, enforced here while the code is generated rather than checked as it runs. An optional argument keeps it too, since the table carries no presence flag, and so does a dummy that fixes an extent, since the table declares no shape. A handle that publishes no table falls back unchanged, which is every derived-type field and result today. The negative-extent gate that guards the general path is kept, now against the filled descriptor in C rather than a separately fetched shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Both remaining allocatable handle kinds now publish the record module arrays already did, so a read-only descriptor argument reaches them without a Python operation lookup: a derived-type field costs 0.32 us rather than 7.64, and an owned result 0.27 us rather than about 7.3. The two kinds differ from a module array, and from each other. A field reaches its entity through the parent's address, so its record cannot be a file-scope constant: it is built when the handle is, released by the capsule that carries it, and its owner is resolved once instead of on every operation, which is where a field previously spent a capsule attribute lookup per call. Whether the bridge takes that address at all depends on the field: a derived-type field passes it, a module member does not. An owned result needs no bridge call for its descriptor. It allocated that descriptor and the callee filled it in place, so the current state is already local and the forwarder hands it straight to the consumer. That record is published before ownership moves into the handle capsule, while the pointer is still named. Both forwarders are declared ahead of the code that publishes them, because a handle getter is emitted before the forwarder it names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A read-only descriptor argument receives a per-call copy of the descriptor the Fortran runtime built, carried by a capsule the handle keeps alive so it survives the binding releasing its own reference. That capsule is replaced on the handle's next descriptor request, which is safe only while nothing else can make one mid-call. A function marked `@nogil` releases the GIL around its native call, so another thread can reach the same handle while the first is still using the descriptor: it replaces the capsule, the copy is freed, and the descriptor in flight is left pointing at released storage. The window is narrow and needs the same handle passed concurrently, but nothing in the binding prevented it. The binding now copies the record into the call's own storage before releasing its reference, so what it passes to the entrypoint belongs to that call alone. This is what the entry-point table path already did by filling call-local storage; the two paths now agree. A projected argument needs no copy, since its descriptor is the persistent one its handle owns rather than a per-call capsule. The comment describing the earlier reasoning claimed a generated binding runs no Python between reading the capsule and returning from the native call. That holds only while the GIL is held, and is corrected here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Every generated accessor for a module array either allocates,
deallocates, or passes the variable to a dummy the callee may define.
PROTECTED forbids all three outside the declaring module, so the bridge
could not compile: the build failed with three Fortran diagnostics about
ALLOCATE, DEALLOCATE and INTENT(INOUT) on the variable, none of which
named the attribute responsible.
The parser now records the attribute and completed policy refuses the
variable, naming the reason once:
Semantic variable 'm.guarded' has unsupported module-variable policy:
module variable 'guarded' is PROTECTED, so a generated accessor cannot
define it outside its module
The attribute travels as internal declaration metadata rather than a
serialized parser field, alongside the other facts that matter to
generation without describing the parse tree, so recorded fixture output
is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The consumer a module array hands its descriptor to declared the dummy `intent(in)`, so a callee reached through that descriptor could read the array but nothing it changed about the allocation could travel back: with `intent(in)` the compiler has no obligation to copy the descriptor back into the module variable when the bridge returns. Declaring it `intent(inout)` restores that obligation. A reader is unaffected, because copying back an unchanged descriptor changes nothing, and reading paths are byte-identical on gfortran and ifx. What it enables is the case PRIK has always refused: a procedure that reallocates its allocatable dummy, reached through the descriptor the runtime builds for that call, now updates the module variable rather than a copy that is discarded. Nothing uses that yet -- the binding still hands the callee a copy, and a writable descriptor argument is still refused at the boundary. This is the bridge half of that support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Passing a module array or a derived-type field to a procedure that reallocates its `intent(inout)` allocatable dummy was refused. The descriptor those entities hand out exists only while the consumer holding it runs, so a callee given a copy would replace the allocation in the copy, and the caller's entity would be left naming released storage. Refusing was the honest response to that. The call is now made inside the consumer, where the descriptor the runtime built is still live, so what the callee writes into it is what Fortran copies back to the entity when the bridge returns. A module array and a field both reach size 6 from size 2 through such a callee on gfortran and ifx, where before the argument was rejected at the boundary. The consumer serves both routes to a descriptor. A handle publishing native entry points builds one inside it; a handle that owns a descriptor already -- a result, or a contract default with no native entity behind it -- hands that over directly. The call appears once either way. Field consumers take `intent(inout)` as module ones now do. They previously took `intent(in)`, under which the copy-back is unspecified; both compilers tested happened to perform it, which is not something to depend on. The inversion covers the case where the descriptor is the entrypoint's only parameter. A value result or a further argument would have to travel out of the consumer, which needs a context record this does not build, so those keep the general path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A read-only descriptor argument was placed by copying the record the Fortran runtime built into call-local storage and passing the copy. That worked, but it meant C writing a C descriptor, which the standard reserves for its own descriptor functions, and it left two ways of placing the same kind of argument depending on whether the callee could change the allocation. Both now go the same way: the call is made inside the consumer holding the runtime's descriptor, and C passes on the pointer it was given rather than a record it assembled. Nothing copies a descriptor, and the generated binding contains no `memcpy` of one. Carrying the other call values through a context record, added with the writable case, is what makes this possible for arguments that are not the entrypoint's only parameter or that return a value. The copy machinery goes with it: the separate placement path, the condition selecting between the two, the detachment that protected a copied record from the capsule carrying it, and a gate for shapes a descriptor dummy cannot have -- Fortran requires deferred shape there, so it could never fire. Reaching a handle's entity no longer uses any of its Python operations: placing an argument went from three such calls, to one, to none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Placing a descriptor argument no longer copies a descriptor, so the parts that existed to produce such a copy and to keep it apart from a real one have nothing left to do. A module array and a field handed their descriptor operation a consumer that copied the record into a capsule; both return decoded facts again, as they did before that consumer existed. The runtime adapter wrapping those capsules, the binding helper that accepted them, and the flag on the handoff recording whether the descriptor was owned or copied are gone with it. That flag existed only to stop a copy standing in where a callee could change the allocation, and nothing copies now. The form a handle's descriptor operation returns no longer has to be announced from generated code through the runtime factory, and the header helpers for copying a descriptor record are unreferenced. Reaching an entity through its published entry points does not depend on whether the argument is projected back to Python, so generated code no longer consults output projection at all when placing one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A callee that re-associated a `pointer, intent(inout)` dummy was silently ignored. The binding packed the descriptor's fields in Python and rebuilt a descriptor in C for the call, so `v => target` in the callee re-pointed that rebuilt copy and nothing else: the handle passed in came back still unassociated, with `associated` False and no shape, and no error was raised. Pointers now take the route allocatables already take. The bridge hands the native entity to a C callback and the call is made inside it, on the descriptor the compiler built, so what the callee does to the entity is what the caller's entity sees. Pointer and allocatable dummies are one mechanism rather than two: the three predicates that named only the allocatable interop kind now name both, and the plan validator accepts either descriptor kind on the direct ABI. Because a direct descriptor is the runtime's to build, a caller-created handle needs storage of its own to hand over, so default construction is now lazy owned storage for every non-optional descriptor argument rather than only result-projecting ones. An argument that does not project a result supplies that storage without also naming what the handle exposes, so a read-only argument no longer stamps its own to_numpy policy onto a handle the caller will go on using elsewhere. Verified on gfortran and ifx: the descriptor matrix is identical on both, and a callee's reallocation and re-association both reach the caller's entity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Passing a handle to an optional allocatable or pointer dummy raised `Unable to establish native descriptor for argument ...: 2` on ifx while working on gfortran. The fix for non-optional dummies did not reach these: routing was decided from `optional_absent`, a property of the declaration rather than of the call, so marking a dummy optional sent both its runtime cases down the old route. The present one then rebuilt its descriptor in C and made exactly the `CFI_establish` call F2018 18.5.5.6 forbids. A present optional argument is an ordinary descriptor argument, so it now takes the direct ABI and the consumer inversion, and a callee that reallocates or re-associates one reaches the caller's entity. Three places had to stop assuming the inverted argument was never optional: the inversion candidate filter excluded it; the consumer call substituted the descriptor for every value in its parameter group, so the present flag was passed the descriptor; and presence was decided only in the packing branch, so a handle publishing a table read as absent. Absence is routed by what the entrypoint can express, not by the Python signature. A generated bridge takes its descriptor dummy unconditionally and reads a separate flag, so the absent branch establishes the unallocated placeholder that pairs with it -- legal precisely because absence is when there is nothing to point at. A direct `bind(c)` entrypoint has no such flag, since PRIK cannot add a parameter to a signature the user wrote, so there absence stays a null descriptor pointer. Establishing a placeholder there would have made an absent argument indistinguishable from a present but unallocated one. A handle whose `descriptor` operation only reported base address, element length and bounds is no longer accepted: rebuilding a descriptor from those fields is the unsound step being removed. The optional fixture now allocates and associates through real entrypoints, which also proves the callee's work reaches the handle. Verified on gfortran and ifx: absent, present-unallocated and present-allocated stay distinct on both routes, and a reallocating callee grows the caller's module variable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
With pointers and optional arguments placed like every other descriptor argument, nothing selects the fact-packed route. It was unreachable rather than merely unused: an array handle's descriptor kind is only ever allocatable or pointer, and both now take the direct ABI, so the selector's final branch could not be reached. Removed: the two fact-packed policy selectors; the six descriptor fact roles on the plan and the planner helpers that named them; the role-count validation chain those roles existed to check; the C lowering that established call-local storage from packed fields and its unpacker; and the two runtime packers that reported a descriptor as base address, element length and bounds. The ABI selector now states the rule it actually applies -- every descriptor a call receives is the runtime's to build, except a result, where the wrapper owns storage the callee allocates into. Three of its four parameters were dead: the descriptor-kind test listed both members of a two-member enum, and the projection test was subsumed by it. Five tests went with it. Four drove the deleted packers through handles that only reported fields; of their invariants, optional presence mapping is kept against the surviving helper, because a present but unassociated pointer must stay distinct from an absent one, while field packing no longer exists and the kind and dtype validation is already asserted against the shared validator in the same file. The fifth pinned a plan edit naming a role that is gone. A fresh contract handle now has a test saying it owns no descriptor until a generated binder attaches one, which is the guard that replaces the empty descriptor the old path would have invented. Verified on gfortran and ifx: the descriptor matrix and the optional cases are unchanged on both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A handle the caller creates is given wrapper-owned descriptor storage the first time it is passed to a descriptor argument. That storage lives as long as the handle, so there was no reason for later calls to keep going back to Python: they can read it from C the way a module array's is read. The binder now publishes the storage it attaches as a native entry-point table, and `_bind_contract_native_array_handle` records it on the handle. Reaching it needs no call-scoped window, since nothing else owns it, so the table's entry point is one shared forwarder that hands the descriptor straight to the consumer. Measured on this fixture, passing such a handle to a `real(8), allocatable` dummy went from about 8.8us to about 0.26us per call -- below the 0.46us a plain NumPy array costs, which still has a buffer to validate. The Python packer is now called once per handle for its lifetime rather than once per call; module, field and result handles are unchanged. The table has to name the storage rather than the local that allocated it: the binder clears `owner_descriptor` once its ownership moves to the handle capsule, so a separate non-owning `owner_storage` is captured right after the descriptor is established. Publishing the cleared local instead stored a null owner, which read as an unallocated array and, on a writable dummy, allocated into a null descriptor. Verified on gfortran and ifx: the descriptor matrix and the optional cases are unchanged, a callee's allocation still reaches such a handle, and a handle bound by one entrypoint works through another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Reading a handle went through the descriptor twice. The generated extraction had the descriptor in hand and reported it as base address, element length and per-axis bounds; the runtime then decoded those fields back into a view, on every call. The view is now built in the generated operation, from the descriptor it already holds. Measured on this fixture, `to_numpy()` on an allocatable handle went from about 12.9us to about 2.7us per call. The view keeps its handle alive rather than only the record describing its storage: the handle releases that storage when it is finalized, so a view that outlived it would otherwise read freed memory. Retention is applied exactly when the view borrows what the owner record holds, which leaves an operation returning an array of its own handed back untouched. Two forms keep reporting fields. A pointer describes its target to another pointer through them, and a view cannot carry the Fortran lower bounds an association preserves; a character element width is only known at runtime. An allocatable never associates, so its extraction returns the view. Verified on gfortran and ifx: values, strides and write-through are unchanged, a view outlives the handle it came from, and the descriptor matrix and optional cases are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
An ordinary array dummy takes an address and extents. When a handle was passed to one, the binding asked the runtime for those facts an operation at a time -- nine round trips per call, the shape among them computed twice -- even though the handle's table already names the storage. The binding now reads the descriptor directly when a handle publishes a table: a generated reader copies the base address, the extents and a contiguity flag out while the runtime holds the descriptor open, and the call proceeds on the address it names. An ordinary call cannot change the allocation behind the dummy, so reading it once before the call is enough. Measured on this fixture, passing a handle to a `real(8) :: v(n)` dummy went from about 17.5us to about 0.27us per call, below the 0.55us a NumPy array costs, which still has a buffer to validate. The kind is deliberately not compared when reading the table: an allocatable and a pointer are equally acceptable at an ordinary dummy. Everything that decides whether the storage fits -- rank, element type, element size, fixed extents and contiguity -- still is, and each diagnostic reuses the runtime's own wording so a handle reads the same whether or not it publishes a table. The reader dereferences a descriptor, so it is emitted only where the module already includes the Fortran interop header. A C-only wrapper never includes it and keeps the shared binder, as do flattened storage and the runtime rank, stride and itemsize shapes. Verified on gfortran and ifx: values, the descriptor matrix, contiguity rejection for a strided pointer target, and the unallocated, unassociated and shape-mismatch diagnostics are all unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
An allocated array of size zero stopped reaching an ordinary array dummy: it was rejected as a shape mismatch where it previously passed. gfortran describes an empty dimension with an extent of -1 and ifx with 0, and the reader added for the array fast path passed the raw value through as the extent the dummy receives. Every other path already normalizes that sentinel -- the runtime does when it decodes descriptor fields, and the generated NumPy view does -- so the reader now does too, and the contiguity walk multiplies the normalized extent rather than the raw one. The suite passed with the regression present because nothing passed a zero-sized handle to an ordinary array dummy. A test now does, and it fails without this change. Verified on gfortran and ifx: a zero-sized handle sums to zero through an ordinary dummy on both, and a three-element one is unaffected. Confirmed separately on both compilers that a null base address agrees with Fortran's own allocated and associated inquiries in every state, empty included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Reading a handle asked two questions where the answer to both was in front of it. `to_numpy()` and `shape` each queried allocation state through one generated operation and then the value through another, and then re-checked what the generated code had already built from the declared type. A generated inquiry over storage the handle owns reads its descriptor directly, so it can report an absent state as None rather than being asked first. The shape inquiry now does; the extraction already did. On such a handle `to_numpy()` went from about 2.7us to about 1.3us and `shape` from about 2.9us to about 1.3us. The route is taken only when the handle both publishes native entry points and owns its descriptor. Publishing alone is not enough: a module or field handle publishes one too, but reaches its entity through the compiler's own inquiries, which say nothing about whether the entity is there. Gating on publication alone made a disassociated pointer report a shape of zero instead of none. `_call_op` is unchanged and still dispatches every operation, including all mutation, for every handle. This route sits beside it. Verified on gfortran and ifx: an unbound handle, an allocated empty one, an allocated one, a deallocated one and a module handle all report the same shape and extraction as before, and a view still outlives the handle it came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Passing a handle to an ordinary array dummy read its descriptor directly only when the module already included the Fortran interop header, and a module includes that when it uses descriptors somewhere else. Whether an argument was placed from C therefore depended on an unrelated variable in the same module rather than on the argument. The header now follows the argument: a module whose ordinary array dummies accept a handle needs it. A module with no Fortran behind it has no descriptors to read and keeps the runtime route. Two further dummy forms are covered. An assumed-shape `values(:)` carries strides and bounds beyond an address and extents, so its reader fills the same record the runtime fills and everything downstream is untouched; a handle reaches such a dummy contiguously, as the runtime already required, so the strides are unit and each upper bound is its extent's last index. An assumed-size `values(*)` collapses an actual of any rank into one axis, so its reader walks the rank the descriptor reports and folds the collapsed axes into a product. Neither states an expected rank when the dummy takes any, the way a character dummy already states no element size. A character dummy is matched on its declared width as well as its kind. The entry-point table carries no width for character storage, so it can neither match a declared one nor be trusted to skip the check, and such a dummy keeps the runtime route where the comparison is made on the dtype. Verified on gfortran and ifx: an assumed-shape dummy, an assumed-size dummy taking rank-1 and rank-2 handles, explicit and multidimensional dummies, and a noncontiguous pointer target refused with the runtime's own wording. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A handle that does not fit an ordinary array dummy was reported in two places. The binding refused a mismatched rank, element type or element size with one message while the runtime, reached when the binding declined, described the same conditions in its own words. Passing a `real(4)` handle to a `real(8)` dummy therefore changed what it said depending on which route ran, and the branch had already changed that message without meaning to. Every refusal is now made where the storage is inspected. A handle of the wrong shape names the rank and element width it carries and the dtype the dummy expects; storage that is not there, a noncontiguous pointer target and an element width the dummy did not declare each report the condition found. The wording is not the runtime's word for word; the conditions are the same ones, and there is one description of each rather than two that can drift. A character dummy is covered by the same route now. Its width is compared against the width the descriptor states rather than one the entry-point table would have to carry, so a handle whose elements are a different length is refused as before while a matching one is placed from C. Handle acceptance is also gated on the source language. A C module's array parameter never accepted a handle -- its binding takes an ndarray -- but completed policy said it did, so the plan disagreed with the code it produced, and that disagreement is what let generated readers dereference a descriptor in a wrapper with no Fortran behind it. Verified on gfortran and ifx: a matching handle is placed without entering Python on any path, and a wrong dtype, a wrong character width, an unallocated handle and a noncontiguous pointer target are each refused with the condition named. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The runtime carried two cross-extension records for the same thing. A borrowed handle published an entry-point table naming its bridge symbols; an owned handle published that table *and* a second capsule holding the descriptor storage it had allocated, with its own magic word, version and release callback. A reader had to know which of the two it was looking at. There is only one question either record answers: how do I reach a live descriptor for this handle? So there is now one record that answers it. `with_descriptor(context, consumer, consumer_context)` enters the entity and runs the consumer while its descriptor is valid -- into Fortran for a module variable or a field, straight onto persistent storage for an owned handle -- and `release` is non-NULL exactly when that context is storage this extension must free. An owned handle's `owner` and its published backend are now the same capsule. The version moves into the capsule name. PyCapsule_GetPointer already refuses a capsule created under any other name, so the two magic words and the two ABI version fields were re-checking what the name had settled; `struct_size` stays as the one tag that catches a layout change made without renaming. `descriptor_size` stays because nothing in a descriptor can be read to establish the producer's CFI_CDESC_T layout, and the kind, rank, element type and width stay so a mismatched actual is refused before any Fortran is entered rather than after. Also fixes the CLI argument tests, which called main() with no argv and so read the xdist worker's empty sys.argv and printed help instead of dispatching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A handle's shape, allocation and association state, element width, contiguity and NumPy view were each reached a different way. A borrowed module array had a Fortran procedure per inquiry and, for its view, built a Python dict of descriptor fields that handles.py decoded back with ctypes. An owned handle called Fortran procedures that took the descriptor it was already holding. All of it describes the same descriptor. So all of it now reads that descriptor, once, where it is valid. Five small consumers -- present, contiguous, element length, shape, view -- run through the backend's one entry point and hand back the finished Python object. Nothing copies a descriptor out, and the bridge emits nothing per variable for any of them: only the descriptor entry point and the mutations that must reach the entity survive. The view is built with the descriptor's own byte strides, so a reversed or non-contiguous pointer target comes through without the intermediate ctypes buffer the Python decode needed. Timings for a rank-1 module allocatable, best of five runs of 20k calls: .shape 4838 -> 1371 ns, .to_numpy() 12666 -> 1331 ns. A derived-type field goes 4001 -> 1668 and 12237 -> 2262. An owned handle pays about 150 ns more than its old inline read for going through the same entry point as everyone else, which is the price of there being one mechanism. One declaration cannot take this route. A bind(C) character dummy must have an assumed or constant length, so `character(len=:), pointer` has no legal descriptor interface; gfortran mistranslates it rather than rejecting it and divides by a zero element length in cfi_desc_to_gfc_desc. Completed policy now says so, and such a handle keeps generated Fortran inquiries and offers neither a view nor an association -- both of which crashed before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
…he matrix Sixteen methods in the C binding and three Fortran procedure builders were left with no callers once every inquiry started reading the descriptor: the descriptor-record builders, and the owned handle's own allocated, associated, contiguous, shape, element-length, descriptor and to_numpy bodies, each of which now runs the same shared consumer as everyone else. The array matrix covers what the one route has to keep working: integer, real(4), complex, interoperable logical, fixed- and deferred-width character, zero-sized, and rank-3 storage, each reporting its own dtype, shape and view and reaching a dummy of its own type; a pointer target that is strided, reversed, or unassociated; a derived-type field whose view outlives the parent name; and reallocation through a dummy landing on the caller's handle. A reversed target is where the direct stride mattered most, so its view is asserted where PointerPolicy makes one available -- shape, sign of the stride, the values, and that a write through it reaches the same storage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The handle now carries a dispatcher, a completed capability set and one versioned backend capsule instead of a dictionary of generated callables and a second owned-descriptor record. Every inquiry -- shape, allocation and association state, element width, contiguity and the NumPy view -- is answered by a shared C consumer run over the descriptor the backend makes live, so nothing is packed into Python values and decoded back, and the bridge no longer carries an inquiry procedure per variable. Argument handoff reads the capsule in C, so a bound handle reaches an ordinary or descriptor dummy without executing a Python frame; the test added here observes exactly that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A call reaches a borrowed entity by running inside the consumer holding its descriptor, and only one call fits inside one consumer. An entrypoint with a second descriptor dummy therefore took the general path, which read the backend's context as the descriptor -- true only of an owned backend, whose context is its own persistent storage. A module array's context is NULL, so two module handles reached the callee as null descriptors and the Fortran runtime freed them: free(): invalid pointer. The general path now asks for a descriptor that outlives a consumer and is told when there is none, naming the argument. Caller-created handles, which own their storage, are placed exactly as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
There was still a second way to reach a descriptor argument: read the backend's context in the wrapper frame and call from there. It worked only because an owned backend's context is its own persistent storage, so a borrowed handle had to be refused, and every argument on that route was packed by a helper in prik.runtime.handles first. Now one consumer is emitted per descriptor argument. Each records what it was handed and enters the next, so the last one makes the call with all of them live, and a callee that reallocates or reassociates any argument writes into the descriptor Fortran copies back to that caller's entity. An owned handle enters the same way, because its entry point hands the consumer its own storage; an absent optional has nothing to enter and passes on the unallocated placeholder recorded for it. A hidden output needs nothing new: its address is carried like any other value, and this frame outlives every consumer it enters. Argument handoff now runs no Python for any supported form, and the two refusals this removes -- more than one descriptor dummy, and a descriptor dummy beside an intent(out) argument -- become supported for borrowed handles too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
allocate, resize and deallocate take their own handle's descriptor out of the backend's context, which is right only for a backend that owns its storage. Nothing published those operations on a borrowed handle, so the raw cast was never reached with one -- but it is the same shape that handed a null descriptor to a callee, and naming the requirement makes a mis-wired capsule an error rather than a call into CFI_allocate with nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The versioned capsule name is what refuses an incompatible producer, so an extension built against the earlier branch-only table has to be regenerated. That is the one thing a builder cannot find out from their own code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The version in the capsule name was only asserted as header text, and the absent-optional path was only ever measured by hand. Both now have a test. A capsule holding this handle's own live backend, republished under another version name, is refused by the reader that asks for the one it understands -- so the same address goes from working to raising, and only the name differs. That is the whole reason the version is spelled in the name rather than compared out of a field the stranger also wrote. An optional descriptor argument runs no Python whether it is omitted, passed as None, or supplied, which is what the C short-circuit on Py_None bought and what the earlier matrix never covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The name is the ABI version, which only means anything if the record cannot move under it -- and it could: adding a field, reordering two, or widening one all left the name at .v1 and the suite green. A same-width reordering is the bad one, because struct_size sees no difference and a consumer that still recognizes the name reads the fields straight through in the wrong order. The record and the name are now pinned together in one test, so changing either alone fails, and the header says which way to resolve it: publish .v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
A version anyone types is a version someone forgets, and the field-by-field checks could not cover the part that matters most: context, with_descriptor and release are bare addresses, so a producer that merely exchanged two of them passed every check and segfaulted on the call. Nothing can sanity-check an address after the fact. So the layout names the capsule. A tag folds sizeof and offsetof for the record and each field in order -- reorder, widen, insert or remove and it moves -- and PyCapsule_GetPointer compares names before it hands back the pointer, so a mismatched producer is refused with nothing read through it. Both sides compute the tag from their own header, so agreement on the name is agreement on the record. struct_size goes: the tag folds the size in and covers reorderings it never could. The v2 in the name stays for people, and for the one drift no tag can see -- a field that keeps its offset and width but changes meaning. The tag lists its fields by hand, so the record and that list are pinned together in a test; adding a field to one and not the other fails there rather than quietly keeping the old name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The tag folded offsets and widths, so a version number was still carrying one case: a field that keeps its shape and takes on a new meaning. That is not a case a hand-kept number is good at -- it works only when someone remembers, which is the property this design set out to remove. Each field now folds its name in along with its offset and width, all three from the one token that names it. Renaming a field when its meaning changes is what you would do anyway, and now every reader built against the old meaning stops recognizing the record. So the version segment has nothing left to distinguish, and the capsule is named for its layout alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The entry still named a version the last two commits removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
… is refused The direct question -- what would a bind(C) procedure with no bridge receive for this dummy -- has one answer per array form, and nothing recorded it. Completed policy now carries it: an explicit-shape, assumed-size or raw C pointer dummy is reached by the address of its first element, and every other form by a CFI descriptor, which has an extent and a signed byte stride per axis. Whether an axis of the actual may run backwards follows from that, plus the dummy's own contiguity requirement, which a descriptor does not lift. The refusals say which of those they come from. A reversed axis is a perfectly good Fortran array section that an address-only entrypoint has no way to describe; a broadcast axis is not a section at all, because Fortran has no form for an element repeated by a zero step; and axes in the wrong order keep the ordering message, which is what a caller can act on. Previously all three read "expected ordering (F)". The mechanism that would carry a signed stride is not built yet, so ordinary arrays still cross as an address with extents beside it, and _array_handoff_ signed_strides says so rather than claiming an ability the transport does not have. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The construction a strided actual needs: CFI_establish only makes contiguous descriptors, so a view has to be a section of one. The parent established here is the smallest contiguous array the view is a section of, with steps chosen to divide the view's, and CFI_section cuts the view back out of it -- which is what carries a signed stride. An axis that runs backwards starts at its far end and walks down; an empty array needs no section at all. The conditions this needs turn out to be the ones validation already enforces: requiring each axis to step a whole number of elements, in increasing order of step, without overlapping, is exactly what guarantees the parent contains the section. Verified against both compilers before wiring: rank-one reversal, step -2, rank-two with either axis reversed, both reversed, and a zero-sized axis, read element by element and through a real assumed-shape dummy, with intent(inout) writeback touching exactly the viewed elements. Lower bounds differ -- gfortran normalizes a section to 0, ifx keeps the parent subscript -- so nothing may assume them. The bridge dummy for such an argument is the array itself rather than an address with extents beside it, so no pointer local, no c_f_pointer and no section reconstruction is emitted for it. Ordinary arrays do not take this route yet: a native handle reaching the same dummy has to be entered through its own descriptor entry point, and that is the next piece. _ORDINARY_ARRAYS_CROSS_AS_DESCRIPTORS names the gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Both sources now arrive at the same slot. A handle already has a descriptor and it is valid only inside its own entry point, so the consumer chain enters it; a NumPy array has none, so a section is built over its storage in the caller's frame, which outlives the call. The chain needed no new shape for this: a slot with a backend is entered, a slot without one is passed on, and that is what already distinguished a present optional from an absent one. An omissible descriptor dummy is declared optional and its presence asked with present(), because C omits such an argument by passing no descriptor rather than a null pointer beside one. Verified on the shapes that matter: a reversed NumPy view, a reversed pointer handle and a reversed bind(C) call all reach an assumed-shape dummy and return the right sum, and the handle path runs no Python frame. Still gated. Removing the extent parameters removes what other declarations read from them -- a result declared dimension(size(x)) among them -- and those extents have to be carried out of the descriptor before this can be the only route. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The route is now the only one for an assumed-shape, deferred-shape or assumed-rank dummy, which is what its direct bind(C) equivalent receives. A reversed NumPy view, a reversed pointer target and a reversed bind(C) call all reach such a dummy and read the elements the caller sees; writing through one reaches the caller's own storage. What the extents left behind is carried out of the descriptor while it is live, because a borrowed one is gone once its consumer returns, and a declaration written in terms of this array's shape still has to read them. The same read reports whether there is storage at all, so an unallocated allocatable and an unassociated pointer are refused as before, and a character dummy is still matched on its declared width -- from the descriptor for a handle, from the array for a NumPy actual. The positive-stride reconstruction is gone. A probe confirmed nothing reaches it: no c_f_pointer, no base pointer, no dense-actual flag and no upper-bound or stride parameters are emitted for these dummies. Explicit-shape, assumed-size and C pointers keep the address handoff, which is all their ABI can carry. An interoperable dummy takes its character width from the descriptor, so a runtime-width character array is declared assumed-length; both compilers accept that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
The matrix the feature needed: rank-one reversal and step -2, rank-two with either axis reversed, both reversed, mixed sign and zero-sized; from NumPy views, module pointer handles, derived-type field handles, through a bridged dummy and a direct bind(C) one, with several descriptors in one call and an optional one omitted, passed as None and supplied. The rank-two cases weight each element by its subscripts, so reading the axes in the wrong order is a different answer rather than the same sum. Two compiler-driven limitations, both found by this matrix and both recorded where they are decided: GNU Fortran's CFI_section resets a character descriptor's elem_len to 1, so a sectioned character array reaches the callee with every element truncated to one character; Intel's is correct. A silently wrong width is worse than a refusal, so completed policy does not section a character array at all. An assumed-rank dummy asserts contiguous storage in its own contract, so a reversed view is refused there as it would be for any dummy that asks for contiguity. Its rank still comes from the descriptor. A profiling test pins the handoff: a reversed borrowed handle reaches an ordinary dummy with no Python frame. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
…receive it GNU Fortran 13 rejected every module with a character array behind a descriptor: an interoperable allocatable or pointer character dummy must declare deferred length, and the generated consumer declared assumed length beside the allocatable attribute. GNU Fortran 11 and Intel's ifx accept that combination silently, which is why it reached CI. Deferring the length is not open either: argument association requires the actual to declare deferred length exactly when the dummy does, so an array whose width is fixed has no allocatable dummy it may be associated with at all. It is therefore taken by an assumed-shape assumed-length dummy, which is the only form that can receive it -- and the form this consumer had before the attribute was added. The cost is the declared lower bound and reach of the allocation state, for fixed-width character arrays only; deferred-length ones and every numeric array keep the allocatable dummy. A codegen test now asserts no interoperable character dummy is allocatable or a pointer with assumed length. Nothing available here compiles that rule -- both local compilers accept it -- so the test reads the generated declaration instead, and fails if the attribute comes back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…ACK example A default `LOGICAL` is four bytes wide under GFortran, and this branch aliases a logical array element for element instead of copying it through a one-byte representation. So `BWORK` and `SELECT` cross as `numpy.int32`, which is the dtype SciPy and f2py have always required here -- their calls in these very tests already spell `selection.astype(np.int32)`. Only the PRIK calls still passed a `numpy.bool_` buffer, and they are what CI refused. The inventory recorded this as a PRIK ABI adapter for `dtrsen` and `dtgsen`. No adapter is generated any more, for those or for anything else, so the entry is retired rather than reworded. Verified outside LAPACK, on a `logical, intent(inout) :: flags(*)` dummy and a `logical, intent(in) :: selection(*)` one: the binding reports `ndarray[int32]` and refuses `numpy.bool_`, GFortran writes 1 for `.true.` into that storage, and reads a caller-supplied 1 back as `.true.` -- one selected element out of `[False, True]`, which is the count `dtrsen` is asserted to return. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cx8Mtgc7Sw7BftrTmwP4Ce
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.