Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 58 additions & 38 deletions docs/native-refactor-progress.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ improvement resumes after the final native audit.
| --- | --- | --- |
| R0 | Native settlement and CI convergence | Complete, [PR #241](https://github.com/pineforge-4pass/pineforge-engine/pull/241) |
| R1 | Native market orders, host, drivers, calendar and forward runner | Complete, [PR #243](https://github.com/pineforge-4pass/pineforge-engine/pull/243) |
| R2 | Resting orders, partial execution, owner-bound children and group effects | Design under review; next implementation slice |
| R2 | Resting orders, partial execution, owner-bound children and group effects | R2a prerequisite complete ([PR #244](https://github.com/pineforge-4pass/pineforge-engine/pull/244)); resting lifecycle implemented, validation and acceptance pending |
| R3 | Accounting, admission, risk and observation ownership | Open |
| R3a | Reversal execution and lifecycle settlement | Complete, [PR #242](https://github.com/pineforge-4pass/pineforge-engine/pull/242) |
| R3b | Migrate remaining physical fill paths to shared settlement | Open |
Expand All @@ -26,56 +26,76 @@ These are the nine phases in the campaign ledger, including the queued
return to parity improvement. No R2 design document or partial implementation
marks R2 complete.

## Last accepted implementation
## Accepted scoped-settlement prerequisite

PR #243 was squash merged as
`5f3299d90a02e3161659720fb8c2e82f7543ef7a`. Its tree
`770f7bd2913c7c9c579d38f5c3ffc8950676bb40` is identical to the independently
reviewed and measured candidate `ce995a2db1c3bd771369f105fcce2d1234d107b8`.
PR #244 completed **R2a scoped native settlement**, the internal prerequisite
for owner-bound closes. It was squash merged as
`e7d023dbdff1c98229155ec5bcdd1e4ac534f5fb`. Its tree
`0201bf052429490fb453bbfd6037e5afd1669626` is identical to the independently
reviewed and measured candidate `794d1f27bf46f23be238f8cdf9173d9a132d3b6b`.
The scope selects all surviving fragments of an opening request in its exact
position cycle, preserving siblings, historical entry costs and one current
execution fee. Existing whole-book settlement remains available.

- Linux Debug with ASan/UBSan: 317/317 CTests, no skips; direct WebSocket
check passed. The native proof includes 46 structured scenarios and actual
runner delivery failure, retry and replay.
- Fixed-population Cloud sweep: 72/72 cases and 4190/4190 scored probes,
with zero differences in full grades, raw trades, substantive verifier
results or non-product inputs against the pinned baseline.
- Fresh local Release build: 309/309 CTests. The scoped-close suite has
12 numerical cases and 3319 checks; additive and existing ABI checks passed.
- Completed fixed-population Cloud retry: 72/72 cases and 4190/4190 scored
probes, with zero full-grade, raw-trade, substantive-verifier, coverage or
configuration differences against accepted R1. The first 71/72 attempt
failed during one Cloud task's startup and was abandoned; attempt 2
completed without a code change.
- All nine PR checks and four post-merge workflows passed. Independent
final candidate review was GREEN.
- The actual gate execution `pineforge-pr-gate-7zlr7` returned
- The actual gate execution `pineforge-pr-gate-m6j9h` returned
**FAIL: `target.not-positive` only**, because results did not improve.
This is retained as a FAIL under the authorized neutral-refactor exception.
The campaign baseline remains unchanged; promotion was skipped.

These results apply to the R1 candidate, not to future R2 edits. The final
receipt is content-addressed in the campaign evidence store as
`7c4487fa35366a2a839f8663223f5f2085e8228ccfae7e772c52acf7f9ff996c`.

## Next increments

1. **Scoped physical closes.** Extend the existing inspection/settlement
path to close only surviving exposure from a particular opening request
in an exact position cycle. Whole-book FIFO remains the default.
This preparatory seam needs its own contract, native checks and review.
2. **Resting-order chronology and partial execution.** Add typed triggers
and live remainders to the existing driver/consumer path. Distinguish
actual tick prints from modeled OHLC crossings; protect limit prices.
3. **Causal relationships.** Activate owner-bound children and apply typed
group cancellation/reduction from committed events, with replacement,
exhaustion and replay behavior explicit.
4. **Integration proof.** Exercise the real native producer/consumer and
review the Pine mapping independently before accepting R2.

The first relationship proposal is not accepted. Its review found ambiguous
tick interpolation, remaining-path ordering, same-point child activation,
and missing nonterminal partial executions. These decisions must be resolved
before relationship implementation. Opening-request identity may cover
several physical fragments; it is not a unique lot id. Canceling a working
remainder does not undo its already committed exposure.
These results apply to R2a. They do not establish acceptance of the later
resting-order implementation, and R2a is not an additional completed roadmap
phase.

## Current R2 implementation

This branch implements the pinned native contract. Validation and review
repairs are still in progress:

1. Limit, stop, stop-limit and trailing requests retain trigger state and
partial remainders. Matching distinguishes actual tick prints from
continuous modeled OHLC crossings, protects limit prices, and applies
explicit request capacity at each original input point.
2. Owner-bound children use committed opening events and cycle identity.
Typed group cancellation/reduction follows committed execution effects;
replacement and exhaustion finish their dependency cleanup before a
command returns. One opening request may own several physical fragments.
Canceling its working remainder does not undo committed exposure.
3. Request/core/event values use `native_order_v2`; identity values remain
`native_order_v1`. Engine, pending, host and consumer C++ boundaries move
together to epoch 13, with broker/stream hash version 13. Existing native
run-spec, calendar/driver value domains and C ABI 4 prefixes remain.

The local Release suite and separate native acceptance checks pass. One
WebSocket transport test is skipped when the selected macOS libcurl lacks
WebSocket support. Native coverage exercises working-order quantities,
scoped ownership and costs, group effects, driver chronology, admission,
replay/reset, and failure prefixes. Independent review findings have
corresponding regression witnesses, including token ownership, event-sized
partial exits, exact deferred-receipt arithmetic and same-cursor execution prices.
The sanitizer ABI control also supplies the historical v12 destructor needed
for its RTTI; this changes only the link-test stub, not runtime behavior.

Final review of the publication candidate, remaining R2 acceptance coverage,
a new fixed-population 4190 comparison, the actual gate and published-head CI
remain required. R2 is not complete. The earlier R2a and separate AUTO-isolation
compatibility results do not substitute for this implementation's proof.

R2 preserves all currently supported calendar, session, timezone, DST and
timeframe behavior. Source labels remain inert in the native kernel;
`from_entry`, source call grouping, Pine OCA policy and dormant revival belong
to codegen/adapters. Range-end reporting remains nonphysical.
The native R2 boundary is pinned; complete Pine lowering remains mandatory
R4 work. The parity-improvement campaign remains paused through the final
native audit.

## Publication and acceptance

Expand Down
39 changes: 25 additions & 14 deletions docs/pages/abi-stability.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,14 +112,20 @@ notice:
- The shape of internal log lines (use them for humans, not parsers).

Rebuild generated and native C++ objects against matching engine headers and
runtime. Deriving placement and opposite-predecessor facts from original
admission evidence changes the native layout and advances `PendingOrder` and
`BacktestEngine` to `engine_script_run_v12`. Exact pre-change fd4c686/v10
headers are authenticated before native, generated-style and standalone
PendingOrder pairing checks. Current/old matching links must succeed and stale
pairings must fail for the expected qualified symbols. Earlier v2-v9 controls
remain; every translation unit must compile before any mismatch is accepted.
No pairing executable runs.
runtime. Resting native request values and expanded host event snapshots
advance `PendingOrder`, `BacktestEngine`, `NativeStrategyHost`, and the private
native consumer to `engine_script_run_v13`. Exact pre-change e7d023d/v12
headers (and the e7d023d `native_order_v1` closure) are authenticated before
native, generated-style, standalone PendingOrder, and return-only
`native_events()` pairing checks. A v12 host that only calls `native_events()`
cannot silently decode a later `CommandEvent` layout: the method lives on the
v13 class even if `sizeof(NativeStrategyHost)` is unchanged. Allocation-free
`NativeFailureContext` (`NativeInRunCause` / `NativeInRunRecipient` /
`NativeInRunCursor`) is part of that same epoch13 host record; it does not
open a C ABI or new engine epoch. Current/old
matching links must succeed and stale pairings must fail for the expected
qualified symbols. Earlier v2–v10 and v12 controls remain; every translation
unit must compile before any mismatch is accepted. No pairing executable runs.

New standalone lifecycle values and `Lifecycle` own the inline namespace
`pineforge::exit_legs::lifecycle_v1`; new admission values, `Draft`, `Journal`
Expand Down Expand Up @@ -159,13 +165,17 @@ its creating strategy module. A fully self-contained old module can still use
its own matching runtime; this check does not turn it into a v11 module.

The integrated representation advances the broker fingerprint domain to
`pineforge-broker-state/v11` and stream fingerprint version to 11. The original
admission observation and prior-book direction are hashed as canonical facts;
the three derived placement views add no redundant folds. Lifecycle definitions, generations, obligations and replay
`pineforge-broker-state/v13` and stream fingerprint version to 13. Native
consumer identity is `native-consumer/v4`; driver v3 shapes, `close_scope_v1`,
and `native_run_spec_v1` stay. Stable `RunIdentity` / `RequestHandle` / `Birth`
remain `native_order_v1`; new request, core, and event values own
`native_order_v2`. The original admission observation and prior-book direction
are hashed as canonical facts; the three derived placement views add no
redundant folds. Lifecycle definitions, generations, obligations and replay
receipts, plus causal journal state remain represented. Existing
reservation, Pine instruction, activation, quantity, predecessor and birth facts
remain represented. The Pine component schema remains 1; it is
independent of the aggregate fingerprint version. Prior v2-v10 fingerprints are
independent of the aggregate fingerprint version. Prior v2–v12 fingerprints are
not comparable. Fingerprints are replay checks, not serialized checkpoints or
complete hashes of private strategy state. The native runner already binds
its strategy-library SHA; its ledger format and Python provenance fingerprints
Expand All @@ -174,8 +184,9 @@ are separate contracts and do not change here.
Determinism is conditional on identical externally supplied market, intent and
fill-report sequences, configuration, code and version. It is not a claim that
live execution prices, quantities or callback arrival are predictable. The
version bump changes linkage and fingerprint bytes; it changes no financial
rule, fill price, fee, quota policy or economic test expectation.
epoch change itself changes linkage and fingerprint identity. The accompanying
native execution behavior is described in [Native engine](native-engine.md);
the ABI boundary alone is not a claim that every execution path is unchanged.

If you find yourself reaching for any of these from outside the closed
PineForge transpiler, you're holding it wrong — file an issue and we'll
Expand Down
75 changes: 52 additions & 23 deletions docs/pages/native-engine.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,19 @@
# Native market engine {#native_engine}
# Native engine {#native_engine}

@tableofcontents

Hand-written C++ strategies can run a **standalone native market-order** path:
one `NativeRunSpec`, one working request roster, one physical lot book, and
Hand-written C++ strategies can run a **standalone native** path: one
`NativeRunSpec`, one working request roster, one physical lot book, and
close-only callbacks. Pine `strategy.*` commands, cap/priority adapters, default
source sizing, and complete Pine policy extraction are **not** this surface.
Codegen and source adapters select those policies separately. Resting
limit/stop/bracket relationships are a later roadmap; do not infer them from
this slice.
Codegen and source adapters select those policies separately. Resting requests
use the general host commands; request-value members live in
`<pineforge/native_order.hpp>` and are not restated here.

The resting-order lifecycle is implemented in the current R2 candidate;
final merge acceptance, compatibility and CI verification are still
pending. [Refactor progress](../native-refactor-progress.md) separates this
candidate from the completed R2a scoped-settlement prerequisite.

Subclass `pineforge::NativeStrategyHost`. Configure with `configure_native`,
then `run` or `stream_*`. Submit from native begin/bar callbacks, or between
Expand Down Expand Up @@ -66,8 +71,13 @@ host high-water.
events, ordinals, floor, or identity at the handoff.

`last_error()` is presentation text. `native_state().failure` is the durable
record (`code`, `operation`, optional `ordinal`, `discriminator`). See
`NativeFailureCode` / `NativeFailureOperation` in `native_host.hpp`.
record (`code`, `operation`, optional `ordinal`, `discriminator`, and
allocation-free `context`). Cause/recipient/cursor facts use
`NativeInRunCause` / `NativeInRunRecipient` / `NativeInRunCursor` selected by
`NativeFailureContextKind`; identifiers belong to the failed spec's
`RunIdentity` (`native_failure_context_in_run`, `native_failed_run_identity`).
A foreign run is dropped, not relabeled. Failure copy/move does not allocate.
See `NativeFailureCode` / `NativeFailureOperation` in `native_host.hpp`.

## NativeRunSpec

Expand Down Expand Up @@ -120,27 +130,41 @@ byte-identical** to the spec. Conflicting values are a preflight refusal:
an unsupported source mutation (`Failed`). Magnifier/source-feed arguments are
not native spec fields.

## Market requests
## Native requests

From a native callback in `Batch` / `Warmup` / `Realtime`:

```cpp
submit(request);
replace(handle, request);
cancel(handle);
submit_market(request);
replace_market(handle, request);
cancel(handle);
```

`submit` / `replace` are the complete host commands. `submit_market` /
`replace_market` keep the existing market-only call sites and must reject
nondefault trigger, capacity, owner, or group extras rather than drop them.
Serialized external C++ calls may command only **between realtime inputs**,
never reentrantly during input processing. There is no C request API in this
slice.

`native_order::Request` is `{ Action, label, comment }`. Label/comment are
inert text. `Action` is:
`native_order::Request` values belong to `native_order_v2`; identity types stay
`native_order_v1`. Label/comment remain inert text. The market default path
still constructs from:

- `order_action::Transact{signed_units}` — finite nonzero
- `order_action::Reduce{units}` — finite positive
- `native_order::Reduce{native_order::ExplicitUnits{units}}` — finite positive
- `execution::Flatten{}` — quantity-free whole-book close

General requests also select typed `Market`, `Limit`, `Stop`, `StopLimit` or
`Trail` triggers; `ImmediateRemaining` or per-point `PointBudget` capacity;
and explicit owner/group relationships. Partial executions retain live
remaining quantity. Owner-bound closes use committed opening exposure and
cycle identity through scoped settlement. Group cancellation/reduction is
caused by committed execution events. See the request header for the exact
value types; source-specific Pine lowering remains codegen/adapter work.

A `quantity_grid`, when present, admits Transact/Reduce quantities on the
exact binary64 grid in `native_order.hpp`. Flatten is not gridded. Rejection
does not rewrite the attempted bits.
Expand All @@ -153,8 +177,8 @@ does not rewrite the attempted bits.
- `Rejected` — rejection ordinal, no handle (`InvalidQuantity` / `OffGrid`).

Fills appear later as `ExecutionAppliedEvent` on the same command history
(`native_events(0)`). Matching walks live requests in acceptance/incarnation
order at a matching driver point. Eligibility is
(`native_events(0)`). Default market requests retain acceptance/incarnation
priority at an eligible matching driver point. Their birth eligibility is
`point_ordinal > birth.acceptance_ordinal` and
`effective_time_ms >= birth.decision_time_lower_bound` (`point_eligible` in
the header). A request accepted on bar *N* cannot fill on that bar’s already
Expand All @@ -167,8 +191,8 @@ absent, replaced, or already terminal → `NotWorking`. Foreign or malformed
handle → `InvalidHandle`. Every outcome is an event. Commands never move
lots or cash.

At a matching point, Reduce/Flatten use **current** exposure, not the
acceptance-cycle book: a flatten accepted while flat can still close a
For independent market requests, Reduce/Flatten use **current** exposure,
not the acceptance-cycle book: a flatten accepted while flat can still close a
same-point opening that already filled. If the book is flat at execution,
Reduce/Flatten terminalize `NoEffectEvent`: no execution identity, no fill,
no fee, no physical action. `MatchRejectedEvent` is an event ordinal without
Expand Down Expand Up @@ -265,14 +289,20 @@ completed run with no events.
Two driver models only: confirmed OHLCV and observed ticks. Mixing them on
one stream is refused. Callbacks stay **close-only** (script-bar calculation).

Confirmed script OHLC: modeled **Opening** at the first contributor’s actual
open (**match**), high/low in the existing AUTO order (**excursion only**),
close (**excursion only**), then calculation, then optional
AfterCalculation close (**match**).
Confirmed OHLC retains the modeled **Opening**, high/low in the existing
AUTO order, close, calculation and optional AfterCalculation close sequence.
Default market requests match at the modeled opening and eligible
AfterCalculation close; high/low/close traversal updates their excursions.
Resting requests additionally match along the remaining continuous modeled
price segments. The earliest eligible hit is resolved before later hits;
trigger activation, partial fills and newly eligible relationships retain
their causal cursor. Geometric fractions keep the original point's time
and identity rather than inventing intermediate timestamps.

Observed tick: the real price at its timestamp/sequence (**match** /
excursion). Sequence, when nonzero, must increase. Off-session **observed**
prints are still delivered and may fill an eligible live request.
prints are still delivered and may fill an eligible live request. Resting
triggers evaluate each actual print without interpolating between prints.

Quiet **tradable** interval with no prints: explicit **carried open** at that
interval’s tradable open, last known price (**match** if eligible), then
Expand Down Expand Up @@ -303,7 +333,6 @@ These are existing refusals, not implied future features:
entry/exit/cancel commands — native hosts latch `Failed`
(`UnsupportedSource`) before mutation
- C-level native request submit/replace/cancel
- Limit, stop, trail, or bracket **relationships**

Rebuild strategy libraries against this engine. An ABI-v4 module without the
native contract is legacy and cannot take `--native-config`.
Expand Down
2 changes: 1 addition & 1 deletion include/pineforge/compat/pine/exit_activation.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

namespace pineforge {
enum class PositionSide;
inline namespace engine_script_run_v12 { struct PendingOrder; }
inline namespace engine_script_run_v13 { struct PendingOrder; }
}
namespace pineforge::compat::pine {

Expand Down
Loading
Loading