Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions docs/native-refactor-progress.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Standalone native engine refactor

Status at 2026-09-12: **3 of 9 roadmap phases complete (33%)**. This counts
completed phases; it is not an estimate of elapsed work or remaining time.

The acceptance boundary is a standalone C++ backtest and forward-execution
state machine. Codegen and explicit adapters own PineScript policy. During
this refactor, unchanged compatibility results are acceptable; parity
improvement resumes after the final native audit.

## Completed and remaining work

| Phase | Scope | Status |
| --- | --- | --- |
| R0 | Native settlement and CI convergence | Complete, [PR #241](https://github.com/pineforge-4pass/pineforge-engine/pull/241) |
| R1 | Native market orders, host, drivers, calendar and forward runner | Complete, [PR #243](https://github.com/pineforge-4pass/pineforge-engine/pull/243) |
| R2 | Resting orders, partial execution, owner-bound children and group effects | Design under review; next implementation slice |
| R3 | Accounting, admission, risk and observation ownership | Open |
| R3a | Reversal execution and lifecycle settlement | Complete, [PR #242](https://github.com/pineforge-4pass/pineforge-engine/pull/242) |
| R3b | Migrate remaining physical fill paths to shared settlement | Open |
| R4 | Complete codegen/adapter policy ownership and native independence | Open |
| R5 | Final requirement and compatibility audit | Open |
| R6 | Resume the parity improvement campaign after the audit | Queued |

These are the nine phases in the campaign ledger, including the queued
return to parity improvement. No R2 design document or partial implementation
marks R2 complete.

## Last accepted implementation

PR #243 was squash merged as
`5f3299d90a02e3161659720fb8c2e82f7543ef7a`. Its tree
`770f7bd2913c7c9c579d38f5c3ffc8950676bb40` is identical to the independently
reviewed and measured candidate `ce995a2db1c3bd771369f105fcce2d1234d107b8`.

- Linux Debug with ASan/UBSan: 317/317 CTests, no skips; direct WebSocket
check passed. The native proof includes 46 structured scenarios and actual
runner delivery failure, retry and replay.
- Fixed-population Cloud sweep: 72/72 cases and 4190/4190 scored probes,
with zero differences in full grades, raw trades, substantive verifier
results or non-product inputs against the pinned baseline.
- All nine PR checks and four post-merge workflows passed. Independent
final candidate review was GREEN.
- The actual gate execution `pineforge-pr-gate-7zlr7` returned
**FAIL: `target.not-positive` only**, because results did not improve.
This is retained as a FAIL under the authorized neutral-refactor exception.
The campaign baseline remains unchanged; promotion was skipped.

These results apply to the R1 candidate, not to future R2 edits. The final
receipt is content-addressed in the campaign evidence store as
`7c4487fa35366a2a839f8663223f5f2085e8228ccfae7e772c52acf7f9ff996c`.

## Next increments

1. **Scoped physical closes.** Extend the existing inspection/settlement
path to close only surviving exposure from a particular opening request
in an exact position cycle. Whole-book FIFO remains the default.
This preparatory seam needs its own contract, native checks and review.
2. **Resting-order chronology and partial execution.** Add typed triggers
and live remainders to the existing driver/consumer path. Distinguish
actual tick prints from modeled OHLC crossings; protect limit prices.
3. **Causal relationships.** Activate owner-bound children and apply typed
group cancellation/reduction from committed events, with replacement,
exhaustion and replay behavior explicit.
4. **Integration proof.** Exercise the real native producer/consumer and
review the Pine mapping independently before accepting R2.

The first relationship proposal is not accepted. Its review found ambiguous
tick interpolation, remaining-path ordering, same-point child activation,
and missing nonterminal partial executions. These decisions must be resolved
before relationship implementation. Opening-request identity may cover
several physical fragments; it is not a unique lot id. Canceling a working
remainder does not undo its already committed exposure.

R2 preserves all currently supported calendar, session, timezone, DST and
timeframe behavior. Source labels remain inert in the native kernel;
`from_entry`, source call grouping, Pine OCA policy and dormant revival belong
to codegen/adapters. Range-end reporting remains nonphysical.

## Publication and acceptance

Publish reviewed increments in an evolving draft PR so implementation and
open decisions are reviewable. Draft and CI-first publication may precede
the full compatibility sweep. Before merge, require native proof, unchanged
fixed-population compatibility evidence, the actual gate result, independent
review of the final candidate, and green CI. Squash merge and verify the
resulting tree and post-merge checks separately. Never translate a neutral
gate failure into PASS or force baseline promotion.
85 changes: 85 additions & 0 deletions docs/native-scoped-close.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Scoped native settlement

The trusted native settlement seam can close the surviving exposure created
by one opening request in one position cycle. It uses the engine's existing
physical book, FIFO allocation and execution fee accounting.

This is a prerequisite for native owner-bound orders. It does not place an
order or implement a trigger, child lifecycle, group effect or broker fill
transport. The native host's current market request API remains unchanged.

## Selector and entry points

`<pineforge/execution_close_scope.hpp>` declares the additive
`pineforge::execution::close_scope_v1` types:

```cpp
struct Book {};
struct OpeningExposure {
std::uint64_t incarnation = 0;
std::int64_t cycle = 0;
};
using CloseScope = std::variant<Book, OpeningExposure>;
```

Trusted subclasses can use these protected methods:

```cpp
execution::SettlementInspection inspect_native_settlement_scoped(
const execution::Action&, const execution::Fill&,
execution::CloseScope) const;

execution::Result settle_native_execution_scoped_at(
const execution::Action&, const execution::Fill&,
const execution::PhysicalExecutionContext&, execution::CloseScope);
```

`Book` retains whole-book Flatten, Reduce and Transact behavior. Every
original settlement and inspection method keeps its signature and selects
Book. Existing C++ aggregate layouts and C ABI exports are unchanged.

`OpeningExposure` permits Flatten and Reduce only. It selects all surviving
physical fragments with the specified opening-request incarnation, in the
engine's exact current cycle. A partially filled opening request may create
several fragments with that incarnation. Labels and the closing fill's own
incarnation do not select exposure. The trusted caller validates run identity;
the selector itself is a run-local value.

## Physical behavior

Selected Reduce closes at most the selected exposure, in FIFO order within
that selection. Selected Flatten removes every selected fragment. Unrelated
lots retain their identity, order, quantities and historical costs. The cycle
ends only when the whole physical book becomes empty.

The caller supplies an already resolved fill price. Inspection quotes one
execution ticket and reports the surviving whole-book quantity, lot count
and notional. The caller may lock that ticket in `Fill::commission_account`
before settlement. Current costs are allocated across the selected close
rows; paid entry costs remain historical and are divided proportionally
between closed quantities and survivors.

Inspection does not authorize a later allocation. Settlement revalidates
the selector against the current book and recomputes allocation. No roster
index, callback, borrowed selector or saved plan is retained.

## Refusals and failure

Validation first checks finite price, explicit fee, quantity and book
integrity in the existing order. It then checks selected action and target
validity, before zero-effect handling. A zero/absent incarnation, stale or
nonpositive cycle, flat selected book, or selected Transact returns
`InvalidCloseTarget`. Invalid targets never fall back to Book.

A zero Reduce on a valid target returns NoEffect; a nonzero explicit charge
on that no-effect call returns InvalidAccounting. For example, a NaN price
with an invalid selector returns InvalidPrice, while an absent owner with
Reduce zero and a finite nonzero fee returns InvalidCloseTarget.

Checked refusals produce no rows or account effects. A selected settlement
cannot report Applied with zero closed units. Exceptions during commitment
retain the existing abort/discard-and-replay contract; this seam does not
promise rollback or in-place retry.

See [refactor progress](native-refactor-progress.md) for the remaining
native order and adapter work.
16 changes: 16 additions & 0 deletions include/pineforge/engine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
#include "broker_events.hpp"
#include "quantity_intent.hpp"
#include "execution.hpp"
#include "execution_close_scope.hpp"
#include "market_admission.hpp"
#include "reservation_expansion.hpp"
#include "order_cancellation.hpp"
Expand Down Expand Up @@ -1829,6 +1830,16 @@ class BacktestEngine {
const execution::PhysicalExecutionContext& context);
execution::SettlementInspection inspect_native_settlement(
const execution::Action& action, const execution::Fill& fill) const;
// Synchronous selected-close extensions. Book preserves the original
// action semantics; OpeningExposure permits only Flatten/Reduce and is
// revalidated against the current physical book on every call.
execution::SettlementInspection inspect_native_settlement_scoped(
const execution::Action& action, const execution::Fill& fill,
execution::CloseScope scope) const;
execution::Result settle_native_execution_scoped_at(
const execution::Action& action, const execution::Fill& fill,
const execution::PhysicalExecutionContext& context,
execution::CloseScope scope);
// Native account value: realized balance plus marked physical lots minus
// their remaining paid entry costs, for every fee type. No Pine sizing or
// end-of-range reporting convention participates in this value.
Expand Down Expand Up @@ -3881,6 +3892,11 @@ class BacktestEngine {
double calc_qty_for_type(double fill_price, double qty_value, int qty_type) const;

private:
execution::Result settle_with_context_scoped(
const execution::Action& action, const execution::Fill& fill,
const execution::LifecycleEffects& lifecycle,
const execution::PhysicalExecutionContext& context,
execution::CloseScope scope);
enum class PositionReductionCause {
SCRIPT_ORDER, // strategy.close / close_all / market exit / reversal
BRACKET_EXIT, // a strategy.exit bracket leg fill
Expand Down
5 changes: 4 additions & 1 deletion include/pineforge/execution.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,10 @@ enum class Status {
UnrepresentableQuantity, InvalidAccounting,
// Caller-supplied lifecycle targets/revisions/operations that cannot be
// applied to the current pending book. Not durable engine state.
InvalidLifecycle
InvalidLifecycle,
// Invalid or unavailable run-local opening exposure on a scoped close.
// Existing whole-book entry points do not return this status.
InvalidCloseTarget = 8
};

struct Result {
Expand Down
22 changes: 22 additions & 0 deletions include/pineforge/execution_close_scope.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#pragma once

#include <cstdint>
#include <variant>

namespace pineforge::execution {
inline namespace close_scope_v1 {

struct Book {};

// Run-local opening provenance, not a unique individual fill-lot identity.
// The trusted caller owns run identity; settlement checks the current cycle
// and selects every surviving physical fragment of this incarnation.
struct OpeningExposure {
std::uint64_t incarnation = 0;
std::int64_t cycle = 0;
};

using CloseScope = std::variant<Book, OpeningExposure>;

} // inline namespace close_scope_v1
} // namespace pineforge::execution
Loading
Loading