Skip to content

Skip or cancel the queued cross-zero second part once its order is canceled - #10214

Open
quantconnect-kronos[bot] wants to merge 2 commits into
masterfrom
kronos/bug-10210-cross-zero-cancel-race
Open

quantconnect-kronos[bot] wants to merge 2 commits into
masterfrom
kronos/bug-10210-cross-zero-cancel-race

Conversation

@quantconnect-kronos

@quantconnect-kronos quantconnect-kronos Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes #10210

What changed

When the first part of a cross-zero order fills, Brokerage.TryHandleRemainingCrossZeroOrder emits PartiallyFilled and places the second part from a Task.Run. A cancel that targets the already filled first part can be acknowledged by the brokerage as Canceled (for example a soft reject with "Not an open order."). On master that closes the order in Lean while the second part is still placed and keeps working.

Brokerage.OnOrderEvents now checks every Canceled or Invalid event against the order's second part:

  • Second part still queued: the task skips placing it.
  • Second part being placed: once placement returns, its brokerage id is registered and a cancel for it is requested.
  • Second part already placed and still open: a cancel for it is requested. This covers a cancel sent against the first part's id before the second part's id was added, but acknowledged after placement returned.

The cancel is requested once per second-part brokerage id, through CancelOrder with the order whose last brokerage id is the second part's. If it returns false or throws, a CrossZeroCancelFailed warning is sent. The second part counts as open while it's in LeanOrderByZeroCrossBrokerageOrderId. Brokerages remove it there through TryGetOrRemoveCrossZeroOrder before emitting its own fill or cancel event, so a cancel that targeted the second part itself doesn't trigger another.

The cancel is requested from a Task.Run, never from the thread raising the event. The event can be raised while the brokerage holds its stream lock, which its CancelOrder takes again. The queued-order marking is a lock-free ConcurrentDictionary update.

Orders with no cancel during the cross-zero lifecycle behave as before. This changes no public API. Alpaca, Tastytrade, Tradier and TradeStation all place the second part through this shared helper, so the fix applies to all four without changes in their repositories.

Tests

New tests in OrderCrossingBrokerageTests. The phony brokerage now holds a placement lock while it places or cancels, and its cancel reads the brokerage id before taking that lock:

  • CancelAcknowledgedBeforeQueuedSecondPartIsNotSubmitted: a cancel is acknowledged when the PartiallyFilled event arrives, before the queued task runs. Expected result: the second part is never placed, the order stays Canceled, and no cross-zero brokerage id is left tracked.
  • CancelAcknowledgedWhileSecondPartIsSubmittedCancelsTheSecondPart: a cancel is acknowledged while the second part is being placed. Expected result: a cancel is then sent for the second part's brokerage id.
  • CancelRequestedWhileSecondPartIsSubmittedAndAcknowledgedAfterCancelsTheSecondPart: a cancel is requested on another thread while the second part is being placed. It targets the first part's id and is acknowledged after placement returns. Expected result: a cancel is then sent for the second part's brokerage id.

All three fail on master. The third also fails on this pull request's first commit. All pass with this change.

  • OrderCrossingBrokerageTests: 19/19 passed.
  • Full suite, built and filtered the way CI runs it (dotnet build /p:Configuration=Release QuantConnect.Lean.sln, then dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "TestCategory!=TravisExclude&TestCategory!=ResearchRegressionTests"): Passed 39848, Failed 0, Skipped 100.

Tested end to end against the TradeStation brokerage built on this change, using the issue's schedule. Holding +5 AAPL, a sell of 10 sends a 5-share closing part, and its FLL frame arrives. While the second part is being placed, a cancel is requested from another thread. It targets the first part's id and is soft-rejected with Not an open order. after placement returned, which closes the order as Canceled.

  • On master and on this pull request's first commit, the second part kept working.
  • With this change, the second part is canceled by its own brokerage id.
  • When the cancel lands before submission, the second part is not placed.

TradeStation enables concurrent order submission, so its stream lock doesn't serialize the cancel behind the second part's placement. The cancel is sent while placement is in flight, and the race is between the two HTTP requests.

Known limitation

When the second part is canceled this way, the brokerage may report one more Canceled event for an order Lean already shows as Canceled. Tradier's CancelOrder emits Canceled itself without going through TryGetOrRemoveCrossZeroOrder. So a regular cancel of a placed second part on Tradier can be followed by one redundant cancel request, which Tradier rejects because the order is already closed.

…nceled

After the first part of a cross-zero order fills, its second part is placed
from a queued task. A cancel acknowledged in between, against the filled first
part's brokerage id, closed the Lean order while the task still placed the
second part, leaving a new opening order working at the brokerage.

Canceled and invalid order events now mark the queued second part: the task
skips placing it when the order closed before submission, and cancels it when
the order closed while it was being placed.

Closes #10210

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:working develop: in progress kronos:review Request review labels Nov 20, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 20, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: when a cancel is requested while the second part is being placed, the second part can still be left working while Lean shows the order Canceled. On TradeStation, the brokerage the issue reproduces, that is what happens every time.

The tracking ends before a cancel that targets the first part can be acknowledged. The task drops the order's entry right after PlaceCrossZeroOrder returns (Brokerage.cs:967). A cancel that read the order's brokerage ids before the second part's id was added still targets the filled first part. If its Canceled arrives after that point, nothing marks the order, so the second part keeps working while the order shows Canceled. That's the result #10210 asks to prevent ("cancellation must account for that order rather than closing Lean based solely on the completed first-half ID").

On TradeStation this isn't a narrow timing window, it's what always happens:

  • TradeStationBrokerage.PlaceCrossZeroOrder places the second part inside _messageHandler.WithLockedStream.
  • CancelOrder reads order.BrokerId.Last() (the first part's id), then waits on that same lock.
  • So a cancel requested during placement is only sent after placement returns. It's soft-rejected with Not an open order. an HTTP round trip later, well after the entry is gone.

Alpaca and Tastytrade don't hold a lock while placing, but a cancel sent against the first part's id just before placement returns ends the same way.

I reproduced this locally with OrderCrossingBrokerageTests' PhonyBrokerage. I added a lock that the second part's placement holds, the way TradeStation's does. CancelOrder reads its id first, then takes the lock and waits 50 ms before emitting Canceled. The cancel is started on another thread from PlacingCrossZeroSecondPart. Result: both parts placed, only the first part's id canceled, order Canceled. The new CancelAcknowledgedWhileSecondPartIsSubmittedCancelsTheSecondPart passes only because the phony CancelOrder runs synchronously inside the placement, which TradeStation's locking can't do.

The cause is the same for every brokerage: a cancel can be acknowledged against the first part's id after the second part is placed. Whatever closes this needs a test where the cancel is requested on another thread during placement and acknowledged after it returns. Possible directions, your call:

  • Keep the order's entry until a cancel can no longer target the first part's id.
  • Have the cancel path pick its target id once the second part's placement has finished, for example TradeStation resolving the id inside the stream lock.

The end-to-end run in the description should also request the cancel while the second part is in flight, the way an algorithm would, not from inside the placement.

The skip-before-submission path looks right. It also helps Alpaca, where a cancel of the filled first part emits Invalid, which now stops the second part from being placed.

I set risk:high. The change touches thread coordination in the shared order-placement path used by four live brokerages, and the case above is still open.

Comment thread Brokerages/Brokerage.cs Resolved
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:high High risk and removed kronos:review:working review: in progress labels Nov 20, 2026
…t is acknowledged late

A cancel can read the filled first part's brokerage id before the second part's
id is added and be acknowledged only after the second part was placed. Any
canceled or invalid event for a Lean order whose second part is still open now
requests, once, the cancellation of that second part.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Contributor Author

Thanks. The case is real, and 9bd9299 fixes it in the shared Brokerage. When a Canceled or Invalid event arrives for a Lean order whose second part is placed and still open, the second part is now canceled by its own brokerage id. This also covers a cancel acknowledged against the first part's id after placement returned. The cancel is requested once per second-part id, off the thread raising the event.

One correction on TradeStation: it sets ConcurrencyEnabled => true, so WithLockedStream takes the shared side of the lock for both placing and canceling. The cancel therefore doesn't wait for the second part's placement; it goes out while placement is in flight. The race is the one you describe for Alpaca and Tastytrade: the cancel's reply arrives after placement returns.

  • Unit test: added CancelRequestedWhileSecondPartIsSubmittedAndAcknowledgedAfterCancelsTheSecondPart, built the way you described. It fails on 828ccd9 and passes now. OrderCrossingBrokerageTests 19/19, full suite: Passed 39848, Failed 0, Skipped 100.
  • End to end: re-tested against TradeStation. The cancel is requested from another thread while the second part is being placed and soft-rejected after placement returns. On master and on 828ccd9 the second part kept working. Now it's canceled. The description is updated.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 20, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 20, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review November 20, 2026 14:51

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. 9bd9299 fixes the case from my earlier review. A cancel that targets the filled first part and is acknowledged after the second part was placed now cancels the second part by its own brokerage id. The hand-off between the event and the submitting task is atomic: OnOrderEvents flips the flag with TryUpdate(false → true), and the task reads and removes it with TryRemove after the lock, once the second part's id is already in LeanOrderByZeroCrossBrokerageOrderId. Either way it ends up being canceled, whichever side of the placement the acknowledgement lands on. The automatic cancel targets the second part's id in TradeStation, Alpaca and Tastytrade, since each one's CancelOrder uses the order's last brokerage id.

I ran it locally. OrderCrossingBrokerageTests passed 19/19 in 5 runs out of 5. With Brokerage.cs from 828ccd9, CancelRequestedWhileSecondPartIsSubmittedAndAcknowledgedAfterCancelsTheSecondPart fails, as the description says.

CI never ran on 9bd9299: that commit has no workflow runs. Every run on 828ccd9 passed. Marking the pull request ready for review should start CI, and it has to be green before this merges.

The label stays risk:high. This changes thread coordination in the order path that four live brokerages share, and CI hasn't run on this commit.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: a delayed compensating cancel can overwrite a confirmed second-part fill with Canceled. The exact Lean base finishes Filled for the same schedule; this head with current TradeStation master finishes Canceled. Executed quantity is still recorded correctly. Details and the reproduction are inline.

@Martin-Molinero Decision needed: should this change require TradeStation's existing cancellation-state protection, or remain limited to preventing submission of a canceled queued part?

I verified that the already-open QuantConnect/Lean.Brokerages.TradeStation#132 (7781c40) resolves this schedule: its cancel path rereads the order under the stream lock and preserves an already-closed order. That would introduce a brokerage-version prerequisite rather than requiring another shared-API redesign.

  • A. Require the TradeStation protection first (recommended): retain the full fix, add this regression test, and ensure the compatible brokerage version is used with it.
  • B. Narrow this change to skipping queued submissions: remove the new compensating-cancel path for now. This avoids the new terminal-status regression, but leaves cancellation during or after submission unresolved.

Reply with the option you choose, or another approach.

Validation: Release build succeeded; the full CI-filtered suite passed 39,848, failed 0, skipped 100. All 19 cross-zero tests passed. Twelve additional lifecycle checks passed in ten repeated runs. Six offline TradeStation production-path checks passed, alongside the failing post-fill schedule, exact-base comparison, and successful PR132 comparison. These used controlled placement/transport, not a new simulator end-to-end run.

CI: this head has no workflow runs or check runs; CI is missing for the commit authored at 2026-11-20 14:34 UTC.

Other implementations: Alpaca and Tastytrade remove second-part tracking before normal terminal notifications. Tradier's existing first-filled-ID cancellation failure is separate and is now tracked in QuantConnect/Lean.Brokerages.Tradier#62; it does not block this change.

Risk: risk:high — the new asynchronous cancellation can replace a confirmed fill's terminal status.

Comment thread Brokerages/Brokerage.cs
#pragma warning disable CA1031 // Do not catch general exception types
try
{
isCanceled = CancelOrder(leanOrder);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mapping can disappear after this task is queued, but the captured order is still canceled unconditionally. I reproduced this through TradeStation's production stream handler and CancelOrder, controlling only placement/transport and pausing the automatic invocation before it enters CancelOrder:

  1. A cancel of the filled first ID emits Canceled and queues this task.
  2. The second ID's FLL arrives before the task proceeds: Lean receives Filled, and TryGetOrRemoveCrossZeroOrder removes the mapping.
  3. The queued task nevertheless cancels that completed second ID. TradeStation returns Not an open order., and its current soft-reject path emits another Canceled.

This head produces PartiallyFilled, Canceled, Filled, Canceled, final status Canceled, total fill −10. With the exact base Brokerage.cs, it produces PartiallyFilled, Canceled, Filled, final status Filled, the same fill quantity. BrokerageTransactionHandler accepts the final cancellation and overwrites Filled (BrokerageTransactionHandler.cs:1316–1318), so this is not just a test-provider artifact. Please cover the second part completing before the compensating cancellation executes, and preserve that confirmed fill. The bounded existing prerequisite is described in the review body.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the base never sends this cancel, so a second part that fills before the compensating cancel lands ends Filled there and Canceled here. That comes from this change.

Checking LeanOrderByZeroCrossBrokerageOrderId again just before CancelOrder would cover your exact schedule, where the fill is processed before the task runs. It wouldn't cover a fill that arrives while the cancel request is in flight. Keeping that fill needs either TradeStation's cancel to leave an already closed order alone (QuantConnect/Lean.Brokerages.TradeStation#132) or dropping the compensating cancel. So I'm leaving the code as it is until the decision in your review is answered, and I'll build whichever option is chosen, with this schedule as a test.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot removed the kronos:review:working review: in progress label Nov 20, 2026
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:idle develop: paused and removed kronos:develop:working develop: in progress labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kronos:develop:idle develop: paused risk:high High risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Queued cross-zero second half can submit after cancellation is acknowledged

0 participants