Repository navigation
Update IndicatorHistory with the bar type its registered indicator consolidates - #10205
quantconnect-kronos[bot] wants to merge 5 commits into
Conversation
…nsolidates When no selector implies a data type, IndicatorHistory fed each symbol the slice's default value, which can be a quote bar for a security with both trade and quote data, such as a future, while the registered indicator consolidates trade bars. Each symbol now gets the bar type of its subscription, which prefers trades, or of the default subscription types when it has none, and an option indicator's contracts get quote bars, as registering the indicator does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for the clear write-up and the issue's reproduction as a test. The futures case is fixed, but the change breaks histories that work on master. I'm requesting changes.
Tick and custom-data histories now produce no indicator values (blocking). The type each symbol gets comes from its subscription, or from the security type's defaults when it has none. Both of those resolve to TradeBar or QuoteBar even when the history in hand holds other data. The symbol then gets updated only from slice.Bars/slice.QuoteBars, so data of any other type is dropped without an error. I reproduced this locally with IndicatorHistory(new SimpleMovingAverage(1), slices). Each case returns 3 rows on master and 0 on this branch:
- SPY added at minute resolution (the default), slices holding only
Ticks. Users reach this throughIndicatorHistory(sma, spy, 100, Resolution.Tick), since the history request returns ticks while the subscription's type isTradeBar. - An equity symbol that isn't subscribed, with tick slices.
- An unsubscribed
SecurityType.Basesymbol with customBaseDataslices, such as a custom-data history request withoutAddData.
The cause is that the subscription's type decides what the slice can provide, not only which type to prefer. The issue's case only comes up when a slice has both a TradeBar and a QuoteBar for a symbol. A smaller change would use the resolved type only as the tie-break in that case and keep master's default value otherwise. Ticks, custom data and single-type slices would then behave as on master. The default-value extraction is also a copy of SliceExtensions.PushThrough's null-type branch. Reshaping the change this way may let you reuse that code instead.
Tests. The new tests only cover slices with bars. Please add the cases above (ticks for a minute-subscribed symbol, and custom data without a subscription) so the fix is shown not to drop them.
Other notes
- CI is green on
ec0d77e. - The description has an end-to-end outcome (0 of 899 timestamps differing on a continuous ES future). Please re-run it after the change, and add a tick-resolution history to that run.
- No other repositories need an update: the public API signatures are unchanged.
Risk: risk:high. This changes the values a public algorithm API returns, and in its current form it drops data that master uses, so it can give users wrong (empty) results.
…story Resolving the bar type from the subscription dropped data the history held of other types, such as ticks or custom data. The resolved type now only picks between a symbol's trade and quote bars when a slice has both, and every other value is pushed as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. I've addressed it in 9f04aac:
The description has the details. |
There was a problem hiding this comment.
Thanks for the quick turnaround. 9f04aac resolves everything from my last review, so I'm approving.
Earlier findings
- Ticks and custom data were dropped: resolved. The resolved type is now used only when a slice has both a
TradeBarand aQuoteBarfor the symbol. Every other value goes throughPushThrough's default-value path, as on master. Ticks, custom data and single-type slices behave as before, and the copied extraction code is gone. - Tests: resolved.
IndicatorHistoryUsesTicks(subscribed and unsubscribed) andIndicatorHistoryUsesCustomDataWithoutSubscriptioncover the cases I reported. - End-to-end: resolved. The re-run on 9f04aac still shows 0 of 899 differing timestamps for ES, against 847 on master. The tick-resolution SPY history returns the same row count as master.
What I checked on this commit
- Moving the
dataType != nullbranch first doesn't change any path.dataTypeis only non-null from theIndicatorBase<IndicatorDataPoint>overload (GetDataTypeFromSelector), which master also sent toPushThrough(dataType). - The tie-break matches registration.
TryGetSubscriptionorders by tick type, so trades win over quotes, as inResolveConsolidator.InitializeOptionIndicatorregisters option contracts withQuoteBarand the underlying with its subscription's type, which is what the option branch does. A non-option indicator on an option contract gets trades, asRegisterIndicatorwould give it. GetDefaultSubscriptionDataTypeis a straight extraction of theWarmUpIndicatorcode, and its behavior there is unchanged.- CI: all checks pass on
9f04aac. API Tests were skipped, as usual for this repository. - No other repositories need an update: no public signatures changed.
Risk: risk:high. This is a bug fix, and it leaves single-type, tick and custom-data histories as they were. But it changes the values IndicatorHistory returns for every symbol whose history has both trade and quote bars: futures, and also equities, crypto and options at intraday resolutions. That is a public algorithm API with wide reach. The ES run shows most rows changing on master's data (847 of 899), so users comparing against earlier histories will see different numbers. A person should merge it knowing that.
There was a problem hiding this comment.
Requesting changes on 9f04aac: the default futures case is fixed, and the earlier tick/custom-data regression is resolved, but the new tie-break also overrides explicit quote inputs that work on master.
Blocking finding — preserve explicit quote inputs. An indicator registered with a quote consolidator now gets trade prices during IndicatorHistory. Custom quote selectors also receive trade bars: a wrapped Field.BidClose silently produces trade closes, and a selector casting to QuoteBar throws. These share one cause: an unresolved dataType is treated as permission to replace the caller's input with the symbol's default subscription type. The inline comment has the base-versus-head reproduction. Please preserve explicit selector/consolidator choices and add regression coverage for both.
Validation
- All 71
AlgorithmIndicatorsTestspassed locally withdotnet test Tests/QuantConnect.Tests.csproj -c Release --filter FullyQualifiedName~QuantConnect.Tests.Algorithm.AlgorithmIndicatorsTests. - CI passes on this head; API Tests were skipped.
- Tested end to end over January 2–4, 2024, with 1,349 shared ES minute timestamps: default ROC history improves from 1,279 mismatches to zero, but explicitly quote-registered ROC history regresses from zero mismatches to 1,279. The custom bid selector returns 450 rows before the change and throws after it. Tick-resolution SPY indicator history returns the same 11,123 rows before and after.
- No dependent repository or generated API update is needed; the correction belongs in the shared
IndicatorHistoryimplementation.
Risk: risk:high retained. This version changes working quote-based histories to wrong prices or exceptions. Preserving those paths and covering them with tests is necessary before approval.
The trade and quote tie-break replaced the input a caller chose: a custom selector reading quote fields got trade bars, or threw when it cast to a quote bar, and an indicator registered with a quote consolidator got trade prices. The tie-break now applies only to an indicator without a selector or registered consolidators; any other keeps the slice's default value, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. I've addressed the requested changes in da0ddd9:
Open question for a maintainer: |
There was a problem hiding this comment.
Requesting changes on da0ddd9. Explicit selectors and quote-registered indicators now keep their inputs, which resolves the other reviewer's finding. But skipping every registered indicator leaves the case the issue reports unfixed.
The registered indicator still gets quote bars (blocking). The issue asks that IndicatorHistory update "the same indicator, registered for live updates" with the type RegisterIndicator consolidates for it. indicator.Consolidators.Count > 0 sends every registered indicator back to master's default value. That covers the common paths:
IndicatorHistoryon a helper-created indicator such asROC(symbol, 1). Your own end-to-end run shows 847 of 899 values still differing from live.- Automatic warm-up:
InitializeIndicatorregisters the indicator before it callsWarmUpIndicator, which goes through this method. A helper indicator on a future is still warmed up with quote bars and then updated live with trade bars.
This has the same cause as the earlier finding. The code guesses the type from the subscription instead of reading what the indicator is actually registered with. So, to your open question: I think the wider fix is in scope, because it is what the issue asks for, and it's also simpler. For a symbol whose subscription's Consolidators holds one of indicator.Consolidators, use that subscription's type (SubscriptionManager.AddConsolidator adds the consolidator there). Fall back to today's resolution only when there's no such subscription. One rule then covers trade-registered, quote-registered and unregistered indicators, and the Consolidators.Count > 0 exclusion goes away. The quote-registered case also stops depending on the order of the slice's data. Today IndicatorHistoryKeepsQuoteBarsOfIndicatorRegisteredWithQuoteConsolidator passes only because the slices add the quote bar last. Please add a test for an indicator registered through a helper, or with a trade consolidator, getting trade values. Keep the explicit-selector rule as it is.
Description. WarmUpIndicator(symbol, indicator, resolution) also goes through this path, so a fresh indicator warmed up on a future now gets trade bars. Please list that among the behavior changes.
Risk: risk:high stays. This changes the values a public algorithm API and indicator warm-up return for every symbol that has both trade and quote bars.
Skipping every registered indicator left the reported case unfixed: an indicator created by a helper, or warmed up after its registration, still got the slice's default quote bars where it consolidates trade bars live. When a slice has both bars for a symbol, the indicator now gets the type of the subscription holding one of its consolidators, and only an unregistered indicator falls back to the type registering it would use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. I've addressed it in 506bcf5:
|
There was a problem hiding this comment.
Requesting changes on 506bcf5. Reading the subscription the indicator is registered on resolves my last finding: helper-created indicators and their automatic warm-up now get trade bars on a future, and a quote-registered indicator gets quote bars whatever the slice order. The explicit-selector rule is unchanged. One case of the new lookup still gives the wrong bar type.
An indicator that consolidates quote ticks gets trade bars (blocking). The new branch uses registeredSubscription.Type, which is Tick for a tick subscription. The final barType == typeof(QuoteBar) ? quoteBar : tradeBar then sends trade bars to an indicator registered on the quote tick subscription. I reproduced it locally: a future added at Resolution.Tick, an SMA registered with new TickQuoteBarConsolidator(TimeSpan.FromHours(1)) (it lands on the Tick/Quote subscription), then IndicatorHistory(sma, slices) over slices that hold both bars.
- This branch returns the trade closes
100, 101, 102in both slice orders. - Master returns the quote closes
200, 201, 202when the quote bar is last, which is the order real future history comes in (your end-to-end run shows master's default value is the quote bar). So this is a regression for that setup.
The subscription's TickType says which bar it consolidates for both tick and bar subscriptions (TradeBar→Trade, QuoteBar→Quote). Picking the bar by registeredSubscription.TickType == TickType.Quote fixes this case and leaves the others as they are. Please add a test for it next to IndicatorHistoryUsesQuoteBarsOfIndicatorRegisteredWithQuoteConsolidator.
What I checked on this commit
ConcurrentSet.Overlapstakes the set's lock, so readingconfig.Consolidatorsis safe against the data feed. The lookup runs once per symbol and is cached.- The Python
RegisterIndicatorpath puts the same consolidator instance (wrapper) inindicator.Consolidatorsand in the subscription, so the lookup finds it. - A trade-tick registration, the default for helpers on a tick-subscribed future, correctly gets trade bars.
- CI passes on
506bcf5. API Tests were skipped, as usual for this repository.
Risk: risk:high stays. This changes the values a public algorithm API and indicator warm-up return for every symbol whose history has both trade and quote bars.
…istory An indicator registered on a quote tick subscription, such as with a tick quote bar consolidator, got trade bars: the subscription's type is Tick, which isn't a quote bar. The bar is now picked by the subscription's tick type, which holds for bar and tick subscriptions alike. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. I've addressed it in 756b69a:
The description is updated to match. |
There was a problem hiding this comment.
Approving 756b69a. It resolves my last finding, and I found nothing new.
Quote ticks: resolved. The bar is now picked by the subscription's TickType, so an indicator registered on the quote tick subscription (such as with a TickQuoteBarConsolidator) gets quote bars. For bar subscriptions this gives the same result as the previous type check (TradeBar→Trade, QuoteBar→Quote). The unregistered fallbacks (TryGetSubscription, GetDefaultSubscriptionDataType(...).Item2) still prefer trades, as RegisterIndicator does. I checked locally: all 78 AlgorithmIndicatorsTests pass on this commit, and IndicatorHistoryUsesQuoteBarsOfIndicatorRegisteredOnQuoteTicks fails in both slice orders with 506bcf5's QCAlgorithm.Indicators.cs.
Risk: risk:high stays. This is a bug fix, but it changes the values a public algorithm API and indicator warm-up return for every symbol whose history has both trade and quote bars (futures, and intraday equities, crypto and options). A person should merge it knowing that.
There was a problem hiding this comment.
Approving. My explicit quote-input finding is resolved: custom selectors keep their original path, and registered indicators use their own subscription's TickType, including quote-tick registrations.
I reran all 78 AlgorithmIndicatorsTests; all passed. I also tested end to end over January 2–4, 2024, comparing history against snapshots of live indicator values:
- Fresh and helper-created ES ROC: 0 of 899 shared values differ for each, versus 847 before the change.
- Quote-registered ROC: 0 of 899 differ.
- Explicit bid-close selector: 0 of 900 differ, with no exception.
- Tick-resolution SPY history with a minute subscription: 11,123 rows before and after.
A separate two-symbol probe, registered for trades on one symbol and quotes on the other, also matches direct updates. No new blocking finding.
Closes #10141
What changed
IndicatorHistoryover slices, when no selector implies a data type, pushed each symbol's default slice value (slice[symbol]). For a security with both trade and quote data, such as a future, that value can be aQuoteBar, while the indicator registered for live updates consolidates trade bars, so history and live values differed.Now, when
IndicatorHistorygets no selector and the indicator takesIndicatorDataPointor an abstract input type, a slice that holds both aTradeBarand aQuoteBarfor a symbol updates that symbol with the bar of the tick type the indicator consolidates (the quote bar forTickType.Quote, else the trade bar), resolved once per symbol:Consolidatorsholds one ofindicator.Consolidators(SubscriptionManager.AddConsolidatorputs it there). This covers helper-created indicators and indicators registered with a trade or a quote consolidator, on bar or tick subscriptions (such as aTickQuoteBarConsolidatoron the quote tick subscription), whatever the slice's data order;RegisterIndicator/ResolveConsolidator(TryGetSubscription, trades over quotes), or, with no subscription, the first of the security type's default data types, ordered the same way, asWarmUpIndicatoralready does (that ordering moved intoGetDefaultSubscriptionDataType, now shared by both);InitializeOptionIndicatorregisters them.Behavior changes, for symbols whose slices hold both trade and quote bars:
IndicatorHistorywithout a selector, on a new or a registered indicator;WarmUpIndicator(symbol(s), indicator, resolution, selector)without a selector, which goes throughIndicatorHistory. That includes the automatic warm-up of helper-created indicators. A new or helper-created indicator warmed up on a future now gets trade bars, matching its live updates.Unchanged, keeping master's behavior through
PushThrough's default-value path:x => ((QuoteBar)x).Bid.Close(selectors that imply a type, such asField.BidClose, still use that type);TradeBarorQuoteBar.Review history
Tickfor a tick subscription, so an indicator on the quote tick subscription got trade bars. 756b69a picks the bar by the subscription'sTickTypeinstead.Tests
AlgorithmIndicatorsTests:IndicatorHistoryUsesTradeBarsWhenSliceAlsoHasQuoteBars(with and without the future added): the issue's reproduction, anSMAand anATR, now computed from the trade closes (100, 101, 102). Fails on master (gets the quote closes 200, 201, 202).IndicatorHistoryUsesTradeBarsOfIndicatorRegisteredByHelper(quote bar last and first in the slice): a helper-createdSMAgets the trade closes. Quote-last fails on master and da0ddd9.IndicatorHistoryUsesQuoteBarsOfIndicatorRegisteredWithQuoteConsolidator(quote bar last and first): the quote closes. Quote-first fails on master and da0ddd9; quote-last fails on 9f04aac.IndicatorHistoryUsesQuoteBarsOfIndicatorRegisteredOnQuoteTicks(quote bar last and first): anSMAregistered with aTickQuoteBarConsolidatoron a tick-subscribed future gets the quote closes. Both fail on 506bcf5.IndicatorHistoryKeepsQuoteBarsForCustomQuoteSelectors(x => ((QuoteBar)x).Bid.Close,x => Field.BidClose(x)): quote values, as on master. Fails on 9f04aac.IndicatorHistoryUsesQuoteBarsWhenSelectorRequiresThem:Field.BidClosestill uses quotes.IndicatorHistoryUsesQuoteBarsForQuoteOnlySecurities: a forex symbol still gets its quote bars.IndicatorHistoryUsesTicks(SPY subscribed at minute resolution, and an unsubscribed equity) andIndicatorHistoryUsesCustomDataWithoutSubscription(an unsubscribedSecurityType.Basesymbol): 3 rows each, as on master. They fail on ec0d77e.IndicatorUpdatedWithSymbol,IndicatorHistoryIsSupportedInPythonForOptionsIndicators) cover the option contracts getting quote bars.TestCategory!=TravisExclude&TestCategory!=ResearchRegressionTests&TestCategory!=RegressionTests):Passed! - Failed: 0, Passed: 39834, Skipped: 100Passed! - Failed: 0on each shardPassed! - Failed: 0, Passed: 5IndicatorHistory(RateOfChange(1), ...)against the helper-createdROC(1): 0 of 899 differ (847 on master).IndicatorHistoryon the helper-createdROC(1)instance itself: 0 of 899 differ (847 on master and da0ddd9).RateOfChange(1)registered with a quote consolidator, on itself: 0 of 899 differ.IndicatorHistory(SimpleMovingAverage(1), ..., lambda x: x.bid.close)against a bid-close SMA registered with a quote consolidator: 0 of 900 differ, with no exception.IndicatorHistory(SimpleMovingAverage(1), SPY, ..., Resolution.Tick)with SPY subscribed at minute resolution: 5549 rows, the same as master.