[SDK] Support MeterConfigurator updates and example - #4313
[SDK] Support MeterConfigurator updates and example#4313adityabavadekar wants to merge 4 commits into
Conversation
Adds MeterProvider::UpdateMeterConfigurator. It sets a new MeterConfig on all meters that already exist, and new meters get the new configurator too. The update takes the same lock as GetMeter, so GetMeter can never hand back a meter with an old config. The meter enabled flag is now an atomic, so the update does not block instrument creation or Collect.
|
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4313 +/- ##
==========================================
+ Coverage 81.14% 81.18% +0.05%
==========================================
Files 446 446
Lines 18922 18941 +19
==========================================
+ Hits 15353 15376 +23
+ Misses 3569 3565 -4
🚀 New features to boost your workflow:
|
Fixes include lists reported by include-what-you-use. Adds a bazel target for the example, and tests for the null configurator and invalid context branches.
eeaf61d to
ab71694
Compare
dbarker
left a comment
There was a problem hiding this comment.
Thanks for the feature and tests! Please see some initial comments below.
| // This example shows how to use MeterProvider::UpdateMeterConfigurator to enable and disable | ||
| // meters by instrumentation scope at runtime. Updating the MeterConfigurator affects all existing | ||
| // and future meters provided by the MeterProvider. It is safe to call concurrently with | ||
| // MeterProvider::GetMeter and with instrument creation and recording on existing meters. |
There was a problem hiding this comment.
existing meters -> existing instruments
| // and stop collection. Code that must tolerate being started with a disabled Meter has to create | ||
| // its instruments again after the Meter is enabled. | ||
|
|
||
| #include <stdint.h> |
There was a problem hiding this comment.
nitpick - use the c++ header <cstdint> and std:: namespace for integer types.
|
|
||
| // A long export interval is used so that this example only exports when it calls ForceFlush. | ||
| metrics_sdk::PeriodicExportingMetricReaderOptions options; | ||
| options.export_interval_millis = std::chrono::milliseconds(60000); |
There was a problem hiding this comment.
consider reducing this (maybe 10 s). I think force flush may block for as long as the export interval plus the export.
| void UpdateMeterConfig(MeterConfig config) noexcept; | ||
|
|
||
| /** Returns whether this meter is enabled by its current MeterConfig. */ | ||
| bool IsMeterEnabled() const noexcept { return meter_enabled_.load(std::memory_order_relaxed); } |
There was a problem hiding this comment.
Consider renaming to IsEnabled()
| MeterConfig meter_config_; | ||
| // MeterConfig state is stored in an atomic variable so that instrument creation and Collect() | ||
| // never block on a concurrent MeterProvider::UpdateMeterConfigurator. | ||
| std::atomic<bool> meter_enabled_{true}; |
There was a problem hiding this comment.
As implemented the dynamic enable/disable has some quirks:
- If the meter is disabled when the instrument is created then the instrument is a noop for the life of the process regardless if the meter is later enabled.
- If an instrument is created from an enabled meter and then that meter is disabled, the instrument still records into storage as if it is enabled. The recording cost does not go away.
This is a bit out of line with the spec:
If a Meter is disabled, it MUST behave equivalently to No-op Meter.
The value of enabled MUST be used to resolve whether an instrument is Enabled. See Instrument Enabled for details.
The instrument needs to also have knowledge of the meter's enabled state. This is also a requirement if we support the sync instrument Enabled API method. I think we should support this method :).
To support the ability to enable and disable instruments consider making this a std::shared_ptr<std::atomic<bool>>. This could then be passed on to each instrument (Synchronous base class), SyncMetricStorage and BoundEntry to allow each instrument to disable (behave as a noop instrument) and enable based on the MeterConfig applied at any time.
I think it would okay to not implement full sharing of this meter_enabled flag and adding the sync instrument Enabled API method in this PR, but there should be a follow up to do so.
| // investigate. Its metrics are collected and exported again. | ||
| // Stage 4: The investigation completes and the external_library meter is disabled again. | ||
| // | ||
| // IMPORTANT: a Meter that is disabled when an instrument is created returns a no-op instrument, |
There was a problem hiding this comment.
Thanks for the explanation here. I think we should move towards meters/instruments that can be enabled and disabled dynamically independent of the MeterConfig on startup. More in this comment .
|
|
||
| provider->UpdateMeterConfigurator(EnableAll()); | ||
| counter->Add(1); | ||
| EXPECT_EQ(CollectScopeNames(reader), (std::set<std::string>{"scope.toggle"})); |
There was a problem hiding this comment.
This assertion only proves that the scope is present; it does not prove that the collected value is correct. All six new UpdateMeterConfigurator* tests use CollectScopeNames(), which discards the metrics and points:
disable -> Add(1) // should be ignored
enable -> Add(1)
expected sum: 1
current sum: 2Both sums produce {"scope.toggle"}, so this test stays green while the disabled measurement is retained and later exported. Please inspect the counter's SumPointData (or add a value-returning collection helper) and assert that the sum is 1 here.
| opentelemetry::nostd::string_view unit) noexcept | ||
| { | ||
| if (!meter_config_.IsEnabled()) | ||
| if (!IsMeterEnabled()) |
There was a problem hiding this comment.
This branch turns the Meter's current disabled state into a permanent object choice: kNoopMeter returns a NoopCounter whose Add() methods are empty. Re-enabling the Meter cannot replace a counter already held by library code:
static auto counter = meter->CreateUInt64Counter("requests"); // Meter is disabled
// The host later enables this Meter.
counter->Add(1); // Still a NoopCounter.This defeats the runtime-enable-for-debugging use case in #4256, especially for third-party instrumentation. The example avoids the failure only by creating every instrument while enabled; an application cannot force a dependency to recreate its instruments.
Please return an instrument whose recording behavior observes the Meter's shared enabled state (or an equivalent revivable proxy), and add a regression test: create while disabled, enable, Add(1), collect, and assert value 1.
|
|
||
| Metrics are exported to stdout via the `OStreamMetricExporter`. The example | ||
| uses a long export interval and calls `ForceFlush` at the end of each | ||
| stage, so each stage exports exactly once. |
There was a problem hiding this comment.
The long interval does not make ForceFlush() the only export trigger. OnInitialized() starts the worker, and DoBackgroundWork() calls CollectAndExportOnce() before its first wait. That startup collection races with Stage 1's recording and ForceFlush():
documented: 4 stage-triggered export batches
observed: 5 batches, with the extra batch's placement varying
Therefore the checked-in exact transcript is not deterministic. The CTest target only runs the executable, and Bazel uses a plain cc_test, so neither validates stdout.
| const std::lock_guard<std::mutex> guard(lock_); | ||
| context_->SetMeterConfigurator(std::move(meter_configurator)); | ||
|
|
||
| for (auto &meter : context_->GetMeters()) |
There was a problem hiding this comment.
for (auto &meter : context_->GetMeters())GetMeters() is explicitly non-thread-safe, so this loop is safe only under a cross-class invariant: every production mutation of meters_ currently reaches AddMeter() or RemoveMeter() through a MeterProvider method that holds lock_. That invariant is not stated here, and it is not structurally enforced because MeterContext exposes both mutators publicly.
T1: provider lock_ -> GetMeters() -> iterate borrowed span
T2: meter_lock_ -> AddMeter() -> meters_ may reallocate
lock_ does not exclude a direct/future T2 caller. The analogous tracer and logger loops explicitly document their ownership invariant, and their collections are provider-private.
Please either enforce this invariant by using ForEachMeter() or making the mutators provider-only, or at minimum add the equivalent ownership comment above this loop so future callers do not introduce a race.
Fixes #4256
Adds the ability to update a
MeterConfigurator(and theMeterConfigofexisting meters) at runtime. Following the pattern used in #4065 and #4224.
This implements the optional requirement to allow updating the
MeterProvider's MeterConfigurator:
https://opentelemetry.io/docs/specs/otel/metrics/sdk/#meterconfigurator
Changes
UpdateMeterConfiguratortoMeterProviderto support setting newmeter configs at runtime. The update holds the same lock as
GetMeter, so ameter is never returned with a config that is out of date with the provider
configurator.
Meterenabled flag to an atomic so updating it does not blockinstrument creation or
Collect.MeterContext::SetMeterConfigurator.Note: Instruments created from a disabled meter remain no-op after the meter is re-enabled, and measurements recorded while disabled still appear in the cumulative total. Both behaviors are documented in the example README.
For significant contributions please make sure you have completed the following items:
CHANGELOG.mdupdated for non-trivial changes