Skip to content

FFL-3568: Share precomputed fixtures and lossless numeric policy - #4101

Open
leoromanovsky wants to merge 14 commits into
developfrom
leo.romanovsky/ffl-3568/precompute-fixture-fanout
Open

leoromanovsky wants to merge 14 commits into
developfrom
leo.romanovsky/ffl-3568/precompute-fixture-fanout

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Add 63 shared precomputed fixtures and an indexed runner for public flag evaluations, cached assignments, exposures, and evaluation events. The snapshot pins 9afc3f75c3fef9643a9c1b5e2b1e5a9adac48177. Ref: FFL-3568.

Motivation

Each SDK needs reproducible checks against the shared response contract. New fixtures must reach consumers without a coordinated SDK release.

Additional Notes

This branch merges develop and retains its numeric converter. Unsupported variation types produce a once-only warning and leave valid flags available. The merge preserves response consent on each retained flag. Missing extraLogging defaults remain compatible. Applicable null targeting keys produce an actionable fixture error because Android requires a string.

After this PR lands, the weekly or manually triggered workflow can open one draft update PR at a time. Scheduled and manual updates share one concurrency group on develop. The index discovers new cases without a runner list change. Each SDK adopts a pinned snapshot independently; consistency follows each consumer's tested merge. The workflow does not merge PRs or release SDKs.

flowchart TD
    A[Add or change shared fixtures] --> B[Review and merge upstream]
    B --> C[Weekly or manual snapshot update]
    C --> D[Open one draft SDK update PR]
    D --> E[Run local conformance tests and SDK CI]
    E --> F{Tests pass?}
    F -->|No| G[Fix SDK or runner; correct fixtures upstream if needed]
    G --> E
    F -->|Yes| H[Review and merge SDK update]
    H --> A
Loading

Java example: #13347 bumped the fixture revision, strengthened fixture assertions, and merged. The Android, iOS, and Flutter updates follow the same review-and-test cycle independently.

Test coverage: 1,104 feature tests and 320 JVM integration tests pass; three fixture cases have declared platform exclusions. Coverage includes numeric reads, unsupported types, cache restoration, and event matching. Snapshot tests preserve raw numeric literals before and after cache restoration. Negative controls detect lost consent and cache values replaced with zero. A deterministic concurrency simulation produces two drafts with separate event groups and one with the shared group; hosted workflow execution is not tested. Detekt reports unresolved generated classes; the local documentation gate could not run because FactoryKit is unavailable.

Runtime validation

Tested commit Validation Result
DataDog/dd-sdk-android@33c91142be45b302e80e5197cdc019e546a5fb28 Native integration: consent refresh, cached consent, protected targeting keys, upload retries, tracking disabled; native and OpenFeature clients on Android API 36 arm64 16 passed, 0 failed, 0 skipped

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Make sure you discussed the feature or bugfix with the maintaining team in an Issue
  • Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 93.75%
• Overall Coverage: 72.30% (-0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 33c9114 | Docs | View more details | Give us feedback!

@aarsilv

aarsilv commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude, working with Aaron:

This PR and #4099 both change FlagValueConverter. Could you stack this PR on #4099 and keep its converter? This PR would then keep the fixture runner, the sync workflow, the extraLogging default and the handling of unsupported variation types. If you prefer your rules for the two differences below, we can change #4099 first.

#4099 now checks whole values in Double reads as well as Int reads. We ran the shared fixtures from DataDog/ffe-system-test-data#65 against it with this PR's runner, merged with develop. The local test copy adds "extraLogging": {} to each fixture assignment. Of 95 runs, 88 pass, 3 are skipped and 4 fail. The 4 failures expect FLAG_NOT_FOUND for an unsupported variationType, and this PR omits those flags.

Two differences to settle:

Could you also update the description's statement that #4099 "still enforces the previous strict-type contract"? #4099 already has lossless Int reads across all numeric types.

@aarsilv

aarsilv commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment from Claude, working with Aaron:

P3 follow-ups after rebasing on #4099:

  • README numeric text. The new section says that inexact conversions return TYPE_MISMATCH. With FFL-3565: Accept every numeric type in Int and Double reads #4099, a Double read of 1.20000000000000001 returns the nearest Double. For NaN, resolve() reports PARSE_ERROR. Could this section link to the existing numeric paragraph in the README?
  • detekt entries. Most new entries support this PR's converter. Examples include BigDecimal.intValueExact(), Result.success(kotlin.Int), and Double.isFinite(). After rebasing, could you remove entries that the resulting code no longer uses? Keep entries for calls that remain, including EnumEntries.none in FlagsJson.
  • Unsupported types. PrecomputeMapper drops flags with an unsupported variationType without logging the cause. Once the client is ready, a lookup reports FLAG_NOT_FOUND, as the fixtures expect. Could the mapper log a warning once through the internal logger to explain the cause?
  • targetingKey: null. The fixture schema permits context.targetingKey to be null. At line 63, CanonicalPrecomputedTest calls context["targetingKey"].asString, which throws for JSON null. None of the 57 current fixtures uses a null targeting key. Could the runner handle null, or could future fixtures with null exclude Android through _skip?

Preserve original numeric precision for typed reads and cache restoration.

Ref: FFL-3568
Merge the dependency branch, pin shared fixtures to main at 9afc3f7, and address runner and numeric review feedback.

Ref: FFL-3568
@leoromanovsky
leoromanovsky changed the base branch from develop to aarsilv/ffl-3565/numeric-type-mismatch November 4, 2026 18:51

Copy link
Copy Markdown
Contributor Author

@aarsilv Follow-ups addressed in a45b99c342 and the numeric documentation follow-up:

  • Stacked on FFL-3565: Accept every numeric type in Int and Double reads #4099 with a merge commit and changed the PR base. Retained its converter structure and 64-character decimal parsing bound. The merged fixture contract requires nearest-Double fractional reads, exact original whole-number reads, and rejection of nonzero underflow. The remaining delta enforces those rules, including rejecting 1e300 as an inexact whole value. The PR description and README no longer describe FFL-3565: Accept every numeric type in Int and Double reads #4099 as strict-type or promise the old printed-form exception.
  • Removed obsolete converter detekt entries; retained the calls the merged implementation uses, including the unsupported-type detection.
  • Added a once-only internal USER warning for omitted unsupported variation types, plus coverage that valid sibling flags survive and the warning requests onlyOnce.
  • Defined null-targeting-key applicability explicitly: Android requires a string, so an applicable null case fails with an instruction to declare the exclusion upstream instead of throwing from asString.

The snapshot pins fixture main at 9afc3f75. Local validation passes: 1,007 feature tests, 259 integration tests with 3 platform exclusions, formatting, and detekt.

Verify legacy Android accepts encrypted dependency metadata and a sibling error map while preserving otherwise valid assignments.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Base automatically changed from aarsilv/ffl-3565/numeric-type-mismatch to develop November 11, 2026 17:45
Retain the current numeric converter and preserve evaluation consent when filtering unsupported fixture assignments. Document the recurring fixture update and test loop.

Ref: FFL-3568

Copy link
Copy Markdown
Contributor Author

🤖 Comment from Codex working with Leo:

Follow-up to the converter request and the fixture-runner follow-ups: 7319c496c9373de4651cd7b686f9398480e4a697 now includes current develop and keeps its numeric converter unchanged.

The merge retains the once-only unsupported-type warning, valid sibling flags, numeric documentation, and actionable null-targeting-key error. It also preserves evaluation consent on retained flags. The new true/false consent test fails when that propagation is removed.

Local tests pass: 1,104 feature tests and 317 JVM integration tests, with three declared fixture exclusions. The release runtime suite passes all 16 tests on Android API 36 arm64, covering consent, cache restoration, retries, and disabled tracking through native and OpenFeature clients. The installed APK matches the local build.

The PR description and future update template now show the fixture update, test, fix, and merge loop. They link the merged Java example and explain independent SDK adoption.

@leoromanovsky
leoromanovsky marked this pull request as ready for review November 13, 2026 03:30
@leoromanovsky
leoromanovsky requested review from a team as code owners November 13, 2026 03:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T23:01:45.482115Z 33c9114 New commits
🔒 Security Review ✅ Completed 2026-11-13T23:04:11.956592Z 33c9114 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Overlapping scheduled and manual fixture updates can create duplicate draft PRs because their concurrency groups differ. A shared group per ref closes this race.

Open Bits AI session

🤖 Bits Code Review · Commit 7319c49

estherk15
estherk15 previously approved these changes Nov 13, 2026

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR adds shared flag-evaluation fixtures, an indexed conformance runner, and automated snapshot updates while retaining develop’s numeric converter and consent propagation. This follow-up adds numeric snapshot coverage and fixes duplicate draft PR creation by serializing scheduled and manual updates; no findings remain open.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 33c9114

@0xnm

0xnm commented Nov 16, 2026

Copy link
Copy Markdown
Member

Is it actually the right place for such huge fixtures verification dump? Won't RUM FIT be a better one?

@leoromanovsky

Copy link
Copy Markdown
Contributor Author

Is it actually the right place for such huge fixtures verification dump? Won't RUM FIT be a better one?

I'll investigate integrating into RUM Fit instead, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants