fix(dev-server): fetch flag pages concurrently instead of serially#732
Draft
davidbrackbill wants to merge 2 commits into
Draft
fix(dev-server): fetch flag pages concurrently instead of serially#732davidbrackbill wants to merge 2 commits into
davidbrackbill wants to merge 2 commits into
Conversation
launchdarkly/gonfalon PR #66623 found dev-server start --source=staging sync takes ~50s against a large project, causing flaky CI healthchecks. Root cause: GetAllFlags paginates the flags list at 100/page and follows each `next` link one at a time, even though every offset is already known once the first page returns its total count. Fetch page 0 to learn the total count, then fetch the remaining pages concurrently (bounded to 6 at once, comfortably under the API's rate limiter). No behavior change for callers — CreateProject/UpdateProject stay fully synchronous, just faster underneath. Measured against the real staging default project (2,985 flags): full sync dropped from ~50s to ~10s. Deliberately does NOT change when AllFlagsState/AvailableVariations become available relative to server startup - see discussion on an earlier draft of this fix for why splitting that further introduces real problems for local/regular dev-server use (silent incomplete state, no completion signal, weaker error visibility) without a clear need for sub-2s startup.
Concurrent page fetches can all get rate-limited at once and, without jitter, would all compute the same X-Ratelimit-Reset wait and retry at the exact same instant - recreating the burst that got them limited. Add a small random extra delay on top of the required wait so concurrent retries spread out instead of waking up in lockstep.
Author
|
Added jitter to the 429 retry backoff (`b0d85c5`) to address the burst/thundering-herd concern raised above: concurrent page fetches that get rate-limited around the same moment now retry at randomized offsets instead of all waking up at the exact same `X-Ratelimit-Reset` instant. Testing path for this:
|
davidbrackbill
commented
Jul 8, 2026
| sleep := resetUnixMillis - timeImpl.Now().UnixMilli() | ||
| log.Printf("Got 429 in API response. Retrying in %d milliseconds.", sleep) | ||
| timeImpl.Sleep(time.Duration(sleep) * time.Millisecond) | ||
| sleep := time.Duration(resetUnixMillis-timeImpl.Now().UnixMilli())*time.Millisecond + timeImpl.Jitter(maxRetryJitter) |
Author
There was a problem hiding this comment.
Adds jitter in case of 429 retry to avoid thundering herds
Contributor
|
Would it be possible to source the required data for the initial sync from the streaming enpoint? (I am not sure if its possible, but if it is, it will be way faster) https://github.com/launchdarkly/ldcli/blob/main/internal/dev_server/adapters/sdk.go#L41 |
Author
@dmashuda |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The LD CLI dev server starts by fetching all flags in the project, which paginates the
GET /api/v2/flags/{projKey}list endpoint at 100 flags/page serially, taking about ~50s in CI for Catamorphic.Fix
GetPaginatedItemsnow fetches page 0 to learn the total count and page size, then fetches the remaining pages concurrently, bounded to 6 in flight at once, and reassembles them in order.Retry429sis unchanged and still covers any individual page that gets rate-limited.As measured locally, the full sync becomes ~50s → ~10s.
Alternative considered and rejected: decouple
AvailableVariationsfrom server startupI initially went further than pagination and additionally split
CreateProject/UpdateProjectso that only the fast SDK-streaming flag fetch (AllFlagsState) was synchronous, and the slowGetAllFlags-backedAvailableVariationsfetch (used only by the local override-picker UI — never by the FD/streaming endpoints, which depend solely onAllFlagsState) ran in a background goroutine after the server started listening. That got healthcheck-ready time down to ~1-3s instead of ~10s.Risks
By adding parallel fetches, we risk increased load on this endpoint during startup, marginally. If several engineers/CI jobs run
dev-server startaround the same time against this shared account-wide project, the aggregate burst effect is worth a sanity check against the API's actual rate-limit policy before merging.Testing
go test ./...passes.go test -race -count=3 ./internal/dev_server/...passes.defaultproject (2,985 flags): full sync ~50s → ~10s, single run, no rate-limit errors observed.Note
Medium Risk
Same request volume is compressed into shorter bursts against the LD API, which may increase 429s under shared load; mitigations are a concurrency cap and jittered retries, with no change to startup semantics or public API contracts.
Overview
Speeds up dev-server flag sync by replacing serial
next-link pagination with offset-based concurrent page fetches (up to 6 in flight) after the first response suppliestotalCount.GetPaginatedItemsno longer takes anhrefor walks_links.next;getFlags/GetAllFlagscallers keep the same public behavior.Retry429sadds up to 250ms jitter on top ofX-Ratelimit-Resetwaits (via newMockableTime.Jitter) so concurrent 429 retries don’t wake in lockstep. Tests cover ordering, concurrency, errors, and jittered retries.Reviewed by Cursor Bugbot for commit b0d85c5. Bugbot is set up for automated code reviews on this repo. Configure here.