Repository navigation
feat(events): persist events to disk so they survive the process (tier 3) - #408
Open
abelonogov-ld wants to merge 247 commits into
Open
abelonogov-ld wants to merge 247 commits into
abelonogov-ld wants to merge 247 commits into
Conversation
…er 1) Tier 1 of the event durability spec: the buffer. AndroidEventBuffer folds an evaluation into summary counters as it is recorded and holds in full only the events that have to be sent one by one, so a burst of evaluations of untracked flags cannot displace anything. AndroidEventProcessor takes over from java-sdk-internal's DefaultEventProcessor in ComponentsImpl, removing the bounded queue that used to sit between the calling thread and the summarizer. Spec: Event Durability, "Tier 1 — the buffer". Co-authored-by: Cursor <cursoragent@cursor.com>
Tier 2 of the event durability spec: recoverable failure. flushAndWait(timeout) delivers what has been recorded and reports whether it got there inside the budget, so an application that knows it is about to go away — backgrounding, or an uncaught exception handler on its way out — gets an answer instead of a fire-and-forget flush. Delivery now reports its outcome so the bounded call can tell delivered from not. Spec: Event Durability, "Tier 2 — recoverable failure", §10. Co-authored-by: Cursor <cursoragent@cursor.com>
Tier 3 of the event durability spec: unrecoverable failure. EventStore stages serialized events in memory and commits them to an append-only log in the no-backup files directory, so events survive a process that never gets to flush. Commit points — track, identify, an explicit flush — write on the caller's thread before returning, and the evaluations counted since the last one are summarized and written with them. Delivery reads closed batches back from disk and deletes them only once LaunchDarkly has accepted them, under a payload ID that stays the same across retries so a redelivered batch is not counted twice. Each process of a multi-process application gets its own log. Spec: Event Durability, "Tier 3 — unrecoverable failure"; the mechanism is Page-Cache Event Persistence. Co-authored-by: Cursor <cursoragent@cursor.com>
The counterpart of the iOS EventPersistenceBenchmark, covering the same five context shapes and three privacy settings so the two platforms can be compared rather than assumed alike. Gated on LD_EVENT_BENCH=1, which also turns on stdout and disables up-to-date checks for the test task, so an ordinary run skips all four and is otherwise unaffected. The answer is that Android does not have the problem iOS had. Its unoptimized path beats iOS's fully optimized one on every row: evaluation, summary only 1.87 us (iOS) -> 68 ns evaluation, full event 17.35 us -> 1.51 us evaluation + track 38.40 us -> 4.92 us Cost still tracks attributes written rather than the redaction decision, as on iOS, but at 48-59 ns per attribute against Codable's 1.14 us. So there is no Codable-shaped overhead to remove, because there is no Codable. Co-authored-by: Cursor <cursoragent@cursor.com>
The next commit drops the split-package hack in favor of declarations that launchdarkly/java-core#214 makes public. That is not in a release yet, so this points the build at a working copy until one carries it. Skipped when the directory is absent, so a clean checkout still resolves the published artifact rather than failing. Revert together with a version bump once the release is out. Co-authored-by: Cursor <cursoragent@cursor.com>
…package AndroidEventBuffer was declared in com.launchdarkly.sdk.internal.events so it could reach the summarizers and the output formatter, which were package-private there. That made a package split across two artifacts, forced the class to be public for the rest of the SDK to use it, and needed a javadoc exclusion plus a classpath workaround to keep it out of the published docs. launchdarkly/java-core#214 makes those declarations public, so the class moves to com.launchdarkly.sdk.android where it belongs. It is package-private now, along with its members and Payload, because everything using it is in that package, so it is out of the published docs by virtue of its visibility rather than by being excluded from javadoc. The build.gradle javadoc block goes back to what it was before tier 1. No behavior change: same summarization, same wire format, same tests. Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush
…y/event-durability-tier3-persistence
Diagnostics are meant to be off in both states, and updateScheduledTasks cancels the periodic task accordingly, but two paths could still reach the sender after the state had changed. Cancelling does not stop a run that has already begun, and the init event is submitted to the executor while still online, so going offline or backgrounding between submission and execution left it to send anyway. Both paths now re-check on the way out, via a predicate that mirrors the scheduling condition so the two cannot drift. The check in sendDiagnosticStats sits ahead of createEventAndReset, which clears the counters it returns, so a suspended period defers its statistics rather than discarding them. Analytics delivery is unchanged: deliverPayload already skips while offline, and it deliberately keeps running in the background so events recorded before the app was backgrounded still get out. Co-authored-by: Cursor <cursoragent@cursor.com>
The Android prefix in this package is reserved for adapters over Android OS APIs - AndroidPlatformState, AndroidTaskExecutor, AndroidEnvironmentReporter. Neither of these touches an Android API, so the prefix put them in the wrong category while also repeating what the repository and package already say. AndroidEventProcessor -> DirectEventProcessor AndroidEventBuffer -> OutboundEventBuffer DirectEventProcessor names what sets it apart from java-sdk-internal's DefaultEventProcessor: nothing sits between the caller and the summarizer. OutboundEventBuffer says which direction the events are going, which matters once tier 3 adds a store to read them back from. DefaultEventProcessor was avoided as a name because java-sdk-internal already exports that simple name and this package imports from it. Rename only, including the test class. No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # launchdarkly-android-client-sdk/src/main/java/com/launchdarkly/sdk/android/DirectEventProcessor.java
- drop OutboundEventBuffer.isEmpty, which nothing calls
- StandardCharsets.UTF_8 in place of Charset.forName("UTF-8")
- scheduleWithFixedDelay, so a cached process does not come back owing every missed run
- split an over-long Javadoc sentence
Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # launchdarkly-android-client-sdk/src/main/java/com/launchdarkly/sdk/android/DirectEventProcessor.java
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # launchdarkly-android-client-sdk/src/main/java/com/launchdarkly/sdk/android/DirectEventProcessor.java # launchdarkly-android-client-sdk/src/main/java/com/launchdarkly/sdk/android/OutboundEventBuffer.java
(cherry picked from commit 123cb81)
Bugbot flagged that diagnosticInitSent is set without consulting the Result. That is deliberate: DefaultEventProcessor does the same, and a failed post returns an unsuccessful Result rather than throwing, so the flag flips either way. Note it at the assignment so the next reader does not read it as an oversight. Co-authored-by: Cursor <cursoragent@cursor.com>
Going offline cancels the periodic flush, so coming back online started a fresh interval and left whatever the outage buffered waiting for it. Each drop re-anchored that interval, so repeated brief losses could defer delivery well past a single one. Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush
…rites The track before the flush queues a commit of its own. When that commit ran first, it covered every recording the flush needed, so the flush correctly had nothing to write on the caller's thread and the test failed. Co-authored-by: Cursor <cursoragent@cursor.com>
flushAsync committed on the caller's thread before handing back its future, so flushAndWait's deadline only began once that commit was done. A write stalled on the disk, or a commit holding the commit lock behind one, held the caller with no bound -- the crashing thread, when the crash handler was the caller. flushAsync now queues that commit on the commit executor, which the network never holds. The delivery still commits before it sends, so a true answer still means the events were written. flush() and blockingFlush() keep committing on the caller's thread. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ATR3-005) closeMovingTo closed and unlocked the log whether or not the rename worked. A log the rename left behind was then no longer this store's to write, so the next append started another one, while the store still counted the first log's events as the open log's. Those events were out of reach for the rest of the session, and the next batch was named for more events than it held. A failed rename now leaves a log that is still there open and locked: later appends go to it, and the next close moves it whole. A log that is gone is closed as before, and its events stop counting. Co-authored-by: Cursor <cursoragent@cursor.com>
Every delivery committed, closed a new batch, and only then retried the batch at the head. For as long as an outage lasted, each flush and each periodic delivery left one more batch: another file on the disk, or another batch held in memory, each with the summary its commit wrote. A delivery now sends the batches already waiting first, and closes what was committed since only once none of them is left blocked. Until then it still commits, into the open log, which closes once it is full. The retry makes the same pass, so a true answer still means the caller's events went too. Co-authored-by: Cursor <cursoragent@cursor.com>
…6, part 2) Capacity never refuses a summary, and the byte capacity applied only where events were persisted, so with persistence off nothing bounded them: a delivery that kept failing left the heap growing for the life of the process. The byte capacity now refuses a summary either way. Full events kept off the disk are still bounded by count alone. Co-authored-by: Cursor <cursoragent@cursor.com>
Summary frames were counted as events in every state the store keeps them in. Each commit point writes one, so during an outage they filled capacity on their own and track and identify were refused, although durability 4.2 says summaries do not consume full-event capacity. Summaries are now staged apart from the full events, and taken as the full events followed by the summaries, so each part of a commit can say how many of its frames are summaries. Every state counts its summaries beside its frames, and capacity leaves them out. A batch the store did not close, from a previous run or another process, has nothing to say what its frames are, and still counts all of them. The file format is unchanged; the bytes of summaries still count against the byte capacity. Co-authored-by: Cursor <cursoragent@cursor.com>
…ng (ATR3-007) Expiry only skipped the batches its own store had a request out for, so another store in this process, or another process, could delete a batch mid-request and leave nothing to retry from. A sweep of sibling environments skipped nothing at all. A sender now holds a shared lock on the batch for as long as its request is out, and expiry deletes a batch only under an exclusive lock. Since POSIX locks belong to the process, the batches sent here are also kept in a registry that expiry in this process asks instead of opening the file. The sweep leaves the directory of any store open in this process alone, and a batch already gone when expiry gets to it is no longer counted as dropped. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ad74c09. Configure here.
Expiry opened a batch "rw" to lock it exclusively, which creates the file if another process delivered or expired it since the listing. For a moment an empty batch stood under its name, and a delivery reading it then reported its events lost. The batch is now opened for writing without being created: through java.nio.file where it exists (API 26+, and the JVM), and through ParcelFileDescriptor before that. Co-authored-by: Cursor <cursoragent@cursor.com>
… (ATR3-030) A directory that could not be listed read as an empty one: the store stopped counting every batch on disk, including the one the flush had just closed, and the flush answered true having sent nothing. A directory that is there and cannot be listed now answers with the batches this store last counted, oldest first, and leaves their counts alone. The delivery sends those and ends as a failure that may pass, so the flush answers false unless the retry can list the directory. Each batch's modification time is now read once per listing, before the sort, rather than inside the comparator. Co-authored-by: Cursor <cursoragent@cursor.com>
…3-031) A log whose rename failed stays open since ATR3-005, so its events are no longer stranded or miscounted, but the flush that could not close it still answered true with no request for them. The delivery now reads, under the commit lock right after the close, whether committed events are left outside every batch, and while they are it ends as a failure that may pass: the retry tries the close again. Co-authored-by: Cursor <cursoragent@cursor.com>
Assembly stops at the first frame that is torn or fails its checksum, and a batch damaged after its first frame was sent with what came before it, removed, and reported as delivered: the rest of its events were gone and nothing said so. The payload now carries how many events made it in. Where a batch named with a count yields fewer, the shortfall is counted as lost once the batch is accepted and removed, and the pass fails. It is not counted at the read, because a batch kept for a retry is read short again. Diagnostics and the request use the count sent. Co-authored-by: Cursor <cursoragent@cursor.com>
…O lock (ATR3-033) Recovery held the I/O lock while it read and checked every log a crash left, so a commit at a commit point, which at IMMEDIATE is a track or an identify, waited for all of them. The logs are now listed under the lock, each read and counted without it, and the lock taken again only to move one into a batch and count it, so a listing cannot count it twice. A commit waits for one rename at most. A log gone by the time of the rename was recovered by another store first, and is skipped without a warning. Co-authored-by: Cursor <cursoragent@cursor.com>
Reaching PENDING_COMMIT_THRESHOLD only schedules a commit, and recording does not wait for it, so a burst of evaluations faster than the commit executor can leave up to the event capacity for the next track or identify to encode. The threshold comment now calls it the common case, with the event capacity as the hard bound, and the IMMEDIATE documentation says a track after such a burst can take milliseconds. The behavior is unchanged, as on iOS, where C-16 was resolved the same way. Co-authored-by: Cursor <cursoragent@cursor.com>
…state Changing the persistence level restarted the client and only called setOffline() once LDClient.init had returned, so for the whole init wait the new client could connect and deliver the events stored before the change. The replacement is now configured offline from the start when the switch is on, and a toggle made during the restart, which there was no client to act on, is applied once the new client is in place. Co-authored-by: Cursor <cursoragent@cursor.com>
…(ATR3-001) A failed delivery no longer deletes the batch or stops the processor. Each environment keeps an in-memory DeliveryRetryState bound as durability §9 specifies: the wait starts at a second and doubles up to the flush interval (at least a second); a refusal unlikely to pass soon (a 4xx other than 400/408/429, or a TLS failure) jumps to about eight and a half minutes and doubles up to an hour; Retry-After sets the wait, capped at an hour and jittered; one accepted delivery resets it. Nothing is posted during a wait. A flush waits out only the quick first retry and otherwise answers false at once; the retry it queues posts the kept batch, under its original payload ID, when the wait is over. Co-authored-by: Cursor <cursoragent@cursor.com>
… (ATR3-002) A 400 is a payload the service cannot read. Kept for good, the batch is the oldest and so the next one posted, which held every batch behind it for the rest of the process. Now a second consecutive 400 for the same batch removes it and counts its events as lost, so the next flush a caller waits on hears about it; delivery of the batches behind it goes on once the backoff allows. Any other response for the batch starts the count again. The count is held only in memory. Co-authored-by: Cursor <cursoragent@cursor.com>
Durability §14.1 now specifies what the code already does: a summary that cannot be encoded is dropped whole, logged, and counted as lost, and its counters are never put back. An encoder failure comes from the data, so restored counters would fail at every later commit. The full-event case already had a test that the flush answers false; this adds the same for a summary, and says in OutboundEventBuffer that the drop is reported through takeSerializationFailure. Co-authored-by: Cursor <cursoragent@cursor.com>
… persist (ATR3-014) A requested delivery committed before checking whether it could send, and close() committed before its final delivery. Where events persist that commit is what puts them on disk. Where they do not (DISABLED, or a store that gave up on writing) it made nothing durable: online, the commit that closes the batch is the drain anyway, and offline it turned pending events and summaries into held bytes that could not be sent and waited in memory exactly as the events would have. Durability §5 and C5b forbid encoding there before a delivery. Both commits now run only where the store persists. aCommitNeverSplitsAnEvaluationAcrossTwoCommits relied on offline flushes committing in a store that does not persist, so it now flushes online against a local server. Co-authored-by: Cursor <cursoragent@cursor.com>
A loss was claimed by whichever waiting flush answered first, through one global mark. Two flushes waiting on the same retry carried the same mark, so the second to answer was told nothing and returned true for a delivery that lost an event; a later flush could likewise take a loss from an earlier one's delivery before that one's retry finished. Each queued delivery now snapshots how far losses had been reported as it was queued, and its callers are told false for any loss after that snapshot and up to their mark, whether or not another flush has reported it since. One loss can be reported to two overlapping flushes, which page-cache 5.3 allows; a flush beginning after the report is not told again. Co-authored-by: Cursor <cursoragent@cursor.com>
…processes (ATR3-015) The comment on deliverPayloadAndRetryLater said deliveries overlap: that close() delivers on the caller's thread and a retry runs beside the delivery that queued it. Every delivery -- requested, periodic, the retry and close()'s final one -- runs on the single scheduler thread, so none overlap within a processor, and close() queues behind a post or retry already running there. EventStore.remove said two deliveries can reach the same batch, which only another process can. The class documentation now says which work runs on which thread, that close() waits behind the scheduler within its budget, and that the overlap that does happen is across processes, deduplicated by payload ID. Co-authored-by: Cursor <cursoragent@cursor.com>
test-helpers 2.1.0 reads a request body with a single InputStream.read, which returns only what has arrived. A body sent in more than one TCP segment was recorded truncated, so a test parsing it failed with MalformedJsonException; aCommitPointMakesRoomInTheSummarizer did in 2 of 10 full runs. HttpServer hard-codes its delegate, so the test sources now carry a copy of NanoHttpdServerDelegate, which takes the jar's place on the unit-test classpath, reading until Content-Length. Guava is added compile-only at the version test-helpers already brings in at runtime. HttpServerBodyTest sends a body in two parts; it fails against the jar's class and passes with the copy. Co-authored-by: Cursor <cursoragent@cursor.com>
NanoHTTPD listens on every address with SO_REUSEADDR. On macOS the kernel then hands out, for port 0, a port another process already listens on at 127.0.0.1 -- 18 times in 20,000 starts on a machine running Android Studio, the emulator and Parallels. "localhost" resolves to 127.0.0.1 first, so the test's requests went to that process: a 404, a 401, or no answer until the read timed out. That is where aNonPositiveCapacityStillReportsRatherThanGoingSilent got a 401 from a server that only answers 202, and where closeNeverDeliversASummaryWithoutTheFeatureEventItCounted waited 10 s for a request its server never saw. The copied delegate now listens on 127.0.0.1, where the kernel will not hand out a port that is taken, and still reports localhost in request URIs. NanoHttpdServerDelegateTest (renamed from HttpServerBodyTest) adds a check that the server refuses a port another loopback listener holds; both its tests fail against the jar's class. Co-authored-by: Cursor <cursoragent@cursor.com>
The test server records a request before it answers it. Two tests switched the server's answer as soon as they saw a retry's post, which can come before that post is answered, so the post could get the answer meant for the next one. aRefusedBatchIsKeptAndPostedAgainOnceTheLongerWaitIsOver failed that way: the batch recorded meanwhile took the 503 meant for the flush after it, its retry took the 202, and the flush found the answers used up. aBatchAnUnheardDeliveryCouldNotSendGoesOutOnceTheServiceRecovers has the same window, since nothing waits for its retry's answer. Both now give each post its answer in order with Handlers.sequential. The other HandlerSwitcher tests switch only after a flush that waited for the answer, or after a quiet period, and are left as they are. Also drops two comments that described the old single-read body truncation as a limit to stay under. Co-authored-by: Cursor <cursoragent@cursor.com>
theStoreAsksThePlatformWhereItLivesOffTheCallersThread flushed to make the store ask the platform for its directory. Where events persist a flush commits on the caller's thread, by design, and whichever commit reaches the disk first is the one that asks. When the caller's flush beat the store's own commit and recovery, the test saw the platform asked on the caller's thread; it did once in 10 full runs. It now waits for the first ask instead. What it guards is unchanged: building the processor and recording on the caller's thread never ask. Co-authored-by: Cursor <cursoragent@cursor.com>
Covers what staging and committing leave on the disk, recovery after a delivered batch and after a torn last frame, discarding an uncounted batch in an unknown format, payload IDs, the order and bodies of batches held in memory, and capacity across closed batches and an earlier run's leftovers. Co-authored-by: Cursor <cursoragent@cursor.com>
…runcate A failed write keeps all its events in memory to deliver from there. Where it was the log's first write and the frames it left could not be truncated, the log was left on disk and the next run recovered and sent them again. Remove the log instead, as iOS does. Co-authored-by: Cursor <cursoragent@cursor.com>
Covers the 429 backoff, a delivery asked for during a retry wait, capacity through flushes that meet a delivery in flight, a flush answered by the failure of the delivery holding its batch, an evaluation not held up by a listing, removal before answering while a commit is stuck writing, debug events against the service clock and alongside a feature event, and an evaluation the summarizer turns away. Co-authored-by: Cursor <cursoragent@cursor.com>
A flush was told of every loss up to its mark that had not been reported when it began, so a flush with nothing of its own, asked for while another delivery that lost an event was in flight, reported that loss too. Each delivery a caller waits on now claims the losses between the last claim and its own mark as its pass ends. Deliveries run one at a time, so the claims follow the order they ran in rather than the order they answer, and a delivery waiting on a retry still keeps the losses before its mark. A delivery nobody waits on claims nothing, so the next caller hears of what it passed. Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Requirements
Related issues
Tier 3 of the Event Durability spec, "Tier 3 — persistence". Stacked on #403 (tier 2) and targets that branch, so this diff shows only tier 3. It will be retargeted once the branches below it merge. The iOS counterpart is launchdarkly/ios-client-sdk#538, and both write the same on-disk format.
Opened as a draft: a multi-threading and readability review turned up items still being worked through (see Known open items).
Describe the solution you've provided
Recorded events go to an append-only log on disk rather than only an in-memory buffer, so an application that dies shortly after recording an event still reports it: the next run delivers what the previous one left behind. That covers what tier 2 cannot:
SIGKILL, ANR kills, native crashes, and the system reclaiming a backgrounded process.EventPersistence(new public enum) andEventProcessorBuilder.eventPersistence(...):DISABLED(default): events stay in memory and the SDK does not touch the filesystem for events.DEFERRED: writes happen on the SDK's commit thread, a moment after recording.IMMEDIATE:trackandidentifywrite on the caller's thread before returning, so there is no window.EventStore: an append-only log under the no-backup files directory, one directory per mobile key. It handles framing, capacity, torn-tail recovery of a log interrupted by a crash, and a fallback to memory when a write fails (for example, a full disk), so a disk failure never takes the application down. Each process appends to its ownopen-<process>log; closedready-<payloadId>batches are shared, so a batch left by a process that never runs again is still delivered. The byte format lives inEventLogFormatand the log this process appends to inDiskEventLog, mirroring the iOS split.DirectEventProcessor): an evaluation appends to a pending run and updates a counter, with no encoding on the caller's thread. A commit encodes the run and its summaries together and stages them in the store.trackandidentifyare commit points. Evaluations commit once a threshold accumulates, on flush, and before delivery.flushAsync()answersfalsewhen events the SDK accepted were lost for good: an event or summary that could not be serialized, or that was too large to store. Each loss is reported to exactly one caller, the one whose delivery closed its batch after the loss, as on iOS. Events dropped for capacity are counted in diagnostics, not reported as losses.close()commits first, then delivers with what is left of one two-second budget. Neither step waits on the caller's thread past the budget, so a stalled disk write cannot become an ANR. Work that outlasts the budget finishes in the background.AnalyticsEventSenderposts a pre-assembled body under an explicit payload ID.DEFAULT_CAPACITYgoes from 100 to 1000, matching iOS, since capacity now also bounds what is held on disk.Describe alternatives you've considered
SQLite. Measured as a store and rejected: it costs more per event than an append to a log the SDK already frames, for durability properties the log gets from
writealone. Seesqlite-event-persistence-findings.mdin the spec folder.Fsync on every commit. What durability is defending against is the process dying, not the device losing power. Bytes handed to the kernel survive the former, and an fsync per event would dominate the cost of
track.One shared log for all processes. That is simpler, but one process's startup recovery could rename a log another live process is still appending to. A log per process avoids that without file locks.
Additional context
Tests. 879 unit tests pass locally (
testDebugUnitTest). New coverage includesEventStoreMultiProcessTest(several stores on one directory, recovery, transient read failures, events still counted against capacity while being written),DirectEventProcessorTest(commit points, retry, capacity across memory and disk, persistence failing partway through, oversized frames reported as losses, a loss reported to the right flush across a retry,close()within budget on a stalled disk write),OutboundEventBufferSerializationTest, andSameContextPerContextEventSummarizerTest.Benchmarks.
EventPersistenceBenchmark(JVM and instrumented) measures recording cost per mode. It runs only whenLD_EVENT_BENCH=1, so ordinary test runs are unaffected.Test app. Now configures
EventPersistence.IMMEDIATE, so the events recorded by Eval+Kill now, which tier 2 loses, are recovered and delivered on the next launch. It also gains Over-Refresh Eval, a main-thread stress loop that compares evaluation cost while filling capacity and after it is full.Known open items (from the threading review, to be resolved before this leaves draft):
flushAsync(), and soflushAndWait, does not commit on the caller's thread asflush()does, so aflushAndWaitthat times out can leave events only in memory, even atIMMEDIATE.Version.
EventPersistenceandEventProcessorBuilder.eventPersistencehave no@sincetag yet; one should be added once the release version is known.Note
Overview
Adds tier-3 event durability: analytics events can be appended to an on-disk log (iOS-compatible
EventLogFormat) undergetNoBackupFilesDir(), batched with stable payload IDs, and delivered with spec-aligned retry/backoff instead of relying solely on java-sdk-internal’s in-memoryDefaultEventProcessor.Public surface: new
EventPersistence(DISABLED/DEFERRED/IMMEDIATE) onEventProcessorBuilder;IMMEDIATEcommitstrack/identifyon the caller’s thread.EventProcessorgainsflushAsync()(used byflushAndWait/ crash flush). ExampleMainActivityswitches to blockingLDClient.init(..., INIT_WAIT_SECONDS).Implementation:
DirectEventProcessorowns deferred encoding, commit thresholds, multi-environment parallel flush, loss reporting forflushAsync, and boundedclose(). Supporting pieces includeEventStore,DiskEventLog,BatchLocks(multi-process send vs expiry), andAnalyticsEventSender(explicitX-LaunchDarkly-Payload-ID). Default event processor wiring inComponentsImplis replaced accordingly;launchdarkly-java-sdk-internalis bumped to 1.12.1.Tests/tooling: large JVM/instrumented coverage (
DirectEventProcessor*,EventStore*,flushAndWait, crash handler); optionalEventPersistenceBenchmarkwhenLD_EVENT_BENCH=1; Gradle/orchestrator tweaks for that benchmark.Reviewed by Cursor Bugbot for commit 2405b45. Bugbot is set up for automated code reviews on this repo. Configure here.