Classify SQLite disk-pressure errors and stop retrying them by elirangoshen · Pull Request #816 · Expensify/react-native-onyx

elirangoshen · 2026-07-29T15:21:38Z

Details

Fixes the app-side symptom of Expensify/App#87869 — iOS bursts of NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error (Sentry APP-19Q).

Root cause (reproduced locally on an iOS simulator): when the device disk is full at database (re)open, SQLite fails to ftruncate the OnyxDB-shm WAL-index file to 32KB (errno 28 ENOSPC → SQLITE_IOERR_SHMSIZE), after which every read and write returns disk I/O error until space frees — then it recovers on its own. Two related shapes of the same condition:

unable to open database file (SQLITE_CANTOPEN) — the -shm file cannot be created at all
cannot rollback - no transaction is active — a batch-write failure whose original error was masked by the ROLLBACK itself failing (this one is a mask, not a cause — any error that already aborted the transaction produces it, so it deliberately stays UNKNOWN; nitro-sqlite stops the masking in 9.7.0)

Before this PR these errors fell in the UNKNOWN storage error class: 5 blind retries per operation (futile — the disk is still full) plus a per-operation alert, amplifying the very log storm being reported (~70x amplification seen in prod logs).

This PR:

Adds a new DISK_PRESSURE storage error class (lib/storage/errors.ts)
classifySQLiteError routes the two disk-pressure messages above to it (lib/storage/providers/classifySQLiteError.ts); the rollback message stays UNKNOWN (bounded retry + shape logging) since the real cause is unknowable
retryOperation drops the write (the in-memory cache stays authoritative), skips retries and eviction (neither can free OS-level disk space), and logs a single throttled alert (60s window) plus one storage quota snapshot per burst (lib/OnyxUtils.ts)
On native, the quota snapshot includes getFreeDiskStorage() bytes, giving prod telemetry the free-disk data to confirm disk pressure as the root cause

Related Issues

Expensify/App#97908

Linked E/App PR

Expensify/App#97944

Automated Tests

Added to tests/unit/onyxUtilsTest.ts:

it.each over the two disk-pressure error strings (disk I/O error, unable to open database file) asserting the operation is not retried
A burst test asserting exactly one throttled alert + one storage quota snapshot across a burst of failing operations, and no "retries exhausted" alert

Full unit suite passes (569 tests).

Manual Tests

Regression smoke (no user-facing change in this PR — it only alters error classification/logging):

Sign in and use the app normally: open a report, create an expense, send a message.
Go offline, create an expense, kill and relaunch the app — verify the offline change is still there and syncs when back online.
Verify no crashes or console errors during normal usage.

Author Checklist

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

Screen.Recording.2026-08-06.at.11.28.20.mov

Full device disk surfaces as "disk I/O error" (SQLITE_IOERR, shm sizing on open), "unable to open database file" (SQLITE_CANTOPEN), or "cannot rollback - no transaction is active" (failed ROLLBACK masking the original write error). All were UNKNOWN: 5 futile retries per operation plus per-operation alerts. New DISK_PRESSURE class drops the write (cache stays authoritative), skips retries and eviction (neither frees OS-level space), and logs one throttled alert + storage quota snapshot per burst — the snapshot carries free-disk bytes so telemetry can confirm disk pressure as the root cause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sosek108

Got some comments,

sosek108 · 2026-07-30T08:07:06Z

+    // Full-disk failures around the database files: SQLITE_IOERR (cannot size the -shm file on reopen,
+    // fails reads too), SQLITE_CANTOPEN (cannot create it), and a failed ROLLBACK masking the original
+    // write error. None can succeed until the OS frees space.
+    if (message.includes('disk i/o error') || message.includes('unable to open database file') || message.includes('cannot rollback - no transaction is active')) {


Are you sure that cannot rollback - no transaction is active is 100% related to disk pressure?

its not 100 percent so I edit the comments to reflect it better

Should we remove it then? Looks more like a generic error than disk-specific

sosek108 · 2026-07-30T08:08:03Z

+    /** Filesystem-level failure around the database files (device disk full, or the files cannot be
+     *  created/read). Owner: operation layer — skip retries and eviction (neither can free OS-level
+     *  space) and log one throttled alert + quota snapshot per burst. */
+    DISK_PRESSURE: 'diskPressure',


How this is different from CAPACITY?

their both related to disk full but CAPACITY usually means the onyx keys is full and evict some keys can fix while in DISK_PRESSURE its when os file system is full so there is no way to fix it unless free space.

sosek108 · 2026-07-30T08:11:35Z

+    if (errorClass === StorageErrorClass.DISK_PRESSURE) {
+        const now = Date.now();
+        if (now - lastDiskPressureLogTime < DISK_PRESSURE_LOG_INTERVAL_MS) {
+            return Promise.resolve();
+        }
+        lastDiskPressureLogTime = now;
+        Logger.logAlert(`Disk-pressure storage error; skipping retries. provider: ${Storage.getStorageProvider().name}. message: ${error?.message}. onyxMethod: ${onyxMethod.name}.`);
+        return reportStorageQuota(error);
+    }


The goal is not to send so many logs to VL, right?

This is why we introduced CircuitBreaker.

They solve different problems. The circuit breaker stops the evict → retry loop for CAPACITY errors. DISK_PRESSURE has no loop to stop — we never retry or evict, the write is just dropped. The only thing left to limit is log volume, and the 60s throttle does that. Putting it through the breaker wouldn't work anyway: the breaker's recovery probe is an eviction, which can't fix a full disk.

"cannot rollback - no transaction is active" follows any error that aborted the transaction, not just disk-full, so let it fall through to UNKNOWN's bounded retry; a genuine full disk still surfaces as disk I/O on retry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

"cannot rollback - no transaction is active" is a mask: nitro-sqlite's rollback-on-error replaced the original failure until 9.7.0, so classifying it as disk pressure would be a guess. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 994f2eea18

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

getDatabaseSize ran the PRAGMAs and getFreeDiskStorage in one Promise.all, so the burst snapshot lost its free-disk bytes exactly when the connection was failing. Degrade bytesUsed to -1 instead and always return the filesystem value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fabioh8010 · 2026-08-07T10:01:24Z

+    // "cannot rollback - no transaction is active" lands here deliberately: it is a mask, not a cause —
+    // nitro-sqlite's rollback-on-error replaced the original failure until 9.7.0. UNKNOWN gives it
+    // bounded retry + shape logging instead of a disk-pressure guess.


Can we remove this?

fabioh8010 · 2026-08-07T10:03:29Z

+        // The PRAGMAs run on the SQLite connection — the very thing that's broken during disk pressure —
+        // while getFreeDiskStorage() is filesystem-level and still works. Degrade bytesUsed to -1 instead
+        // of failing the snapshot, so the free-disk bytes (the signal that matters) always get logged.


Another comment that could be simplified e.g. "the very thing that's broken during disk pressure" is useless

I simlified it

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

os-botify · 2026-08-07T15:51:11Z

🚀 Published to npm in 3.0.97 🎉

melvin-bot Bot assigned elirangoshen Jul 29, 2026

Shorten disk-pressure comments

5de9702

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sosek108 reviewed Jul 30, 2026

View reviewed changes

sosek108 approved these changes Jul 30, 2026

View reviewed changes

fabioh8010 approved these changes Aug 5, 2026

View reviewed changes

elirangoshen mentioned this pull request Aug 6, 2026

[TESTING ONLY] Pin react-native-onyx to disk-pressure classification (Onyx PR #816) Expensify/App#97944

Draft

40 tasks

Regenerate API docs for disk-pressure additions

994f2ee

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

elirangoshen marked this pull request as ready for review August 6, 2026 09:34

elirangoshen requested a review from a team as a code owner August 6, 2026 09:34

melvin-bot Bot requested review from inimaga and removed request for a team August 6, 2026 09:34

chatgpt-codex-connector Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread lib/OnyxUtils.ts

elirangoshen mentioned this pull request Aug 6, 2026

Patch react-native-nitro-sqlite so failed ROLLBACKs stop masking real SQLite errors Expensify/App#97954

Open

40 tasks

fabioh8010 reviewed Aug 7, 2026

View reviewed changes

Tighten and remove verbose classifier comments per review

5378d4d

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

inimaga approved these changes Aug 7, 2026

View reviewed changes

inimaga merged commit 26edbbf into Expensify:main Aug 7, 2026
12 of 14 checks passed

fabioh8010 mentioned this pull request Aug 7, 2026

[Tracking] Onyx Improvements Expensify/App#80355

Open

Uh oh!

Conversation

elirangoshen commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

Related Issues

Linked E/App PR

Automated Tests

Manual Tests

Author Checklist

Screenshots/Videos

Uh oh!

sosek108 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elirangoshen Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

os-botify Bot commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elirangoshen commented Jul 29, 2026 •

edited

Loading

elirangoshen Jul 30, 2026 •

edited

Loading