measure(diagnostics): instrument the batch INSERT and the delete fast path; attribute DELETE (index-maint 82%)

Items 2 and 3 of the agreed sequence (plus 4: AesGcmEncryption parity).

Item 3 - the §2 profiler had a blind spot that cost two rounds of ad-hoc probes: the batch INSERT path had NO
stamps at all, and the fixed-width bulk-delete fast path bypasses DeleteRecordsCore, so a 10K-row DELETE
recorded nothing while 100% of its wall time happened there. Added:
  - Table.InsertBatch: Validate (covers validation + serialization) and RowLocate (batch PK probes);
  - the fixed-width bulk-delete fast path: RowLocate (contiguous span read), IndexMaintenance (PK DeleteBulk +
    every loaded hash index), EngineWrite (tombstone / commit-time buffer).
Still uncovered and recorded in the plan: the bulk UPDATE path's index-maintenance block and the SQL/engine
overhead outside the table (parse, plan cache, commit) - which is why a stage report's total is not wall time.

Item 2 - DELETE attributed (10K by PK, 50K rows, one batch transaction):
  arm                             ops/s   wall us/row   in stages   uninstrumented
  plaintext + idx_docs_name     103,495          9.66        4.29             5.37
  at-rest + idx_docs_name       100,209          9.98        5.15             4.83
  at-rest, no index             140,489          7.12        4.01             3.11
Stage shares at-rest: index-maintenance 81.9%, row-locate 16.3%, engine-write (the tombstone) 1.8%.
So DELETE carries NO at-rest tax (1.03x) and the plan's expected culprit (the tombstone) is noise: the cost is
index maintenance, paid per loaded index per row - the delete decodes each indexed column out of the
fixed-width record, which for TEXT means an overflow-arena read (and a decrypt at-rest), purely to compute the
hash key. The lever is therefore bulk/deferred index removal (DeferredIndexUpdater exists), not a cheaper
delete. Recorded in plan section 7 so Phase 5 is planned against the right cause.

Item 4 - AesGcmEncryption (DatabaseFile / PageEncryption / single-file provider) had the same per-call defect
as CryptoService: a fresh AesGcm + OS-CSPRNG nonce per page/blob operation. Now one cipher per instance (they
are long-lived holders) and [random prefix(8)][counter(4)] nonces, throwing instead of wrapping the invocation
field; Dispose releases the cached cipher so reuse-after-dispose cannot encrypt with the pre-clear key.
AesGcmEncryptionNonceTests pins round-trips (array + span + page APIs), nonce uniqueness with the counter
sequence, concurrency of ONE instance (never measured in this project before), and page nonce uniqueness.

Gate: SharpCoreDB.slnx 0 errors; SharpCoreDB.Tests 1857 total / 0 failed / 16 skipped (+4 tests).
Docs: plan section 7 (DELETE measurement + instrumentation coverage) and CHANGELOG Performance.

Author: MPCoreDeveloper

docs/CHANGELOG.md

@@ -35,6 +35,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **`USING DISKANN` silently built a `FlatIndex`** — `VectorQueryOptimizer.BuildIndex` fell through to the `Flat` default arm for the `"DISKANN"` string, so the DDL produced an exact scan behind a DiskANN-shaped name. `DISKANN` now maps to `VectorIndexType.DiskAnn`.
 
 ### Performance
+- **Page/blob encryption no longer imports the key per operation** — `AesGcmEncryption` (used by `DatabaseFile`, `PageEncryption` and the single-file provider, all of which hold one instance for a long time) constructed a fresh `AesGcm` and drew an OS-CSPRNG nonce on every page or blob operation. It now caches one cipher per instance and builds nonces as `[random prefix(8)][operation counter(4)]` with the same uniqueness argument as `CryptoService` (no repeat within an instance, 2^-64 across instances — stronger than a random nonce per call), and it throws rather than wrapping the invocation field. `Dispose` releases the cached cipher, so a reuse-after-dispose cannot silently encrypt with the pre-clear key. Pinned by `AesGcmEncryptionNonceTests`, including that one instance is safe under concurrent use (an assumption this project had never measured) and that page nonces are unique.
+
 
 - **Per-record encryption is no longer dominated by cipher setup** — `CryptoService` constructed a fresh `AesGcm` (key import: **0.69 µs of a 1.34 µs call**, 59%) and drew a nonce from the OS CSPRNG on *every* `Encrypt`/`Decrypt`, while the storage layer calls it once per record **and once per overflow-arena block** — so a row with three TEXT columns paid that setup three to four times, which turned out to be the entire at-rest write tax (measured per-row: +1.5 µs fixed-size = one call; +6.4 µs with TEXT = three to four calls). The cipher is now cached per key (keyed by the full key bytes with structural comparison — never a fingerprint, which could serve a wrong-but-similar key and silently corrupt data), nonces are `[random prefix(8)][operation counter(4)]` built from the counter that already guards GCM exhaustion, and `ResetEncryptionCounter` swaps the prefix *first* so even a reset without key rotation cannot replay a nonce. That construction is a **stronger** uniqueness guarantee than a random nonce per call, whose collision probability grew with the record count. Published, same-machine, four-arm fair-PK comparison: at-rest tax **INSERT 1.90× → 1.34×, UPDATE 1.62× → 1.31×, READ 1.51× → 1.34×, DELETE 1.10× → 1.08×** (INSERT **63,532 → 88,906 ops/s**, +40%; UPDATE +24%; READ +20%), and the gaps vs SQLite in the shipping default posture close to **UPDATE 1.8× → 1.6×, INSERT 2.8× → 2.1×, READ 1.33× → 1.08×**. The security-relevant properties are pinned by `CryptoServiceNonceTests` (nonce uniqueness + counter sequence, key-switch round-trips, thread safety under concurrency, prefix swap on reset, per-instance prefixes).
 

docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md

@@ -1073,6 +1073,34 @@ the SQL path reach the contiguous fast path where one exists.
 **Lower priority than UPDATE** (Direct-API DELETE is already 118.9–132.5K), but it falls out of
 Phase 2's engine work almost for free, so sequence it there rather than as its own project.
 
+**Measured (2026-09-14), and the answer is not the row copy.** The delete path had NO stage coverage at all
+before this measurement — `WritePathProfiler` recorded nothing for a 10K-row DELETE because the fixed-width
+fast path bypasses `DeleteRecordsCore` entirely, so the profiler is now wired into that fast path
+(`RowLocate` around the contiguous span read, `IndexMaintenance` around the PK `DeleteBulk` + every loaded
+hash index, `EngineWrite` around the tombstone/buffer step). With it, on the acceptance DELETE shape (10K
+deletes by PK, 50K rows, one batch transaction):
+
+| arm | ops/s | wall µs/row | in stages | uninstrumented |
+|---|---:|---:|---:|---:|
+| plaintext, with `idx_docs_name` | 103,495 | 9.66 | 4.29 | 5.37 |
+| at-rest, with `idx_docs_name` | 100,209 | 9.98 | 5.15 | 4.83 |
+| at-rest, **without** the index | 140,489 | 7.12 | 4.01 | 3.11 |
+
+Stage shares at-rest: **`index-maintenance` 81.9%** (42.2 ms of 10K deletes), `row-locate` 16.3%,
+`engine-write` (the tombstone) **1.8%**. Two conclusions that change the plan:
+- **the at-rest tax on DELETE is zero (1.03×)** — unlike INSERT/UPDATE, there is no encryption cost to chase;
+- **index maintenance is the cost (a 1.40× win from dropping one index, 82% of the instrumented time)**, and it
+  is paid *per loaded index per row*: the delete decodes each indexed column out of the fixed-width record —
+  which for a TEXT column means an overflow-arena read (and, at-rest, a decrypt) — purely to compute the hash
+  key. So the lever is **not** "make the delete cheaper" but "do the index removal in bulk / deferred", which
+  is what `DeferredIndexUpdater` exists for. The tombstone (the part the plan expected to matter) is noise.
+
+**Instrumentation coverage (2026-09-14, §2).** Added: `Table.InsertBatch` (Validate — covering validation and
+serialization — plus RowLocate around the batch PK probes; the path had none) and the fixed-width bulk-delete
+fast path (RowLocate / IndexMaintenance / EngineWrite). Still uncovered, recorded honestly: the bulk **UPDATE**
+path's index-maintenance block and the SQL/engine overhead outside the table (parse, plan cache, commit) —
+the last column above is exactly that share, and it is why the totals in a stage report are not wall time.
+
 ---
 
 ## 8. Acceptance targets

src/SharpCoreDB/DataStructures/Table.CRUD.cs

@@ -10,6 +10,7 @@ namespace SharpCoreDB.DataStructures;
 using SharpCoreDB.Services;
 using SharpCoreDB.Storage.Hybrid;
 using SharpCoreDB.Optimizations;
+using SharpCoreDB.Diagnostics;
 
 /// <summary>
 /// CRUD operations for Table - Insert, Select, Update, Delete.
@@ -258,11 +259,18 @@ public long[] InsertBatch(List<Dictionary<string, object>> rows)
         if (this.isReadOnly) throw new InvalidOperationException(ReadOnlyInsertError);
 
         // ✅ PHASE 1 OPTIMIZATION: Validate and serialize OUTSIDE lock
+        // §2 instrumentation: this path had NO stage coverage before (measured 2026-09-14), which is why
+        // attributing the at-rest INSERT tax needed ad-hoc probes instead of the profiler. This stamp covers
+        // validation *and* serialization (the method does both).
+        long validateStart = Diagnostics.WritePathProfiler.Stamp();
         var (serializedRows, validatedRows) = ValidateAndSerializeBatchOutsideLock(rows);
-        
+        Diagnostics.WritePathProfiler.Add(Diagnostics.WritePathProfiler.Stage.Validate, validateStart);
+
         // ✅ PHASE 2A FRIDAY: Batch validate primary keys BEFORE critical section
         // This improves cache locality and fails fast on duplicates
+        long pkProbeStart = Diagnostics.WritePathProfiler.Stamp();
         ValidateBatchPrimaryKeysUpfront(validatedRows);
+        Diagnostics.WritePathProfiler.Add(Diagnostics.WritePathProfiler.Stage.RowLocate, pkProbeStart);
 
         // ✅ MINIMAL LOCK: Only for PK check, engine insert, and index updates
         this.rwLock.EnterWriteLock();
@@ -2675,8 +2683,10 @@ private bool HasExplicitNamedIndex(string column)
         // Physical deletes (PageBased marks slots deleted; Columnar/AppendOnly are logical).
         if (StorageMode == StorageMode.PageBased)
         {
+            long pageDeleteStart = WritePathProfiler.Stamp();
             foreach (var (storagePosition, _) in recordsToDelete)
                 engine.Delete(Name, storagePosition);
+            WritePathProfiler.Add(WritePathProfiler.Stage.EngineWrite, pageDeleteStart);
         }
         else if (StorageMode != StorageMode.Columnar)
         {
@@ -2687,6 +2697,7 @@ private bool HasExplicitNamedIndex(string column)
 
         // Primary-key B-tree cleanup: bulk-delete in descending key order (one pass through the
         // rightmost leaf path instead of arbitrary per-row order → fewer separator promotions).
+        long indexStart = WritePathProfiler.Stamp();
         if (this.PrimaryKeyIndex >= 0)
         {
             var pkCol = this.Columns[this.PrimaryKeyIndex];
@@ -2727,6 +2738,8 @@ private bool HasExplicitNamedIndex(string column)
             kvp.Value.RemoveBatchKeys(keys, positions);
         }
 
+        WritePathProfiler.Add(WritePathProfiler.Stage.IndexMaintenance, indexStart);
+
         // Unloaded indexes rebuild lazily (columnar only - page-based indexes stay in sync).
         if (StorageMode == StorageMode.Columnar)
         {
@@ -2744,15 +2757,20 @@ private bool HasExplicitNamedIndex(string column)
                 // Transactional delete: buffer the physical offsets so the in-place marker is
                 // applied at COMMIT (rollback discards the buffer). Durable in O(delete) — the
                 // flush-time full-file rewrite is no longer needed for transactional deletes.
+                long bufferStart = WritePathProfiler.Stamp();
                 foreach (var position in positions.Where(static position => position >= 0))
                 {
                     this.storage.BufferTombstoneForCommit(DataFile, position);
                 }
+
+                WritePathProfiler.Add(WritePathProfiler.Stage.EngineWrite, bufferStart);
             }
             else
             {
                 // Durable DELETE: physically mark the removed records so a reopen skips them.
+                long tombstoneStart = WritePathProfiler.Stamp();
                 TombstoneDeletedPositions(positions);
+                WritePathProfiler.Add(WritePathProfiler.Stage.EngineWrite, tombstoneStart);
 
                 // Legacy variable-length (non-fixed-width) columnar tables keep older stale versions
                 // of a key in the file (UPDATE appends a new version). Tombstoning only the newest
@@ -4042,7 +4060,11 @@ this.storage is null ||
 
         // Resolve every target record's position through the PK B-tree and read + verify the whole
         // contiguous span (one range read) — shared by the UPDATE and DELETE contiguous fast paths.
+        // §2 instrumentation: this fast path bypassed DeleteRecordsCore entirely (measured 2026-09-14: the
+        // profiler recorded nothing for a 10K-row DELETE while 100% of the wall time was spent here).
+        long deleteLocateStart = WritePathProfiler.Stamp();
         var raw = TryReadContiguousFixedWidthRecords(keys, stride, layout, positions, encrypted);
+        WritePathProfiler.Add(WritePathProfiler.Stage.RowLocate, deleteLocateStart);
         if (raw is null)
         {
             return false;
@@ -4052,6 +4074,7 @@ this.storage is null ||
         // loaded hash-index entry, decoding only the indexed columns from the raw fixed-width records
         // (no full-row deserialization). Variable values resolve through the overflow arena, mirroring
         // the fixed-width codec used by the generic path.
+        long deleteIndexStart = WritePathProfiler.Stamp();
         this.Index.DeleteBulk(keys);
 
         var arena = GetOverflowArena();
@@ -4099,19 +4122,26 @@ this.storage is null ||
             hashIdx.RemoveBatchKeys(decoded, positions);
         }
 
+        WritePathProfiler.Add(WritePathProfiler.Stage.IndexMaintenance, deleteIndexStart);
+
         if (this.storage is { IsInTransaction: true })
         {
             // Transactional delete: buffer the physical offsets so the in-place marker is applied
             // at COMMIT (see DeleteRecordsCore — rollback discards the buffer).
+            long deleteBufferStart = WritePathProfiler.Stamp();
             foreach (var position in positions.Where(static position => position >= 0))
             {
                 this.storage.BufferTombstoneForCommit(DataFile, position);
             }
+
+            WritePathProfiler.Add(WritePathProfiler.Stage.EngineWrite, deleteBufferStart);
         }
         else
         {
             // Durable DELETE: physically mark the removed records so a reopen skips them.
+            long deleteTombstoneStart = WritePathProfiler.Stamp();
             TombstoneDeletedPositions(positions);
+            WritePathProfiler.Add(WritePathProfiler.Stage.EngineWrite, deleteTombstoneStart);
         }
 
         Interlocked.Add(ref _cachedRowCount, -count);

src/SharpCoreDB/Services/AesGcmEncryption.cs

@@ -22,6 +22,75 @@ public sealed class AesGcmEncryption(byte[] key, bool disableEncrypt = false) :
     private readonly byte[] _key = disableEncrypt ? [] : [.. key];
     private readonly ArrayPool<byte> _pool = ArrayPool<byte>.Shared;
 
+    /// <summary>
+    /// PERF: one cipher per INSTANCE instead of one per call. Constructing <see cref="AesGcm"/> imports the
+    /// key — measured **0.69 µs of a 1.34 µs Encrypt call, 59%** on this machine — and the holders of this
+    /// class (<c>DatabaseFile</c>, <c>PageEncryption</c> and the single-file provider) keep one instance for
+    /// a long time, so the cache is both safe and effective. GCM keeps no per-instance mutable state, so
+    /// concurrent use of the shared cipher is safe — asserted by a concurrency test, not assumed.
+    /// </summary>
+    private AesGcm? _cipher;
+
+    /// <summary>
+    /// SECURITY: the 64-bit random fixed field of this instance's GCM nonces; every nonce is
+    /// <c>[prefix(8)][counter(4)]</c> with <see cref="_operationCount"/> as the invocation field, capped at
+    /// <see cref="Constants.CryptoConstants.MAX_GCM_OPERATIONS"/> so it cannot wrap. Two instances with the
+    /// same key collide only if their 64-bit prefixes do, independent of how many records they write — a
+    /// random nonce per call had a collision probability that grew with the operation count instead.
+    /// </summary>
+    private byte[] _noncePrefix = CreateNoncePrefix();
+
+    /// <summary>Invocation field of this instance's nonces; see <see cref="_noncePrefix"/>.</summary>
+    private long _operationCount;
+
+    /// <summary>Bytes of the nonce reserved for the per-instance random fixed field.</summary>
+    private const int NoncePrefixSize = 8;
+
+    /// <summary>Draws a fresh 64-bit nonce prefix from the OS CSPRNG.</summary>
+    private static byte[] CreateNoncePrefix()
+    {
+        var prefix = new byte[NoncePrefixSize];
+        RandomNumberGenerator.Fill(prefix);
+        return prefix;
+    }
+
+    /// <summary>The instance's cipher, created once (first use) and disposed with the instance.</summary>
+    private AesGcm Cipher
+    {
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        get => Volatile.Read(ref _cipher) ?? CreateCipher();
+    }
+
+    private AesGcm CreateCipher()
+    {
+        var created = new AesGcm(_key, TagSize);
+        var existing = Interlocked.CompareExchange(ref _cipher, created, null);
+        if (existing is not null)
+        {
+            created.Dispose(); // another thread got there first
+            return existing;
+        }
+
+        return created;
+    }
+
+    /// <summary>
+    /// Writes <c>[prefix(8)][counter(4)]</c> into <paramref name="nonce"/>. Throws rather than wrapping the
+    /// invocation field: a repeated GCM nonce under one key leaks the keystream.
+    /// </summary>
+    private void BuildNonce(Span<byte> nonce)
+    {
+        long count = Interlocked.Increment(ref _operationCount);
+        if (count >= Constants.CryptoConstants.MAX_GCM_OPERATIONS)
+        {
+            throw new InvalidOperationException(
+                $"Encryption limit reached ({count} operations). Key rotation required to prevent GCM nonce reuse.");
+        }
+
+        Volatile.Read(ref _noncePrefix).CopyTo(nonce);
+        System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(nonce[NoncePrefixSize..], (uint)count);
+    }
+
     // Size constants for AES-GCM
     private const int NonceSize = 12; // AesGcm.NonceByteSizes.MaxSize = 12
     private const int TagSize = 16;   // AesGcm.TagByteSizes.MaxSize = 16
@@ -154,12 +223,12 @@ public byte[] Encrypt(byte[] data, ReadOnlySpan<byte> associatedData)
         if (disableEncrypt)
             return data;
 
-        using var aes = new AesGcm(_key, TagSize);
+        var aes = Cipher;
 
         Span<byte> nonce = stackalloc byte[NonceSize];
         Span<byte> tag = stackalloc byte[TagSize];
 
-        RandomNumberGenerator.Fill(nonce);
+        BuildNonce(nonce);
 
         byte[]? cipherArray = null;
         try
@@ -213,7 +282,7 @@ public byte[] Decrypt(byte[] encryptedData, ReadOnlySpan<byte> associatedData)
         if (cipherLength < 0)
             throw new ArgumentException("Invalid encrypted data length", nameof(encryptedData));
 
-        using var aes = new AesGcm(_key, TagSize);
+        var aes = Cipher;
 
         ReadOnlySpan<byte> nonce = encryptedData.AsSpan(0, NonceSize);
         ReadOnlySpan<byte> cipher = encryptedData.AsSpan(NonceSize, cipherLength);
@@ -257,12 +326,12 @@ public int Encrypt(ReadOnlySpan<byte> data, Span<byte> output, ReadOnlySpan<byte
         if (output.Length < totalSize)
             throw new ArgumentException("Output buffer too small", nameof(output));
 
-        using var aes = new AesGcm(_key, TagSize);
+        var aes = Cipher;
 
         Span<byte> nonce = stackalloc byte[NonceSize];
         Span<byte> tag = stackalloc byte[TagSize];
 
-        RandomNumberGenerator.Fill(nonce);
+        BuildNonce(nonce);
 
         byte[]? cipherArray = null;
         try
@@ -338,7 +407,7 @@ public int Decrypt(ReadOnlySpan<byte> encryptedData, Span<byte> output, ReadOnly
         if (output.Length < cipherLength)
             throw new ArgumentException("Output buffer too small", nameof(output));
 
-        using var aes = new AesGcm(_key, TagSize);
+        var aes = Cipher;
 
         var nonce = encryptedData[..NonceSize];
         var cipher = encryptedData.Slice(NonceSize, cipherLength);
@@ -375,12 +444,12 @@ public void EncryptPage(Span<byte> page, ReadOnlySpan<byte> associatedData)
         if (dataSize <= 0)
             throw new ArgumentException("Page buffer too small for encryption overhead", nameof(page));
 
-        using var aes = new AesGcm(_key, TagSize);
+        var aes = Cipher;
 
         Span<byte> nonce = stackalloc byte[NonceSize];
         Span<byte> tag = stackalloc byte[TagSize];
 
-        RandomNumberGenerator.Fill(nonce);
+        BuildNonce(nonce);
 
         byte[]? tempArray = null;
         try
@@ -432,7 +501,7 @@ public void DecryptPage(Span<byte> page, ReadOnlySpan<byte> associatedData)
         if (cipherLength <= 0)
             throw new ArgumentException("Page buffer too small for decryption", nameof(page));
 
-        using var aes = new AesGcm(_key, TagSize);
+        var aes = Cipher;
 
         var nonce = page[..NonceSize];
         var cipher = page.Slice(NonceSize, cipherLength);
@@ -461,7 +530,13 @@ public void DecryptPage(Span<byte> page, ReadOnlySpan<byte> associatedData)
     [MethodImpl(MethodImplOptions.AggressiveInlining)]
     public void Dispose()
     {
+        // Dispose the cached cipher as well: leaving it alive would let a reuse-after-dispose silently
+        // encrypt with the pre-clear key, which is exactly the trap the key clearing below exists to avoid.
+        Interlocked.Exchange(ref _cipher, null)?.Dispose();
+
         if (_key.Length > 0)
             Array.Clear(_key);
+
+        Array.Clear(_noncePrefix);
     }
 }