Dedicated rollup writer threads, a storage format reference, and rollup code cleanup.

README.md

@@ -233,6 +233,7 @@ Use `analytics.GetStatistics()` and host logging to monitor the engine. See [dep
 | [Product analytics][product] | [Secondary indexes][indexes] |
 | [Service telemetry][telemetry] | [HTTP API reference][http-api] |
 | [Sites and tenants][tenants] | [Extensibility][extensibility] |
+| [Deployment and lifecycle][deployment] | [Storage format][storage-format] |
 
 ## Build and contribute
 
@@ -291,6 +292,7 @@ Released under the [MIT license](https://github.com/ZoneTree/ZoneTree.Analytics/
 [accuracy]: https://analytics.zonetree.dev/docs/concepts/time-and-accuracy/
 [architecture]: https://analytics.zonetree.dev/docs/concepts/architecture/
 [configuration]: https://analytics.zonetree.dev/docs/reference/configuration/
+[storage-format]: https://analytics.zonetree.dev/docs/reference/storage-format/
 [web-reports]: https://analytics.zonetree.dev/docs/guides/web-reports/
 [rollups]: https://analytics.zonetree.dev/docs/performance/rollups/
 [product]: https://analytics.zonetree.dev/docs/guides/product-analytics/

apps/ZoneTree.Analytics.Dashboard/server/DashboardAnalyticsOptions.cs

@@ -34,7 +34,7 @@ public static AnalyticsOptions CreatePerformanceSamples(string directory, TimeZo
       int workers, ILogger logger = null)
   {
     var options = Create(directory, PerformanceSite, timeZone, logger);
-    options.Ingestion.WorkerCount = workers;
+    SetWorkers(options, workers);
     options.Rollups.ValueAggregates["load_time"] = ValueAggregates.Percentiles;
     return options;
   }
@@ -43,7 +43,7 @@ public static AnalyticsOptions CreateBenchmark(string directory, TimeZoneInfo ti
       int workers, bool percentiles, ILogger logger = null)
   {
     var options = Create(directory, BenchmarkSite, timeZone, logger);
-    options.Ingestion.WorkerCount = workers;
+    SetWorkers(options, workers);
     options.Ingestion.QueueFullBehavior = QueueFullBehavior.Wait;
     if (percentiles) options.Rollups.ValueAggregates["load_time"] = ValueAggregates.Percentiles;
     return options;
@@ -54,6 +54,14 @@ public static AnalyticsOptions CreateBenchmarkReport(string directory, string ti
       bool percentiles, ILogger logger = null) =>
       CreateBenchmark(directory, TimeZoneInfo.FindSystemTimeZoneById(timeZoneId), 1, percentiles, logger);
 
+  // Generated ingestion runs as many rollup writers as workers, so the rows that every worker
+  // buffers are merged and written in parallel at the same scale.
+  static void SetWorkers(AnalyticsOptions options, int workers)
+  {
+    options.Ingestion.WorkerCount = workers;
+    options.Ingestion.RollupWriterCount = workers;
+  }
+
   static AnalyticsOptions Create(string directory, string site, TimeZoneInfo timeZone, ILogger logger)
   {
     var options = new AnalyticsOptions();

docs-site/content/docs/performance/tuning/tuning.mdx

@@ -20,6 +20,7 @@ Measure a consistent Release build. Include representative filtered and unfilter
 | Setting | Tradeoff |
 | --- | --- |
 | WorkerCount | More processing concurrency, with larger combined per-worker budgets |
+| RollupWriterCount | More parallel rollup writes, with one more thread each; rows of every worker are merged before they are written |
 | QueueCapacity | More burst capacity, with more queued memory |
 | MaxBufferedRollupRows | More aggregation state held per worker before writing |
 | Sessions.MaxCachedSessions | More active-session state held per worker |

docs-site/content/docs/reference/configuration/configuration.mdx

@@ -102,6 +102,7 @@ The dashboard extends raw retention and enables selected aggregates and indexes.
 | --- | --- | --- |
 | `QueueCapacity` | `int` | Maximum number of queued events per worker. Default is 100,000. |
 | `WorkerCount` | `int` | Number of background ingestion workers. Events are routed to workers by visitor, so each visitor's events are processed in order. Default is 1, which handles well over a hundred thousand events per second. |
+| `RollupWriterCount` | `int` | Number of dedicated rollup writer threads, 1 to 64. Each writer owns a share of the rollup rows: it merges the rows that all workers buffered for them and writes each row once per flush, instead of once per worker. Default is 1, which suits single-processor machines; with several workers on several cores, more writers write rows in parallel. |
 | `FlushInterval` | `TimeSpan` | How often buffered aggregates and session states are written to storage. Rollup-based queries lag behind ingestion by at most this interval. Default is 2 seconds. Must not exceed approximately 49.7 days, the timer's maximum period. |
 | `MaxBufferedRollupRows` | `int` | Forces a flush when a worker buffers more distinct rollup rows than this. Default is 100,000. |
 | `QueueFullBehavior` | `QueueFullBehavior` | Behavior of `AnalyticsEngine.Track` when the queue is full. Default is `QueueFullBehavior.Drop`, so request threads never block. |

docs-site/content/docs/reference/engine/engine.mdx

@@ -80,7 +80,7 @@ Partition record counts are approximate and may be null for partitions not opene
 
 ## Errors and cancellation
 
-`AnalyticsException` is the base for engine-specific errors. `AnalyticsConfigurationException` reports incompatible options or dataset configuration. `AnalyticsQueryException` reports unanswerable or bounded queries; `AnalyticsQueryTimeoutException` is its timeout subclass.
+`AnalyticsException` is the base for engine-specific errors. `AnalyticsConfigurationException` reports incompatible options or dataset configuration. `AnalyticsQueryException` reports unanswerable or bounded queries; `AnalyticsQueryTimeoutException` is its timeout subclass. `AnalyticsStorageException` reports a flush whose rollup rows could not be written; its inner exception is the storage failure, and the rows are retried by later flushes.
 
 Invalid arguments use standard argument exceptions. Caller cancellation uses the cancellation token; configured query timeout additionally protects expensive report execution.
 
@@ -132,6 +132,8 @@ than ZoneTree and is not exposed.
 
 For shared indexes, restrict iterators using the returned logical `KeyPrefix`. Each accessor exposes a logical namespace within the shared physical store.
 
+See the [storage format reference](../storage-format/storage-format.mdx) for directory layouts, exact key and payload encodings, metadata generations, and recovery behavior.
+
 ## Related pages
 
 - [First events and reports](../../getting-started/first-events/first-events.mdx)

docs-site/content/docs/reference/index.mdx

@@ -11,7 +11,7 @@ Look up exact API contracts and defaults. These pages complement the worked exam
 
 ## Find the contract
 
-Use the [engine API](./engine/engine.mdx) for entry points, [query model](./query-model/query-model.mdx) for calculation semantics, [configuration](./configuration/configuration.mdx) for settings, and [HTTP API](./http-api/http-api.mdx) for integration.
+Use the [engine API](./engine/engine.mdx) for entry points, [query model](./query-model/query-model.mdx) for calculation semantics, [configuration](./configuration/configuration.mdx) for settings, [HTTP API](./http-api/http-api.mdx) for integration, and [storage format](./storage-format/storage-format.mdx) for persisted layouts and recovery contracts.
 
 ## In this section
 
@@ -20,6 +20,7 @@ Use the [engine API](./engine/engine.mdx) for entry points, [query model](./quer
   <DocCard number="02" title="Query model reference" summary="Query fields, metrics, filters, time ranges, and execution-plan results." href="/docs/reference/query-model/" icon="tabler:code"/>
   <DocCard number="03" title="Configuration reference" summary="Look up engine, storage, retention, privacy, and web integration settings." href="/docs/reference/configuration/" icon="tabler:code"/>
   <DocCard number="04" title="HTTP API reference" summary="Send collection and reporting requests, interpret JSON results, and handle API limits." href="/docs/reference/http-api/" icon="tabler:code"/>
+  <DocCard number="05" title="Storage format" summary="Directory layout, binary keys and values, metadata, partition routing, and recovery contracts." href="/docs/reference/storage-format/" icon="tabler:database"/>
 </Grid>
 
 ## Continue exploring