diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..53da4a4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,97 @@ +name: Bug Report +description: Something isn't working — SQLITE_BUSY still occurring, incorrect behavior, crash +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to file a bug report. The more detail you provide, the faster we can help. + + - type: input + id: package_version + attributes: + label: Package Version + description: Which version of EntityFrameworkCore.Sqlite.Concurrency? + placeholder: e.g. 10.1.0 + validations: + required: true + + - type: input + id: dotnet_version + attributes: + label: .NET Version + description: Output of `dotnet --version` + placeholder: e.g. 10.0.100 + validations: + required: true + + - type: input + id: os + attributes: + label: Operating System + placeholder: e.g. Windows 11, Ubuntu 24.04, macOS 15 + validations: + required: true + + - type: textarea + id: error_message + attributes: + label: Error Message / Stack Trace + description: The full exception message and stack trace. Remove any sensitive connection string details. + render: text + validations: + required: true + + - type: textarea + id: connection_string + attributes: + label: Connection String (redacted) + description: Your connection string with any sensitive values replaced with `***` + placeholder: "Data Source=app.db" + validations: + required: true + + - type: textarea + id: registration + attributes: + label: Registration Code + description: How you registered the context (AddConcurrentSqliteDbContext, AddConcurrentSqliteDbContextFactory, UseSqliteWithConcurrency, etc.) + render: csharp + validations: + required: true + + - type: textarea + id: reproduction + attributes: + label: Reproduction Steps + description: Minimal steps to reproduce. A failing test or small console app is ideal. + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What should have happened? + validations: + required: true + + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened? + validations: + required: true + + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: I am not using `Cache=Shared` in my connection string + required: false + - label: I am using `IDbContextFactory` for concurrent workloads (not a shared `DbContext`) + required: false + - label: The database is on local disk (not NFS/SMB/network path) + required: false diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..7b649da --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,41 @@ +name: Feature Request +description: Suggest a new feature or improvement +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: | + Thanks for suggesting an improvement. Please describe the problem you're trying to solve — not just the solution. + + - type: textarea + id: problem + attributes: + label: Problem Description + description: What problem are you hitting? What can't you do today? + placeholder: "I'm trying to... but currently..." + validations: + required: true + + - type: textarea + id: solution + attributes: + label: Proposed Solution + description: What would you like to see added or changed? + validations: + required: true + + - type: textarea + id: alternatives + attributes: + label: Alternatives Considered + description: Any workarounds or alternative approaches you've tried? + validations: + required: false + + - type: textarea + id: context + attributes: + label: Additional Context + description: Any other context, links, or examples that would help. + validations: + required: false diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..759adee --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,34 @@ +## Summary + + + +## Type of Change + +- [ ] Bug fix (non-breaking — fixes incorrect behavior) +- [ ] New feature (non-breaking — adds functionality) +- [ ] Breaking change (changes existing public API or behavior) +- [ ] Documentation update +- [ ] CI / build change +- [ ] Performance improvement + +## Testing + +- [ ] New tests added covering the change +- [ ] All existing tests pass (`dotnet test EFCore.Sqlite.Concurrency.Test/`) +- [ ] If performance-sensitive: benchmarks run and results included below + +## Checklist + +- [ ] `CHANGELOG.md` updated under `[Unreleased]` +- [ ] XML doc comments added/updated for all public API changes +- [ ] No `Cache=Shared` introduced in any connection string +- [ ] `packages.lock.json` updated if dependencies changed (`dotnet restore`) +- [ ] Breaking changes documented (or none — confirm below) + +## Breaking Changes + + + +## Benchmark Results (if applicable) + + diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e7eb7fe..d5c1e70 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -23,10 +23,18 @@ jobs: cache-dependency-path: EntityFrameworkCore.Sqlite.Concurrency/packages.lock.json - name: Restore dependencies - run: dotnet restore EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj + run: | + dotnet restore EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj + dotnet restore EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj - - name: Build + - name: Build library run: dotnet build EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release --no-restore + - name: Build test project + run: dotnet build EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj -c Release --no-restore + + - name: Test + run: dotnet test EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj --framework net10.0 -c Release --no-build + - name: Pack - run: dotnet pack EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release --no-build -o out \ No newline at end of file + run: dotnet pack EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release --no-build -o out \ No newline at end of file diff --git a/.gitignore b/.gitignore index 9e25fe7..772e9f9 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,13 @@ EntityFrameworkCore.Sqlite.Concurrency/bin/ Tests/bin/ Tests/obj/ + +EFCore.Sqlite.Concurrency.Test/bin/ + +EFCore.Sqlite.Concurrency.Test/obj/ + +EFCore.Sqlite.Concurrency.Benchmarks/bin/ + +EFCore.Sqlite.Concurrency.Benchmarks/obj/ + +BenchmarkDotNet.Artifacts/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..1b2ad7e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,100 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +--- + +## [Unreleased] + +--- + +## [10.1.0] - 2026-07-05 + +### Added +- **Channel-based write queue** — replaces `SemaphoreSlim` with `Channel` + single + background writer. Eliminates thundering-herd wakeup storms under high concurrent write load. + All callers park in FIFO order; zero API change required. +- **`WriteQueueCapacity` option** — new `int?` on `SqliteConcurrencyOptions` (default `null` = unbounded). + When set, creates a `BoundedChannel` with `BoundedChannelFullMode.Wait` for back-pressure control. +- **netstandard 2.0 TFM** — `SqliteConnectionEnhancer`, `SqliteWriteQueue`, and `SqliteConcurrencyOptions` + now available on any runtime supporting netstandard 2.0. EF Core APIs remain net10.0-only. +- **BenchmarkDotNet project** — `EFCore.Sqlite.Concurrency.Benchmarks/` with reproducible benchmarks + for bulk insert, concurrent writes, and parallel reads. +- **Stress tests** — `EFCore.Sqlite.Concurrency.Test/` with four correctness-under-load xUnit tests + covering all write paths at 50 concurrent writers. + +### Fixed +- `ThreadSafeSqliteContext.ExecuteWriteAsync` always used default `SqliteConcurrencyOptions` instead + of reading the registered interceptor's configured options. Fixed via `TryGetInterceptor` lookup. +- `BulkInsertSafeAsync` used `Skip(n).Take(1000)` — O(n²) in LINQ-to-objects for large lists. + Replaced with `Chunk(1000)` for O(n) enumeration. +- `BulkInsertSafeAsync` did not call `ChangeTracker.Clear()` between batches, causing unbounded + memory growth during large imports. Fixed. + +--- + +## [10.0.3] - 2026-01-15 + +### Added +- `AddConcurrentSqliteDbContextFactory` — registers `IDbContextFactory` with all concurrency + settings. Recommended pattern for background services, `Task.WhenAll`, and `Channel` consumers. +- Structured logging for `SQLITE_BUSY*` events via `ILoggerFactory` (resolved from DI automatically). +- `GetWalCheckpointStatusAsync` — runs `PRAGMA wal_checkpoint(PASSIVE)` and returns a typed + `WalCheckpointStatus` (IsBusy, TotalWalFrames, CheckpointedFrames, CheckpointProgress). +- `TryReleaseMigrationLockAsync` — detects and optionally clears a stale `__EFMigrationsLock` row + left by a crashed migration process. +- `SynchronousMode` option — configures `PRAGMA synchronous` (Off / Normal / Full / Extra). +- `UpgradeTransactionsToImmediate` option — opt out of the `BEGIN → BEGIN IMMEDIATE` rewrite. + +### Fixed +- `SQLITE_BUSY_SNAPSHOT` (extended code 517) now correctly restarts the full operation lambda + instead of retrying the same statement — the only correct fix for a stale WAL read snapshot. +- Exponential backoff now uses full jitter (`[baseDelay, 2×baseDelay]`) to prevent thundering herd. + +--- + +## [10.0.2] - 2025-12-01 + +### Added +- Startup validation for `SqliteConcurrencyOptions` — invalid values (`MaxRetryAttempts ≤ 0`, + negative `BusyTimeout`) now throw `ArgumentOutOfRangeException` at startup. + +### Fixed +- `Cache=Shared` in connection strings now throws `ArgumentException` at startup (was silently + breaking WAL mode semantics in prior versions). + +--- + +## [10.0.1] - 2025-11-01 + +### Fixed +- Corrected minor bug causing READ-ONLY lock file error on startup (`PrepareForConnectionOpen` + now strips `FileAttributes.ReadOnly` and deletes stale `.db-shm` files). + +--- + +## [10.0.0] - 2025-10-01 + +### Added +- Initial release. +- `UseSqliteWithConcurrency` — drop-in replacement for `UseSqlite` with WAL mode, write + serialization, `BEGIN IMMEDIATE` transaction upgrade, and exponential-backoff retry. +- `AddConcurrentSqliteDbContext` — DI registration for request-scoped workloads. +- `BulkInsertOptimizedAsync` — batched bulk insert with WAL-optimized transactions. +- `SaveChangesSerializedAsync` — explicit serialized save with retry. +- `ExecuteWithRetryAsync` — generic operation retry wrapper. +- `ThreadSafeSqliteContext` — optional base `DbContext` with write serialization built in. +- `ThreadSafeFactory` — DI-free factory for non-DI environments. +- Per-database `SemaphoreSlim` write lock via `SqliteConnectionEnhancer`. +- `SqliteDiagnostics` — opt-in Spectre.Console diagnostics UI (`-p:IncludeSpectre=true`). +- `MemoryPackExtensions` — opt-in MemoryPack serialization (`-p:IncludeMemoryPack=true`). + +[Unreleased]: https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/compare/v10.1.0...HEAD +[10.1.0]: https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/compare/v10.0.3...v10.1.0 +[10.0.3]: https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/compare/v10.0.2...v10.0.3 +[10.0.2]: https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/compare/v10.0.1...v10.0.2 +[10.0.1]: https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/compare/v10.0.0...v10.0.1 +[10.0.0]: https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/releases/tag/v10.0.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..c1965b6 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,101 @@ +# Contributing to EntityFrameworkCore.Sqlite.Concurrency + +Thank you for your interest in contributing. This document covers prerequisites, build and test commands, benchmark instructions, and the PR process. + +--- + +## Prerequisites + +- [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0) +- Git +- Any editor (Visual Studio, Rider, VS Code with C# Dev Kit) + +--- + +## Build + +```bash +# Restore (locked packages enforced) +dotnet restore EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj + +# Build (Release) +dotnet build EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release + +# Pack NuGet package +dotnet pack EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release --no-build -o out +``` + +Optional feature flags: + +```bash +dotnet build ... -p:IncludeMemoryPack=true # enables MemoryPackExtensions.cs +dotnet build ... -p:IncludeSpectre=true # enables SqliteDiagnostics.cs +``` + +--- + +## Tests + +```bash +dotnet test EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj --framework net10.0 -c Release +``` + +The test suite contains four correctness-under-load stress tests covering all write paths at 50 concurrent writers with isolated temp databases. All four must pass before submitting a PR. + +--- + +## Benchmarks + +The `EFCore.Sqlite.Concurrency.Benchmarks/` project uses [BenchmarkDotNet](https://benchmarkdotnet.org/) to measure real performance. Always run in Release mode. + +```bash +cd EFCore.Sqlite.Concurrency.Benchmarks + +# Full precision run — used for README/docs numbers (slow: 10–30 min) +dotnet run -c Release -- --filter * + +# Quick developer run — 3 iterations, good for relative comparisons +dotnet run -c Release -- --job short --filter * + +# Run a specific benchmark class +dotnet run -c Release -- --filter *BulkInsert* +dotnet run -c Release -- --filter *Concurrent* +dotnet run -c Release -- --filter *Read* +``` + +Results are written to `BenchmarkDotNet.Artifacts/results/` as markdown, CSV, and JSON. + +--- + +## PR Process + +1. **Fork** the repository and create a feature branch from `main`. +2. **Write tests** — all new behavior must be covered by the stress test project or a new test class. +3. **Run the full test suite** — `dotnet test` must pass with 0 failures. +4. **Update `CHANGELOG.md`** — add your change under `[Unreleased]` in the appropriate subsection (Added / Fixed / Changed). +5. **Update XML doc comments** — all public APIs must have `` and `` tags. Missing doc comments produce build warnings that are treated as errors in CI. +6. **Submit PR** — use the PR template. One PR per concern. + +--- + +## Commit Message Convention + +``` +type: short description (imperative, under 72 chars) + +Optional body explaining why, not what. +``` + +Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `perf`, `test`, `bump` + +--- + +## Architecture Notes + +See [CLAUDE.md](CLAUDE.md) for a full component map and key design decisions. + +Key invariants to preserve: +- **One write queue per database file** — keyed by normalized connection string in `SqliteConnectionEnhancer._writeQueues` +- **`IsWriteLockHeld` reentrancy guard** — must be set `true` by the queue writer before executing each request, and `false` in `finally` +- **Options equality excludes `LoggerFactory`** — enforced in `SqliteConcurrencyOptions.Equals` +- **`Cache=Shared` must be rejected at startup** — incompatible with WAL mode diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/BaselineSqliteContext.cs b/EFCore.Sqlite.Concurrency.Benchmarks/BaselineSqliteContext.cs new file mode 100644 index 0000000..9511c19 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/BaselineSqliteContext.cs @@ -0,0 +1,19 @@ +using Microsoft.EntityFrameworkCore; + +namespace EFCore.Sqlite.Concurrency.Benchmarks; + +/// Plain UseSqlite context — baseline comparison without concurrency features. +public class BaselineSqliteContext : DbContext +{ + private readonly string _connectionString; + + public BaselineSqliteContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder options) => + options.UseSqlite(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity(e => e.HasKey(x => x.Id)); +} diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/BenchmarkEntity.cs b/EFCore.Sqlite.Concurrency.Benchmarks/BenchmarkEntity.cs new file mode 100644 index 0000000..f73da29 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/BenchmarkEntity.cs @@ -0,0 +1,8 @@ +namespace EFCore.Sqlite.Concurrency.Benchmarks; + +public class BenchmarkEntity +{ + public Guid Id { get; set; } = Guid.NewGuid(); + public string Payload { get; set; } = string.Empty; + public int Value { get; set; } +} diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/BulkInsertBenchmarks.cs b/EFCore.Sqlite.Concurrency.Benchmarks/BulkInsertBenchmarks.cs new file mode 100644 index 0000000..9e5f5e3 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/BulkInsertBenchmarks.cs @@ -0,0 +1,84 @@ +using BenchmarkDotNet.Attributes; +using EntityFrameworkCore.Sqlite.Concurrency; + +namespace EFCore.Sqlite.Concurrency.Benchmarks; + +/// +/// Compares BulkInsertOptimizedAsync against naive per-entity SaveChanges. +/// Both benchmarks use a single writer — this measures batching + WAL gains only, +/// not write serialization (which has no baseline to compare against). +/// +[MemoryDiagnoser] +[SimpleJob] +public class BulkInsertBenchmarks +{ + [Params(1_000, 10_000)] + public int EntityCount { get; set; } + + private string _baselinePath = string.Empty; + private string _concurrentPath = string.Empty; + + [GlobalSetup] + public void Setup() + { + _baselinePath = Path.Combine(Path.GetTempPath(), $"bench_baseline_{Guid.NewGuid():N}.db"); + _concurrentPath = Path.Combine(Path.GetTempPath(), $"bench_concurrent_{Guid.NewGuid():N}.db"); + + using var baseline = new BaselineSqliteContext($"Data Source={_baselinePath}"); + baseline.Database.EnsureCreated(); + + using var concurrent = new ConcurrentSqliteContext($"Data Source={_concurrentPath}"); + concurrent.Database.EnsureCreated(); + } + + [GlobalCleanup] + public void Cleanup() + { + foreach (var ext in new[] { "", "-wal", "-shm" }) + { + TryDelete(_baselinePath + ext); + TryDelete(_concurrentPath + ext); + } + } + + [IterationSetup] + public void IterationSetup() + { + // Clear rows between iterations so counts stay stable + using var b = new BaselineSqliteContext($"Data Source={_baselinePath}"); + b.Entities.RemoveRange(b.Entities); + b.SaveChanges(); + + using var c = new ConcurrentSqliteContext($"Data Source={_concurrentPath}"); + c.Entities.RemoveRange(c.Entities); + c.SaveChanges(); + } + + /// Baseline: one SaveChanges call per entity — the common antipattern. + [Benchmark(Baseline = true, Description = "Plain EF Core (SaveChanges per entity)")] + public async Task Baseline_SaveChangesPerEntity() + { + await using var ctx = new BaselineSqliteContext($"Data Source={_baselinePath}"); + for (var i = 0; i < EntityCount; i++) + { + ctx.Entities.Add(new BenchmarkEntity { Payload = $"item-{i}", Value = i }); + await ctx.SaveChangesAsync(); + } + } + + /// Package: BulkInsertOptimizedAsync — batched, WAL-tuned, ChangeTracker cleared. + [Benchmark(Description = "BulkInsertOptimizedAsync")] + public async Task Package_BulkInsertOptimizedAsync() + { + await using var ctx = new ConcurrentSqliteContext($"Data Source={_concurrentPath}"); + var entities = Enumerable.Range(0, EntityCount) + .Select(i => new BenchmarkEntity { Payload = $"item-{i}", Value = i }) + .ToList(); + await ctx.BulkInsertOptimizedAsync(entities); + } + + private static void TryDelete(string path) + { + if (File.Exists(path)) try { File.Delete(path); } catch { } + } +} diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/ConcurrentSqliteContext.cs b/EFCore.Sqlite.Concurrency.Benchmarks/ConcurrentSqliteContext.cs new file mode 100644 index 0000000..fc66d5e --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/ConcurrentSqliteContext.cs @@ -0,0 +1,20 @@ +using EntityFrameworkCore.Sqlite.Concurrency; +using Microsoft.EntityFrameworkCore; + +namespace EFCore.Sqlite.Concurrency.Benchmarks; + +/// UseSqliteWithConcurrency context for benchmarking. +public class ConcurrentSqliteContext : DbContext +{ + private readonly string _connectionString; + + public ConcurrentSqliteContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder options) => + options.UseSqliteWithConcurrency(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity(e => e.HasKey(x => x.Id)); +} diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/ConcurrentWriteBenchmarks.cs b/EFCore.Sqlite.Concurrency.Benchmarks/ConcurrentWriteBenchmarks.cs new file mode 100644 index 0000000..a01a717 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/ConcurrentWriteBenchmarks.cs @@ -0,0 +1,59 @@ +using BenchmarkDotNet.Attributes; +using EntityFrameworkCore.Sqlite.Concurrency; + +namespace EFCore.Sqlite.Concurrency.Benchmarks; + +/// +/// Measures throughput of concurrent writers via SaveChangesSerializedAsync. +/// No baseline: plain SQLite throws SQLITE_BUSY under this load — the baseline IS the error. +/// +[MemoryDiagnoser] +[SimpleJob] +public class ConcurrentWriteBenchmarks +{ + [Params(10, 50)] + public int ConcurrentWriters { get; set; } + + private const int RowsPerWriter = 10; + private string _dbPath = string.Empty; + + [GlobalSetup] + public void Setup() + { + _dbPath = Path.Combine(Path.GetTempPath(), $"bench_concurrent_writes_{Guid.NewGuid():N}.db"); + using var ctx = new ConcurrentSqliteContext($"Data Source={_dbPath}"); + ctx.Database.EnsureCreated(); + } + + [GlobalCleanup] + public void Cleanup() + { + foreach (var ext in new[] { "", "-wal", "-shm" }) + if (File.Exists(_dbPath + ext)) try { File.Delete(_dbPath + ext); } catch { } + } + + [IterationSetup] + public void IterationSetup() + { + using var ctx = new ConcurrentSqliteContext($"Data Source={_dbPath}"); + ctx.Entities.RemoveRange(ctx.Entities); + ctx.SaveChanges(); + } + + /// N concurrent writers each writing RowsPerWriter rows via the write queue. + [Benchmark(Description = "Concurrent SaveChangesSerializedAsync")] + public async Task ConcurrentSaveChangesSerializedAsync() + { + var cs = $"Data Source={_dbPath}"; + var tasks = Enumerable.Range(0, ConcurrentWriters).Select(async writerIdx => + { + await using var ctx = new ConcurrentSqliteContext(cs); + var entities = Enumerable.Range(0, RowsPerWriter) + .Select(i => new BenchmarkEntity { Payload = $"w{writerIdx}-r{i}", Value = writerIdx * 100 + i }) + .ToList(); + ctx.Entities.AddRange(entities); + await ctx.SaveChangesSerializedAsync(); + }); + await Task.WhenAll(tasks); + } +} diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/EFCore.Sqlite.Concurrency.Benchmarks.csproj b/EFCore.Sqlite.Concurrency.Benchmarks/EFCore.Sqlite.Concurrency.Benchmarks.csproj new file mode 100644 index 0000000..f018ec6 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/EFCore.Sqlite.Concurrency.Benchmarks.csproj @@ -0,0 +1,21 @@ + + + + Exe + net10.0 + enable + enable + true + true + + + + + + + + + + + + diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/Program.cs b/EFCore.Sqlite.Concurrency.Benchmarks/Program.cs new file mode 100644 index 0000000..5f2c50d --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/Program.cs @@ -0,0 +1,9 @@ +using BenchmarkDotNet.Running; +using EFCore.Sqlite.Concurrency.Benchmarks; + +// Usage: +// dotnet run -c Release — full precision run (slow, for README numbers) +// dotnet run -c Release -- --job short — quick developer run (3 iterations) +// dotnet run -c Release -- --filter *Bulk* — run only BulkInsertBenchmarks + +BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args); diff --git a/EFCore.Sqlite.Concurrency.Benchmarks/ReadBenchmarks.cs b/EFCore.Sqlite.Concurrency.Benchmarks/ReadBenchmarks.cs new file mode 100644 index 0000000..5f3ae5d --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Benchmarks/ReadBenchmarks.cs @@ -0,0 +1,78 @@ +using BenchmarkDotNet.Attributes; +using Microsoft.EntityFrameworkCore; + +namespace EFCore.Sqlite.Concurrency.Benchmarks; + +/// +/// Compares parallel read throughput: WAL mode (package) vs default journal mode (baseline). +/// Baseline uses plain UseSqlite without WAL — reads block behind any active write. +/// Package uses WAL mode — reads run concurrently with writes. +/// +[MemoryDiagnoser] +[SimpleJob] +public class ReadBenchmarks +{ + private const int ParallelReaders = 20; + private const int SeedRows = 500; + + private string _baselinePath = string.Empty; + private string _concurrentPath = string.Empty; + + [GlobalSetup] + public void Setup() + { + _baselinePath = Path.Combine(Path.GetTempPath(), $"bench_read_baseline_{Guid.NewGuid():N}.db"); + _concurrentPath = Path.Combine(Path.GetTempPath(), $"bench_read_concurrent_{Guid.NewGuid():N}.db"); + + // Seed both databases with SeedRows rows + using (var b = new BaselineSqliteContext($"Data Source={_baselinePath}")) + { + b.Database.EnsureCreated(); + b.Entities.AddRange(Enumerable.Range(0, SeedRows) + .Select(i => new BenchmarkEntity { Payload = $"seed-{i}", Value = i })); + b.SaveChanges(); + } + + using (var c = new ConcurrentSqliteContext($"Data Source={_concurrentPath}")) + { + c.Database.EnsureCreated(); + c.Entities.AddRange(Enumerable.Range(0, SeedRows) + .Select(i => new BenchmarkEntity { Payload = $"seed-{i}", Value = i })); + c.SaveChanges(); + } + } + + [GlobalCleanup] + public void Cleanup() + { + foreach (var path in new[] { _baselinePath, _concurrentPath }) + foreach (var ext in new[] { "", "-wal", "-shm" }) + if (File.Exists(path + ext)) try { File.Delete(path + ext); } catch { } + } + + /// Baseline: parallel reads using default journal mode (no WAL). + [Benchmark(Baseline = true, Description = "Plain EF Core parallel reads")] + public async Task Baseline_ParallelReads() + { + var cs = $"Data Source={_baselinePath}"; + var tasks = Enumerable.Range(0, ParallelReaders).Select(async _ => + { + await using var ctx = new BaselineSqliteContext(cs); + return await ctx.Entities.CountAsync(); + }); + await Task.WhenAll(tasks); + } + + /// Package: parallel reads using WAL mode — reads never block behind writes. + [Benchmark(Description = "WAL-mode parallel reads")] + public async Task Package_ParallelReads() + { + var cs = $"Data Source={_concurrentPath}"; + var tasks = Enumerable.Range(0, ParallelReaders).Select(async _ => + { + await using var ctx = new ConcurrentSqliteContext(cs); + return await ctx.Entities.CountAsync(); + }); + await Task.WhenAll(tasks); + } +} diff --git a/EFCore.Sqlite.Concurrency.Test/ConcurrencyStressTests.cs b/EFCore.Sqlite.Concurrency.Test/ConcurrencyStressTests.cs new file mode 100644 index 0000000..ecfab84 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Test/ConcurrencyStressTests.cs @@ -0,0 +1,188 @@ +using EntityFrameworkCore.Sqlite.Concurrency; +using Microsoft.EntityFrameworkCore; + +namespace EFCore.Sqlite.Concurrency.Test; + +/// +/// Stress tests that verify correctness under concurrent write load. +/// Each test owns an isolated temp SQLite file and runs 50–100 concurrent writers. +/// Assertions verify that no rows are lost, duplicated, or corrupted. +/// +public class ConcurrencyStressTests +{ + // ── Test 1: SaveChangesSerializedAsync ──────────────────────────────────── + + [Fact] + public async Task SaveChangesSerializedAsync_WritesAllRows() + { + const int writers = 500; + const int rowsPerWriter = 100; + const int expected = writers * rowsPerWriter; + + using var db = new TempDatabase(); + + var tasks = Enumerable.Range(0, writers).Select(async writerIdx => + { + await using var ctx = db.CreateContext(); + + var entities = Enumerable.Range(0, rowsPerWriter) + .Select(i => new StressEntity + { + Id = Guid.NewGuid(), + WriterIndex = writerIdx, + Payload = $"w{writerIdx}-r{i}" + }) + .ToList(); + + ctx.Entities.AddRange(entities); + await ctx.SaveChangesSerializedAsync(); + }); + + await Task.WhenAll(tasks); + + await using var verify = db.CreateContext(); + var all = await verify.Entities.ToListAsync(); + var ids = all.Select(e => e.Id).ToHashSet(); + + Assert.Equal(expected, all.Count); + Assert.Equal(expected, ids.Count); // no duplicate IDs + } + + // ── Test 2: BulkInsertOptimizedAsync ───────────────────────────────────── + + [Fact] + public async Task BulkInsertOptimizedAsync_WritesAllRows() + { + const int writers = 100; + const int entitiesPerWriter = 500; + const int expected = writers * entitiesPerWriter; + + using var db = new TempDatabase(); + + var tasks = Enumerable.Range(0, writers).Select(async writerIdx => + { + await using var ctx = db.CreateContext(); + + var entities = Enumerable.Range(0, entitiesPerWriter) + .Select(i => new StressEntity + { + Id = Guid.NewGuid(), + WriterIndex = writerIdx, + Payload = $"bulk-w{writerIdx}-r{i}" + }) + .ToList(); + + await ctx.BulkInsertOptimizedAsync(entities); + }); + + await Task.WhenAll(tasks); + + await using var verify = db.CreateContext(); + var all = await verify.Entities.ToListAsync(); + var ids = all.Select(e => e.Id).ToHashSet(); + + Assert.Equal(expected, all.Count); + Assert.Equal(expected, ids.Count); + } + + // ── Test 3: ThreadSafeSqliteContext.ExecuteWriteAsync ──────────────────── + + [Fact] + public async Task ExecuteWriteAsync_ThreadSafeSqliteContext_WritesAllRows() + { + const int writers = 500; + const int rowsPerWriter = 100; + const int expected = writers * rowsPerWriter; + + using var db = new TempDatabase(); + + var tasks = Enumerable.Range(0, writers).Select(async writerIdx => + { + // ThreadSafeSqliteContext must be constructed with the connection string + // so it can find the shared write queue for this database file. + var ctx = new ThreadSafeSqliteStressContext(db.ConnectionString); + await using (ctx) + { + await ctx.ExecuteWriteAsync(async c => + { + var entities = Enumerable.Range(0, rowsPerWriter) + .Select(i => new StressEntity + { + Id = Guid.NewGuid(), + WriterIndex = writerIdx, + Payload = $"tssc-w{writerIdx}-r{i}" + }); + + c.Entities.AddRange(entities); + }); + } + }); + + await Task.WhenAll(tasks); + + await using var verify = db.CreateContext(); + var all = await verify.Entities.ToListAsync(); + var ids = all.Select(e => e.Id).ToHashSet(); + + Assert.Equal(expected, all.Count); + Assert.Equal(expected, ids.Count); + } + + // ── Test 4: Mixed concurrent reads + writes ─────────────────────────────── + + [Fact] + public async Task MixedReadsAndWrites_NoExceptionsOrDataLoss() + { + const int writers = 500; + const int readers = 200; + const int rowsPerWriter = 100; + const int expected = writers * rowsPerWriter; + + using var db = new TempDatabase(); + + // Pre-insert one seed row so readers have something to count from the start. + await using (var seed = db.CreateContext()) + { + seed.Entities.Add(new StressEntity { Payload = "seed" }); + await seed.SaveChangesSerializedAsync(); + } + + var writerTasks = Enumerable.Range(0, writers).Select(async writerIdx => + { + await using var ctx = db.CreateContext(); + var entities = Enumerable.Range(0, rowsPerWriter) + .Select(i => new StressEntity + { + Id = Guid.NewGuid(), + WriterIndex = writerIdx, + Payload = $"mixed-w{writerIdx}-r{i}" + }) + .ToList(); + + ctx.Entities.AddRange(entities); + await ctx.SaveChangesSerializedAsync(); + }); + + var readResults = new System.Collections.Concurrent.ConcurrentBag(); + var readerTasks = Enumerable.Range(0, readers).Select(async _ => + { + await using var ctx = db.CreateContext(); + // WAL mode allows reads to run concurrently with writes. + var count = await ctx.Entities.CountAsync(); + readResults.Add(count); + }); + + await Task.WhenAll(writerTasks.Concat(readerTasks)); + + // Verify all writes landed + await using var verify = db.CreateContext(); + var all = await verify.Entities.ToListAsync(); + + // +1 for the seed row + Assert.Equal(expected + 1, all.Count); + Assert.Equal(expected + 1, all.Select(e => e.Id).Distinct().Count()); + + // All reader results must be non-negative (reads always saw valid data) + Assert.All(readResults, count => Assert.True(count >= 0)); + } +} diff --git a/EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj b/EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj new file mode 100644 index 0000000..c282f5d --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj @@ -0,0 +1,37 @@ + + + + net10.0 + enable + enable + false + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/EFCore.Sqlite.Concurrency.Test/StressDbContext.cs b/EFCore.Sqlite.Concurrency.Test/StressDbContext.cs new file mode 100644 index 0000000..ee14ec2 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Test/StressDbContext.cs @@ -0,0 +1,27 @@ +using EntityFrameworkCore.Sqlite.Concurrency; +using EntityFrameworkCore.Sqlite.Concurrency.Models; +using Microsoft.EntityFrameworkCore; + +namespace EFCore.Sqlite.Concurrency.Test; + +public class StressDbContext : DbContext +{ + private readonly string _connectionString; + + public StressDbContext(string connectionString) + { + _connectionString = connectionString; + } + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder options) => + options.UseSqliteWithConcurrency(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity(e => + { + e.HasKey(x => x.Id); + e.Property(x => x.Payload).HasMaxLength(256); + }); +} diff --git a/EFCore.Sqlite.Concurrency.Test/StressEntity.cs b/EFCore.Sqlite.Concurrency.Test/StressEntity.cs new file mode 100644 index 0000000..8b6fec8 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Test/StressEntity.cs @@ -0,0 +1,8 @@ +namespace EFCore.Sqlite.Concurrency.Test; + +public class StressEntity +{ + public Guid Id { get; set; } = Guid.NewGuid(); + public int WriterIndex { get; set; } + public string Payload { get; set; } = string.Empty; +} diff --git a/EFCore.Sqlite.Concurrency.Test/TempDatabase.cs b/EFCore.Sqlite.Concurrency.Test/TempDatabase.cs new file mode 100644 index 0000000..a8c35b0 --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Test/TempDatabase.cs @@ -0,0 +1,40 @@ +namespace EFCore.Sqlite.Concurrency.Test; + +/// +/// Creates an isolated SQLite temp file for a single test. Deleted on Dispose. +/// +public sealed class TempDatabase : IDisposable +{ + private readonly string _dbPath; + private bool _disposed; + + public string ConnectionString { get; } + + public TempDatabase() + { + _dbPath = Path.Combine(Path.GetTempPath(), $"stress_{Guid.NewGuid():N}.db"); + ConnectionString = $"Data Source={_dbPath}"; + + // Create schema + using var ctx = CreateContext(); + ctx.Database.EnsureCreated(); + } + + public StressDbContext CreateContext() => new(ConnectionString); + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + + // Allow EF Core connection pool to drain before deleting the file. + Thread.Sleep(50); + + foreach (var ext in new[] { "", "-wal", "-shm" }) + { + var path = _dbPath + ext; + if (File.Exists(path)) + try { File.Delete(path); } catch { /* best-effort */ } + } + } +} diff --git a/EFCore.Sqlite.Concurrency.Test/ThreadSafeSqliteStressContext.cs b/EFCore.Sqlite.Concurrency.Test/ThreadSafeSqliteStressContext.cs new file mode 100644 index 0000000..452436b --- /dev/null +++ b/EFCore.Sqlite.Concurrency.Test/ThreadSafeSqliteStressContext.cs @@ -0,0 +1,21 @@ +using EntityFrameworkCore.Sqlite.Concurrency; +using Microsoft.EntityFrameworkCore; + +namespace EFCore.Sqlite.Concurrency.Test; + +/// +/// Concrete subclass of for stress tests. +/// +public class ThreadSafeSqliteStressContext : ThreadSafeSqliteContext +{ + public ThreadSafeSqliteStressContext(string connectionString) : base(connectionString) { } + + public DbSet Entities => Set(); + + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity(e => + { + e.HasKey(x => x.Id); + e.Property(x => x.Payload).HasMaxLength(256); + }); +} diff --git a/EntityFrameworkCore.Sqlite.Concurrency.sln b/EntityFrameworkCore.Sqlite.Concurrency.sln index 3f414b8..d0b26d0 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency.sln +++ b/EntityFrameworkCore.Sqlite.Concurrency.sln @@ -5,6 +5,8 @@ VisualStudioVersion = 17.0.31903.59 MinimumVisualStudioVersion = 10.0.40219.1 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "EFCore.Sqlite.Concurrency", "EntityFrameworkCore.Sqlite.Concurrency\EFCore.Sqlite.Concurrency.csproj", "{CDD465FA-9DE5-48C7-931E-5D3B0A9AEED5}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "EFCore.Sqlite.Concurrency.Test", "EFCore.Sqlite.Concurrency.Test\EFCore.Sqlite.Concurrency.Test.csproj", "{5EFE0CEB-C626-44AF-A963-D437A9816B2B}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -18,5 +20,9 @@ Global {CDD465FA-9DE5-48C7-931E-5D3B0A9AEED5}.Debug|Any CPU.Build.0 = Debug|Any CPU {CDD465FA-9DE5-48C7-931E-5D3B0A9AEED5}.Release|Any CPU.ActiveCfg = Release|Any CPU {CDD465FA-9DE5-48C7-931E-5D3B0A9AEED5}.Release|Any CPU.Build.0 = Release|Any CPU + {5EFE0CEB-C626-44AF-A963-D437A9816B2B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {5EFE0CEB-C626-44AF-A963-D437A9816B2B}.Debug|Any CPU.Build.0 = Debug|Any CPU + {5EFE0CEB-C626-44AF-A963-D437A9816B2B}.Release|Any CPU.ActiveCfg = Release|Any CPU + {5EFE0CEB-C626-44AF-A963-D437A9816B2B}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection EndGlobal diff --git a/EntityFrameworkCore.Sqlite.Concurrency.sln.DotSettings.user b/EntityFrameworkCore.Sqlite.Concurrency.sln.DotSettings.user index c89bb65..93b250f 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency.sln.DotSettings.user +++ b/EntityFrameworkCore.Sqlite.Concurrency.sln.DotSettings.user @@ -1,2 +1,5 @@  - ForceIncluded \ No newline at end of file + ForceIncluded + <SessionState ContinuousTestingMode="0" IsActive="True" Name="All tests from Solution" xmlns="urn:schemas-jetbrains-com:jetbrains-ut-session"> + <Solution /> +</SessionState> \ No newline at end of file diff --git a/EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj b/EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj index a5029f6..c6ecbc3 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj +++ b/EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj @@ -1,9 +1,10 @@  - net10.0 + net10.0;netstandard2.0 enable enable + latest EntityFrameworkCore.Sqlite.Concurrency true true @@ -11,7 +12,7 @@ EntityFrameworkCore.Sqlite.Concurrency - 10.0.4 + 10.1.0 Mike Gotfryd @@ -21,50 +22,37 @@ - Eliminates 'SQLITE_BUSY' / 'database is locked' errors in multi-threaded - Entity Framework Core apps. Provides automatic write serialization, 10x faster - bulk inserts, and true parallel reads for SQLite. A drop-in, high-performance, - thread-safe addition to Microsoft.EntityFrameworkCore.Sqlite in .NET 10. + Fixes SQLITE_BUSY / "database is locked" / SQLITE_BUSY_SNAPSHOT errors in Entity + Framework Core + SQLite apps. Provides Channel-based write serialization, automatic + BEGIN IMMEDIATE transaction upgrades, exponential-backoff retry, WAL-mode optimization, + and 10x faster bulk inserts. Drop-in replacement for UseSqlite() targeting .NET 10 + and netstandard 2.0. - - sqlite sqlite3 entity-framework-core efcore concurrency thread-safe multi-threading database-locked sqlite-busy performance bulk-insert parallel-reads write-ahead-logging wal dotnet-10 entity-framework orm database-provider high-performance async transactions locking queue - + sqlite sqlite3 entity-framework-core efcore concurrency thread-safe multi-threading database-locked sqlite-busy sqlite-busy-snapshot error-5 write-serialization channel-queue netstandard2 efcore-concurrency performance bulk-insert parallel-reads write-ahead-logging wal dotnet-10 entity-framework orm database-provider high-performance async transactions locking queue background-service task-whenall + — registers IDbContextFactory with all concurrency - settings. Use this for Task.WhenAll, background services, Channel consumers, and any - workload that creates concurrent database operations. DbContext is not thread-safe; the factory - pattern gives each concurrent flow its own independent instance. -• Structured logging: pass ILoggerFactory (or let DI resolve it) to get Warning logs for - SQLITE_BUSY/SQLITE_BUSY_SNAPSHOT events, Error logs for SQLITE_LOCKED, and Debug logs for - BEGIN IMMEDIATE upgrades — all through your existing logging pipeline. -• GetWalCheckpointStatusAsync — runs PRAGMA wal_checkpoint(PASSIVE) and returns a typed - WalCheckpointStatus with IsBusy, TotalWalFrames, CheckpointedFrames, and CheckpointProgress. - Call periodically to detect long-running readers blocking WAL reclamation before it degrades - read performance. -• TryReleaseMigrationLockAsync — detects and optionally clears a stale __EFMigrationsLock - row left behind by a crashed migration process. Prevents indefinite blocking on Database.Migrate() - in multi-instance deployments. -• SynchronousMode option — configures PRAGMA synchronous (Off / Normal / Full / Extra). - Default remains Normal (recommended for WAL: safe after app crash, fast writes). -• UpgradeTransactionsToImmediate option — opt out of the BEGIN → BEGIN IMMEDIATE rewrite - if you manage write transactions explicitly yourself. Default remains true. +• Channel-based write queue: replaces SemaphoreSlim with Channel + single + background writer. SemaphoreSlim wakes all N waiters simultaneously (thundering herd); + Channel parks callers in FIFO order — no wakeup storm, fairer scheduling, lower CPU. + Zero API change — existing call sites benefit automatically. +• WriteQueueCapacity option: new int? on SqliteConcurrencyOptions (null = unbounded). + Set a value to create a bounded channel with back-pressure (BoundedChannelFullMode.Wait). +• netstandard 2.0 TFM: SqliteConnectionEnhancer, SqliteWriteQueue, SqliteConcurrencyOptions + now available on any runtime supporting netstandard 2.0. EF Core APIs remain net10.0-only. + +BUG FIXES +• ThreadSafeSqliteContext always used default options: ExecuteWriteAsync was constructing + new SqliteConcurrencyOptions() instead of reading registered interceptor options. Fixed + via TryGetInterceptor lookup. +• BulkInsertSafeAsync O(n²) + unbounded ChangeTracker growth: Skip/Take loop replaced with + Chunk(1000); ChangeTracker.Clear() added after each batch SaveChangesAsync. NO BREAKING CHANGES — all existing call sites compile and behave correctly without modification. ]]> @@ -98,14 +86,45 @@ NO BREAKING CHANGES — all existing call sites compile and behave correctly wit true - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + diff --git a/EntityFrameworkCore.Sqlite.Concurrency/out/EntityFrameworkCore.Sqlite.Concurrency.10.0.4.nupkg b/EntityFrameworkCore.Sqlite.Concurrency/out/EntityFrameworkCore.Sqlite.Concurrency.10.0.4.nupkg new file mode 100644 index 0000000..81d2dc1 Binary files /dev/null and b/EntityFrameworkCore.Sqlite.Concurrency/out/EntityFrameworkCore.Sqlite.Concurrency.10.0.4.nupkg differ diff --git a/EntityFrameworkCore.Sqlite.Concurrency/out/EntityFrameworkCore.Sqlite.Concurrency.10.0.4.snupkg b/EntityFrameworkCore.Sqlite.Concurrency/out/EntityFrameworkCore.Sqlite.Concurrency.10.0.4.snupkg new file mode 100644 index 0000000..28fc40a Binary files /dev/null and b/EntityFrameworkCore.Sqlite.Concurrency/out/EntityFrameworkCore.Sqlite.Concurrency.10.0.4.snupkg differ diff --git a/EntityFrameworkCore.Sqlite.Concurrency/packages.lock.json b/EntityFrameworkCore.Sqlite.Concurrency/packages.lock.json index c9da38b..b1bf311 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/packages.lock.json +++ b/EntityFrameworkCore.Sqlite.Concurrency/packages.lock.json @@ -1,190 +1,358 @@ { "version": 1, "dependencies": { + ".NETStandard,Version=v2.0": { + "Microsoft.Data.Sqlite": { + "type": "Direct", + "requested": "[7.0.20, )", + "resolved": "7.0.20", + "contentHash": "IIjX4K1EjTg73Ed/ZsjHWyAtcRAGoS42eiHa30HIbTYO62dGgq+yWRpkxZbTSvEjUlUvUmWORxrEmKloRianUw==", + "dependencies": { + "Microsoft.Data.Sqlite.Core": "7.0.20", + "SQLitePCLRaw.bundle_e_sqlite3": "2.1.4" + } + }, + "Microsoft.Extensions.DependencyInjection.Abstractions": { + "type": "Direct", + "requested": "[8.0.2, )", + "resolved": "8.0.2", + "contentHash": "3iE7UF7MQkCv1cxzCahz+Y/guQbTqieyxyaWKhrRO91itI9cOKO76OHeQDahqG4MmW5umr3CcCvGmK92lWNlbg==", + "dependencies": { + "Microsoft.Bcl.AsyncInterfaces": "8.0.0", + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "Microsoft.Extensions.Logging.Abstractions": { + "type": "Direct", + "requested": "[8.0.3, )", + "resolved": "8.0.3", + "contentHash": "dL0QGToTxggRLMYY4ZYX5AMwBb+byQBd/5dMiZE07Nv73o6I5Are3C7eQTh7K2+A4ct0PVISSr7TZANbiNb2yQ==", + "dependencies": { + "Microsoft.Extensions.DependencyInjection.Abstractions": "8.0.2", + "System.Buffers": "4.5.1", + "System.Memory": "4.5.5" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "NETStandard.Library": { + "type": "Direct", + "requested": "[2.0.3, )", + "resolved": "2.0.3", + "contentHash": "st47PosZSHrjECdjeIzZQbzivYBJFv6P2nv4cj2ypdI204DO+vZ7l5raGMiX4eXMJ53RfOIg+/s4DHVZ54Nu2A==", + "dependencies": { + "Microsoft.NETCore.Platforms": "1.1.0" + } + }, + "System.Threading.Channels": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "CMaFr7v+57RW7uZfZkPExsPB6ljwzhjACWW1gfU35Y56rk72B/Wu+sTqxVmGSk4SFUlPc3cjeKND0zktziyjBA==", + "dependencies": { + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "Microsoft.Bcl.AsyncInterfaces": { + "type": "Transitive", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==", + "dependencies": { + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.Data.Sqlite.Core": { + "type": "Transitive", + "resolved": "7.0.20", + "contentHash": "B8/42FqMMRq3mf5qnJOF7XhB/UEcHcwdmuI6D025uTRcPM0R1DbMw+6C77Op1f/qKAHsDHmdTID5CZLpYBvmlw==", + "dependencies": { + "SQLitePCLRaw.core": "2.1.4" + } + }, + "Microsoft.NETCore.Platforms": { + "type": "Transitive", + "resolved": "1.1.0", + "contentHash": "kz0PEW2lhqygehI/d6XsPCQzD7ff7gUJaVGPVETX611eadGsA3A877GdSlU0LRVMCTH/+P3o2iDTak+S08V2+A==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "SQLitePCLRaw.bundle_e_sqlite3": { + "type": "Transitive", + "resolved": "2.1.4", + "contentHash": "EWI1olKDjFEBMJu0+3wuxwziIAdWDVMYLhuZ3Qs84rrz+DHwD00RzWPZCa+bLnHCf3oJwuFZIRsHT5p236QXww==", + "dependencies": { + "SQLitePCLRaw.lib.e_sqlite3": "2.1.4", + "SQLitePCLRaw.provider.e_sqlite3": "2.1.4" + } + }, + "SQLitePCLRaw.core": { + "type": "Transitive", + "resolved": "2.1.4", + "contentHash": "inBjvSHo9UDKneGNzfUfDjK08JzlcIhn1+SP5Y3m6cgXpCxXKCJDy6Mka7LpgSV+UZmKSnC8rTwB0SQ0xKu5pA==", + "dependencies": { + "System.Memory": "4.5.3" + } + }, + "SQLitePCLRaw.lib.e_sqlite3": { + "type": "Transitive", + "resolved": "2.1.4", + "contentHash": "2C9Q9eX7CPLveJA0rIhf9RXAvu+7nWZu1A2MdG6SD/NOu26TakGgL1nsbc0JAspGijFOo3HoN79xrx8a368fBg==" + }, + "SQLitePCLRaw.provider.e_sqlite3": { + "type": "Transitive", + "resolved": "2.1.4", + "contentHash": "CSlb5dUp1FMIkez9Iv5EXzpeq7rHryVNqwJMWnpq87j9zWZexaEMdisDktMsnnrzKM6ahNrsTkjqNodTBPBxtQ==", + "dependencies": { + "SQLitePCLRaw.core": "2.1.4" + } + }, + "System.Buffers": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "N8GXpmiLMtljq7gwvyS+1QvKT/W2J8sNAvx+HVg4NGmsG/H+2k/y9QI23auLJRterrzCiDH+IWAw4V/GPwsMlw==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Memory": "4.6.3" + } + }, + "System.Memory": { + "type": "Transitive", + "resolved": "4.6.3", + "contentHash": "qdcDOgnFZY40+Q9876JUHnlHu7bosOHX8XISRoH94fwk6hgaeQGSgfZd8srWRZNt5bV9ZW2TljcegDNxsf+96A==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Numerics.Vectors": "4.6.1", + "System.Runtime.CompilerServices.Unsafe": "6.1.2" + } + }, + "System.Numerics.Vectors": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "sQxefTnhagrhoq2ReR0D/6K0zJcr9Hrd6kikeXsA1I8kOCboTavcUC4r7TSfpKFeE163uMuxZcyfO1mGO3EN8Q==" + }, + "System.Runtime.CompilerServices.Unsafe": { + "type": "Transitive", + "resolved": "6.1.2", + "contentHash": "2hBr6zdbIBTDE3EhK7NSVNdX58uTK6iHW/P/Axmm9sl1xoGSLqDvMtpecn226TNwHByFokYwJmt/aQQNlO5CRw==" + }, + "System.Threading.Tasks.Extensions": { + "type": "Transitive", + "resolved": "4.5.4", + "contentHash": "zteT+G8xuGu6mS+mzDzYXbzS7rd3K6Fjb9RiZlYlJPam2/hU7JCBZBVEcywNuR+oZ1ncTvc/cq0faRr3P01OVg==", + "dependencies": { + "System.Runtime.CompilerServices.Unsafe": "4.5.3" + } + } + }, "net10.0": { "Microsoft.Data.Sqlite": { "type": "Direct", - "requested": "[10.0.2, )", - "resolved": "10.0.2", - "contentHash": "kST8oJAIM1GBKZ9kpzqy+U7DAZ50m3whANKVc9NpL7PYFHmPRM1qIHjG99BsmhvUMwku+imJHWWi3kcG4OCs7Q==", + "requested": "[10.0.9, )", + "resolved": "10.0.9", + "contentHash": "/eBwiZPcNisn0qZX+Zk4YCftlK/vnoWqv7hHnmSk8MjPxFdYYkmPObpogT0MfCCWN6oAIZnMCo0SoOtZlbbmgQ==", "dependencies": { - "Microsoft.Data.Sqlite.Core": "10.0.2", + "Microsoft.Data.Sqlite.Core": "10.0.9", "SQLitePCLRaw.bundle_e_sqlite3": "2.1.11", "SQLitePCLRaw.core": "2.1.11" } }, "Microsoft.EntityFrameworkCore.Sqlite": { "type": "Direct", - "requested": "[10.0.2, )", - "resolved": "10.0.2", - "contentHash": "xprEItydc2HDhThghBsuq/2t9UoEhJvzKWE2Mx1+hlJdCZBUoZWmbuua9pZy9vTE+aVVssX261cx5F6nUpKzoA==", - "dependencies": { - "Microsoft.EntityFrameworkCore.Sqlite.Core": "10.0.2", - "Microsoft.Extensions.Caching.Memory": "10.0.2", - "Microsoft.Extensions.Configuration.Abstractions": "10.0.2", - "Microsoft.Extensions.DependencyModel": "10.0.2", - "Microsoft.Extensions.Logging": "10.0.2", + "requested": "[10.0.9, )", + "resolved": "10.0.9", + "contentHash": "HH9/nnRF/YmRrc3hUlgXjMBYKH5kFmd5UWC81l9U0ySQhwHTcgvDPSewB8DyQHzFJzNGgG7VFK6ynC6+XQz9WA==", + "dependencies": { + "Microsoft.EntityFrameworkCore.Sqlite.Core": "10.0.9", + "Microsoft.Extensions.Caching.Memory": "10.0.9", + "Microsoft.Extensions.Configuration.Abstractions": "10.0.9", + "Microsoft.Extensions.DependencyModel": "10.0.9", + "Microsoft.Extensions.Logging": "10.0.9", "SQLitePCLRaw.bundle_e_sqlite3": "2.1.11", "SQLitePCLRaw.core": "2.1.11" } }, "Microsoft.Extensions.DependencyInjection": { "type": "Direct", - "requested": "[10.0.5, )", - "resolved": "10.0.5", - "contentHash": "v1SVsowG6YE1YnHVGmLWz57YTRCQRx9pH5ebIESXfm5isI9gA3QaMyg/oMTzPpXYZwSAVDzYItGJKfmV+pqXkQ==", + "requested": "[10.0.9, )", + "resolved": "10.0.9", + "contentHash": "NijozhERJDIaJ4k5TSMy1jOi0cSC2HfkvRD/Sl+kGSSKgVbFnF4GxgtMN/MrzHB8D1JxIrD4xSer9Blh9v3axQ==", "dependencies": { - "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.5" + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.9" } }, "Microsoft.Extensions.Logging.Abstractions": { "type": "Direct", - "requested": "[10.0.2, )", - "resolved": "10.0.2", - "contentHash": "RZkez/JjpnO+MZ6efKkSynN6ZztLpw3WbxNzjLCPBd97wWj1S9ZYPWi0nmT4kWBRa6atHsdM1ydGkUr8GudyDQ==", + "requested": "[10.0.9, )", + "resolved": "10.0.9", + "contentHash": "9S/DFt4cohlMPpzIxjG6kk0L8MuN2vDm9pbMCulxtJzzk82oJHVLBd8vuQxaPskaYQwKqmFmbannf5eoChgjYg==", "dependencies": { - "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.2" + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.9" } }, "Microsoft.SourceLink.GitHub": { "type": "Direct", - "requested": "[10.0.201, )", - "resolved": "10.0.201", - "contentHash": "qxYAmO4ktzd9L+HMdnqWucxpu7bI9undPyACXOMqPyhaiMtbpbYL/n0ACyWIJlbyEJrXFwxiOaBOSasLtDvsCg==", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", "dependencies": { - "Microsoft.Build.Tasks.Git": "10.0.201", - "Microsoft.SourceLink.Common": "10.0.201", - "System.IO.Hashing": "10.0.5" + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" } }, "Microsoft.Build.Tasks.Git": { "type": "Transitive", - "resolved": "10.0.201", - "contentHash": "DMYBnrFZvLnBKn14VavEuuIr31CY6YY2i2L9P8DorS/Qp6ifRR8ZPLdJCFLFfjikNq8DykbYyLd/RP6lSqHcWw==", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", "dependencies": { - "System.IO.Hashing": "10.0.5" + "System.IO.Hashing": "10.0.8" } }, "Microsoft.Data.Sqlite.Core": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "/8ubtRdyED4qyxmL1CAu3UZZUEgc9l0dPCItW5tlOkYlKOb3QNhjGuEeb7uTKAqm8L7kwFOx1fCsndiylrzZAg==", + "resolved": "10.0.9", + "contentHash": "iZrONyMKPjxfVZnUktqO30QjzNwAGH+AxM61s8lKQnVhgbQ3bn0hiXI129ZmVicEbIcwljyy2OVsIYUR51ZHKQ==", "dependencies": { "SQLitePCLRaw.core": "2.1.11" } }, "Microsoft.EntityFrameworkCore": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "d3+XKbLSHPCu3vwpXECoXcFbvGKmAhEeUmc1xy2czmuPnEF7rZN2HP5ZGMwCMbAKk4B01+nS4HixSMo2Vf/Y9g==", + "resolved": "10.0.9", + "contentHash": "tu85SRzOT021V7EQlViCiAE7TqldVn469Y6lt5TEn/+XC4/MeNCHgMRSxqYuWqvF4zAQZUhCmtNEZuM3ss4LeA==", "dependencies": { - "Microsoft.EntityFrameworkCore.Abstractions": "10.0.2", - "Microsoft.EntityFrameworkCore.Analyzers": "10.0.2", - "Microsoft.Extensions.Caching.Memory": "10.0.2", - "Microsoft.Extensions.Logging": "10.0.2" + "Microsoft.EntityFrameworkCore.Abstractions": "10.0.9", + "Microsoft.EntityFrameworkCore.Analyzers": "10.0.9", + "Microsoft.Extensions.Caching.Memory": "10.0.9", + "Microsoft.Extensions.Logging": "10.0.9" } }, "Microsoft.EntityFrameworkCore.Abstractions": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "BzAwIU5mYeOmnKbEXrkwx7feW2V+zUTrK/kRonSib94tjvc0/iRj2a4N6YGXRhTNjaFP3tvCMIDaX1vIFF6dkg==" + "resolved": "10.0.9", + "contentHash": "GRMaiPkqYna/gCsyDffYDWmefGPC3hDrdMw+2rrGcQwhs6uZOsaMQXMJnoXQ35tx9SkBV2ieRRU9N/jLOO6BZw==" }, "Microsoft.EntityFrameworkCore.Analyzers": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "5C8aKN2HFhXfLhkrVvQ6yH990nw35pS7HNnwdBc628zIREaKJ4/rkdCXzxOdZo/SnCT3Kg7a/CqnhlzuJhlD4Q==" + "resolved": "10.0.9", + "contentHash": "aiEFB+C5EsZGqxvMPazE07hbWsp4iPaufJpanGt5O+lrwv7mJLrqma5haVIgFAPCyhQkmk75XSCEubT1zUjxtA==" }, "Microsoft.EntityFrameworkCore.Relational": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "1fUeyNmqDNfMogJ2ut7OKO57/WGjjkHMYeX51SpA3PwP7ftbx8g/Z3fbErD+1q14DILrqJfsszYsYhGssBRfDg==", + "resolved": "10.0.9", + "contentHash": "dEoyYKgiaZHHgOFm1WMWm1sFEsEuhPWufX4L9PekKtqd/RaIcPjkCjvbrVvJtApErb5wPSJhYvnTlxhH+p9h2g==", "dependencies": { - "Microsoft.EntityFrameworkCore": "10.0.2", - "Microsoft.Extensions.Caching.Memory": "10.0.2", - "Microsoft.Extensions.Configuration.Abstractions": "10.0.2", - "Microsoft.Extensions.Logging": "10.0.2" + "Microsoft.EntityFrameworkCore": "10.0.9", + "Microsoft.Extensions.Caching.Memory": "10.0.9", + "Microsoft.Extensions.Configuration.Abstractions": "10.0.9", + "Microsoft.Extensions.Logging": "10.0.9" } }, "Microsoft.EntityFrameworkCore.Sqlite.Core": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "IAaYGhV7lKXbG7c5G9mMlc1buAiUXME0RbLPrAFdlmtRuCnUaKbjcupJAQJPz2q2VEjwzm1Mx2WprizFVls0QQ==", + "resolved": "10.0.9", + "contentHash": "3bPEmACAaPJJSw+m5XwTSi3yZnVtaifa4d8gLsNMzW0Qu28jS5kADSfgJRBlq49RJ1K098VCzEDRJwM8gE6f2w==", "dependencies": { - "Microsoft.Data.Sqlite.Core": "10.0.2", - "Microsoft.EntityFrameworkCore.Relational": "10.0.2", - "Microsoft.Extensions.Caching.Memory": "10.0.2", - "Microsoft.Extensions.Configuration.Abstractions": "10.0.2", - "Microsoft.Extensions.DependencyModel": "10.0.2", - "Microsoft.Extensions.Logging": "10.0.2", + "Microsoft.Data.Sqlite.Core": "10.0.9", + "Microsoft.EntityFrameworkCore.Relational": "10.0.9", + "Microsoft.Extensions.Caching.Memory": "10.0.9", + "Microsoft.Extensions.Configuration.Abstractions": "10.0.9", + "Microsoft.Extensions.DependencyModel": "10.0.9", + "Microsoft.Extensions.Logging": "10.0.9", "SQLitePCLRaw.core": "2.1.11" } }, "Microsoft.Extensions.Caching.Abstractions": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "WIRPDa/qoKHmJhTAPCO/zLu9kRLQ2Fd6HD5tzgdXJ3xGEVXDHP6FvakKJjynwKrVDld8H4G4tcbW53wuC/wxMQ==", + "resolved": "10.0.9", + "contentHash": "5fGxcw2vuYp8s0wio9H1ECiuk4iKSdTIlNuigdLIrkhg+5XAwgFVDB/5Ots3pfN/QhABLYXutA79JFtnUKDSHA==", "dependencies": { - "Microsoft.Extensions.Primitives": "10.0.2" + "Microsoft.Extensions.Primitives": "10.0.9" } }, "Microsoft.Extensions.Caching.Memory": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "MkdPYdtsu0Ta4m9Di4XnWVdO9u+wi1LtvisoR1EteIxsXWO/+3iyAPH6RZbw2lBlWZu9lastbl2YsHVIaL9j+g==", + "resolved": "10.0.9", + "contentHash": "G9mregdatGWMCQWeCw012LDeJVP7G/XIxH8Ddbjc8bD1//dA+8VVQdcRE9jI1moyoJxSSZhHITUnNQ8FUDl5+Q==", "dependencies": { - "Microsoft.Extensions.Caching.Abstractions": "10.0.2", - "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.2", - "Microsoft.Extensions.Logging.Abstractions": "10.0.2", - "Microsoft.Extensions.Options": "10.0.2", - "Microsoft.Extensions.Primitives": "10.0.2" + "Microsoft.Extensions.Caching.Abstractions": "10.0.9", + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.9", + "Microsoft.Extensions.Logging.Abstractions": "10.0.9", + "Microsoft.Extensions.Options": "10.0.9", + "Microsoft.Extensions.Primitives": "10.0.9" } }, "Microsoft.Extensions.Configuration.Abstractions": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "KC5PslaTDnTuTvyke0KYAVBYdZ7IVTsU3JhHe69BpEbHLcj1YThP3bIGtZNOkZfast2AuLnul5lk4rZKxAdUGQ==", + "resolved": "10.0.9", + "contentHash": "qGhRPd3VxfLV9UqatVOiD9mAeUbj2KiMwGFYC5uXlzExiZQoe4X/hdmzGIU7BQjNLTqCnnbTHVyBglG3668/HA==", "dependencies": { - "Microsoft.Extensions.Primitives": "10.0.2" + "Microsoft.Extensions.Primitives": "10.0.9" } }, "Microsoft.Extensions.DependencyInjection.Abstractions": { "type": "Transitive", - "resolved": "10.0.5", - "contentHash": "iVMtq9eRvzyhx8949EGT0OCYJfXi737SbRVzWXE5GrOgGj5AaZ9eUuxA/BSUfmOMALKn/g8KfFaNQw0eiB3lyA==" + "resolved": "10.0.9", + "contentHash": "g41l/30G3K4B/d/L8kjux0+30e27c8D0FVQ/PFCpbekgfDpj9mnDhieP67EqXWvl1EWNeZh2rpR4F5B/jcDOHA==" }, "Microsoft.Extensions.DependencyModel": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "DHXzj/FKmUDmhzw7AHbMXbNNcgO8Cb9PDbYOUDICjFAdIALy2WbC6pm+dFKiAUNtcYMucZu2iMVw1ef/sdcZFw==" + "resolved": "10.0.9", + "contentHash": "SCDTQ6HubnRvTUjR7dgMKHZvNoCb03t44ttHL8trlFTGgfDteWn/0nRdOxDhcI+lTWhKgd/flCVJEtAOPhSLNg==" }, "Microsoft.Extensions.Logging": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "a0EWuBs6D3d7XMGroDXm+WsAi5CVVfjOJvyxurzWnuhBN9CO+1qHKcrKV1JK7H/T4ZtHIoVCOX/YyWM8K87qtw==", + "resolved": "10.0.9", + "contentHash": "N7Gm9SjugYjmmnhwbBKC9DFqGqjfJvh6YfOJgtwh0AW0Xpok3dIVors1ik050XmUxKAgAc7nNngDIJyFb06K2g==", "dependencies": { - "Microsoft.Extensions.DependencyInjection": "10.0.2", - "Microsoft.Extensions.Logging.Abstractions": "10.0.2", - "Microsoft.Extensions.Options": "10.0.2" + "Microsoft.Extensions.DependencyInjection": "10.0.9", + "Microsoft.Extensions.Logging.Abstractions": "10.0.9", + "Microsoft.Extensions.Options": "10.0.9" } }, "Microsoft.Extensions.Options": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "1De2LJjmxdqopI5AYC5dIhoZQ79AR5ayywxNF1rXrXFtKQfbQOV9+n/IsZBa7qWlr0MqoGpW8+OY2v/57udZOA==", + "resolved": "10.0.9", + "contentHash": "hyNdX4c2UwkRkzb9byw0H2DQkRzwBM3mzY2sCM9egwzTyg8dvQJmp5noQHGEaaCORQrNK3DD2gREBsc2DlXS4A==", "dependencies": { - "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.2", - "Microsoft.Extensions.Primitives": "10.0.2" + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.9", + "Microsoft.Extensions.Primitives": "10.0.9" } }, "Microsoft.Extensions.Primitives": { "type": "Transitive", - "resolved": "10.0.2", - "contentHash": "QmSiO+oLBEooGgB3i0GRXyeYRDHjllqt3k365jwfZlYWhvSHA3UL2NEVV5m8aZa041eIlblo6KMI5txvTMpTwA==" + "resolved": "10.0.9", + "contentHash": "fmEbAUFsaIKirgLt/lYhuFRBwhcSJN31jjHgCdbQxJiWOum6EdLjkbgGuukSP9z/a+9LibaxII/kF+GwOXgC4g==" }, "Microsoft.SourceLink.Common": { "type": "Transitive", - "resolved": "10.0.201", - "contentHash": "QbBYhkjgL6rCnBfDbzsAJLlsad13TlBHqYCFDIw56OO2g6ix+9RsmY8uxiQGdWwFKbZXaXyAA6jDCzFYVGCZDw==" + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" }, "SQLitePCLRaw.bundle_e_sqlite3": { "type": "Transitive", @@ -215,8 +383,8 @@ }, "System.IO.Hashing": { "type": "Transitive", - "resolved": "10.0.5", - "contentHash": "8IBJWcCT9+e4Bmevm4T7+fQEiAh133KGiz4oiVTgJckd3Q76OFdR1falgn9lpz7+C4HJvogCDJeAa2QmvbeVtg==" + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" } } } diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/Models/SqliteConcurrencyOptions.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/Models/SqliteConcurrencyOptions.cs index dc5fb54..3f12218 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/src/Models/SqliteConcurrencyOptions.cs +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/Models/SqliteConcurrencyOptions.cs @@ -112,6 +112,28 @@ public class SqliteConcurrencyOptions : IEquatable /// public bool UpgradeTransactionsToImmediate { get; set; } = true; + /// + /// Bounded capacity of the per-database write queue. + /// + /// + /// + /// When set, the write queue uses a bounded + /// with BoundedChannelFullMode.Wait. Callers that enqueue writes while the channel is + /// at capacity will wait asynchronously until space is available, providing explicit + /// back-pressure under heavy write load. + /// + /// + /// When (the default), the channel is unbounded and callers are + /// never delayed by back-pressure. The underlying queue still serializes writes in FIFO + /// order; only the maximum depth is unrestricted. + /// + /// + /// This value is used only when the write queue is first created for a given database. + /// Changing it after first use has no effect. + /// + /// + public int? WriteQueueCapacity { get; set; } + /// /// Optional logger factory used to create the interceptor's /// . Not included in equality comparison. @@ -160,7 +182,8 @@ public bool Equals(SqliteConcurrencyOptions? other) && CommandTimeout == other.CommandTimeout && WalAutoCheckpoint == other.WalAutoCheckpoint && SynchronousMode == other.SynchronousMode - && UpgradeTransactionsToImmediate == other.UpgradeTransactionsToImmediate; + && UpgradeTransactionsToImmediate == other.UpgradeTransactionsToImmediate + && WriteQueueCapacity == other.WriteQueueCapacity; // LoggerFactory is intentionally excluded from equality: it is infrastructure // metadata and does not affect SQLite behaviour. } @@ -182,5 +205,6 @@ public override int GetHashCode() CommandTimeout, WalAutoCheckpoint, SynchronousMode, - UpgradeTransactionsToImmediate); + UpgradeTransactionsToImmediate, + WriteQueueCapacity); } diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/Polyfills.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/Polyfills.cs new file mode 100644 index 0000000..dd53375 --- /dev/null +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/Polyfills.cs @@ -0,0 +1,32 @@ +#if NETSTANDARD2_0 +// C# 9+ init-only setters and record types require IsExternalInit on netstandard2.0. +namespace System.Runtime.CompilerServices +{ + internal static class IsExternalInit { } +} + +// HashCode.Combine was introduced in .NET Core 2.1 / .NET Standard 2.1. +// Provide a minimal polyfill for the netstandard2.0 build. +namespace System +{ + internal static class HashCode + { + public static int Combine( + T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7) + { + unchecked + { + var h = 17; + h = h * 31 + (v1 is null ? 0 : v1.GetHashCode()); + h = h * 31 + (v2 is null ? 0 : v2.GetHashCode()); + h = h * 31 + (v3 is null ? 0 : v3.GetHashCode()); + h = h * 31 + (v4 is null ? 0 : v4.GetHashCode()); + h = h * 31 + (v5 is null ? 0 : v5.GetHashCode()); + h = h * 31 + (v6 is null ? 0 : v6.GetHashCode()); + h = h * 31 + (v7 is null ? 0 : v7.GetHashCode()); + return h; + } + } + } +} +#endif diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyExtensions.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyExtensions.cs index 9521a97..c6efa52 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyExtensions.cs +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyExtensions.cs @@ -154,12 +154,9 @@ public static async Task SaveChangesSerializedAsync( var connectionString = context.Database.GetDbConnection().ConnectionString; var enhancedConnectionString = SqliteConnectionEnhancer.GetOptimizedConnectionString(connectionString); - var writeLock = SqliteConnectionEnhancer.GetWriteLock(enhancedConnectionString); + var queue = SqliteConnectionEnhancer.GetWriteQueue(enhancedConnectionString); - await writeLock.WaitAsync(cancellationToken); - SqliteConnectionEnhancer.IsWriteLockHeld.Value = true; - - try + return await queue.EnqueueAsync(async () => { var delayMs = 50; for (var attempt = 1; ; attempt++) @@ -175,12 +172,7 @@ public static async Task SaveChangesSerializedAsync( delayMs = Math.Min(delayMs * 2, 2000); } } - } - finally - { - SqliteConnectionEnhancer.IsWriteLockHeld.Value = false; - writeLock.Release(); - } + }, cancellationToken); } // EF Core wraps SqliteException in DbUpdateException when SaveChangesAsync fails, @@ -215,19 +207,13 @@ public static async Task BulkInsertOptimizedAsync( var connectionString = context.Database.GetDbConnection().ConnectionString; var enhancedConnectionString = SqliteConnectionEnhancer.GetOptimizedConnectionString(connectionString); - var writeLock = SqliteConnectionEnhancer.GetWriteLock(enhancedConnectionString); - - await writeLock.WaitAsync(cancellationToken); - SqliteConnectionEnhancer.IsWriteLockHeld.Value = true; + var queue = SqliteConnectionEnhancer.GetWriteQueue(enhancedConnectionString); - try + await queue.EnqueueAsync(async () => { await using var transaction = await context.Database.BeginTransactionAsync(cancellationToken); - var batchSize = 1000; - var batches = entities.Chunk(batchSize); - - foreach (var batch in batches) + foreach (var batch in entities.Chunk(1000)) { await context.AddRangeAsync(batch, cancellationToken); await context.SaveChangesAsync(cancellationToken); @@ -235,11 +221,7 @@ public static async Task BulkInsertOptimizedAsync( } await transaction.CommitAsync(cancellationToken); - } - finally - { - SqliteConnectionEnhancer.IsWriteLockHeld.Value = false; - writeLock.Release(); - } + return 0; + }, cancellationToken); } } diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyInterceptor.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyInterceptor.cs index 86ae2ce..45ad667 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyInterceptor.cs +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConcurrencyInterceptor.cs @@ -12,7 +12,6 @@ namespace EntityFrameworkCore.Sqlite.Concurrency; public class SqliteConcurrencyInterceptor : DbCommandInterceptor, IDbConnectionInterceptor, IDbTransactionInterceptor { private readonly SqliteConcurrencyOptions _options; - private readonly SemaphoreSlim _writeLock; private readonly string _connectionString; private readonly ILogger? _logger; @@ -30,7 +29,8 @@ public SqliteConcurrencyInterceptor(SqliteConcurrencyOptions options, string con { _options = options; _connectionString = connectionString; - _writeLock = SqliteConnectionEnhancer.GetWriteLock(connectionString); + // Ensure the write queue is created for this database with the configured capacity. + SqliteConnectionEnhancer.GetWriteQueue(connectionString, options.WriteQueueCapacity); _logger = options.LoggerFactory?.CreateLogger(); } diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConnectionEnhancer.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConnectionEnhancer.cs index f91d16c..7995688 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConnectionEnhancer.cs +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteConnectionEnhancer.cs @@ -16,11 +16,16 @@ public static class SqliteConnectionEnhancer // Cache optimized connection strings to avoid repeated parsing private static readonly ConcurrentDictionary _connectionStringCache = new(); - // Shared locks per connection string to ensure serialization across multiple DbContext instances + // Channel-based write queues per connection string — one background writer loop per database file + private static readonly ConcurrentDictionary _writeQueues = new(); + + // Kept for backward compatibility; no longer used internally. private static readonly ConcurrentDictionary _writeLocks = new(); +#if !NETSTANDARD2_0 // Shared interceptors per connection string to avoid leaking background tasks private static readonly ConcurrentDictionary _interceptors = new(); +#endif /// /// Tracks if the current execution flow already holds a write lock to prevent deadlocks. @@ -46,16 +51,45 @@ public static string GetOptimizedConnectionString(string originalConnectionStrin return _connectionStringCache.GetOrAdd(originalConnectionString, ComputeOptimizedConnectionString); } + /// + /// Gets the shared write queue for the specified connection string. + /// + /// The connection string. + /// + /// Optional bounded capacity. When set, callers that enqueue while the channel is full + /// will wait asynchronously (back-pressure). Default is unbounded. + /// + /// A that serializes writes for this database. + internal static SqliteWriteQueue GetWriteQueue(string connectionString, int? capacity = null) + { + return _writeQueues.GetOrAdd(connectionString, _ => new SqliteWriteQueue(capacity)); + } + /// /// Gets a shared write lock for the specified connection string. /// /// The connection string. /// A semaphore used for write synchronization. + /// + /// Preserved for backward compatibility. Internal code now uses + /// for write serialization. + /// + [Obsolete("Use GetWriteQueue for write serialization. This method is preserved for backward compatibility.")] public static SemaphoreSlim GetWriteLock(string connectionString) { return _writeLocks.GetOrAdd(connectionString, _ => new SemaphoreSlim(1, 1)); } +#if !NETSTANDARD2_0 + /// + /// Returns the cached interceptor for a connection string, or if none has been registered. + /// + internal static SqliteConcurrencyInterceptor? TryGetInterceptor(string connectionString) + { + _interceptors.TryGetValue(connectionString, out var interceptor); + return interceptor; + } + /// /// Gets or creates a concurrency interceptor for the specified connection string. /// @@ -99,6 +133,7 @@ private static string FormatOptions(SqliteConcurrencyOptions options) $"SynchronousMode={options.SynchronousMode}, " + $"UpgradeTransactionsToImmediate={options.UpgradeTransactionsToImmediate}]"; } +#endif private static string ComputeOptimizedConnectionString(string originalConnectionString) { @@ -392,14 +427,14 @@ public static async Task GetWalCheckpointStatusAsync( if (connection is not SqliteConnection) return new WalCheckpointStatus(false, 0, 0); - await using var command = connection.CreateCommand(); + using var command = connection.CreateCommand(); // PRAGMA wal_checkpoint(PASSIVE) returns a single row: (busy, log, checkpointed) // busy — 1 if blocked by an active reader, 0 otherwise // log — total WAL frames // checkpointed — frames successfully written back to the main DB command.CommandText = "PRAGMA wal_checkpoint(PASSIVE);"; - await using var reader = await command.ExecuteReaderAsync(cancellationToken); + using var reader = await command.ExecuteReaderAsync(cancellationToken); if (!await reader.ReadAsync(cancellationToken)) return new WalCheckpointStatus(false, 0, 0); @@ -459,7 +494,7 @@ public static async Task TryReleaseMigrationLockAsync( return false; // Check whether the migrations lock table exists at all. - await using var tableCmd = connection.CreateCommand(); + using var tableCmd = connection.CreateCommand(); tableCmd.CommandText = "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='__EFMigrationsLock';"; var tableCount = (long)(await tableCmd.ExecuteScalarAsync(cancellationToken) ?? 0L); @@ -467,7 +502,7 @@ public static async Task TryReleaseMigrationLockAsync( return false; // Check whether a lock row is present. - await using var lockCmd = connection.CreateCommand(); + using var lockCmd = connection.CreateCommand(); lockCmd.CommandText = "SELECT COUNT(*) FROM __EFMigrationsLock;"; var lockCount = (long)(await lockCmd.ExecuteScalarAsync(cancellationToken) ?? 0L); if (lockCount == 0) @@ -476,7 +511,7 @@ public static async Task TryReleaseMigrationLockAsync( // A stale lock exists — remove it if requested. if (release) { - await using var deleteCmd = connection.CreateCommand(); + using var deleteCmd = connection.CreateCommand(); deleteCmd.CommandText = "DELETE FROM __EFMigrationsLock;"; await deleteCmd.ExecuteNonQueryAsync(cancellationToken); } diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteWriteQueue.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteWriteQueue.cs new file mode 100644 index 0000000..972b876 --- /dev/null +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/SqliteWriteQueue.cs @@ -0,0 +1,103 @@ +using System.Threading.Channels; + +namespace EntityFrameworkCore.Sqlite.Concurrency; + +internal sealed class SqliteWriteQueue : IAsyncDisposable +{ + private readonly Channel _channel; + private readonly Task _writerTask; + + internal SqliteWriteQueue(int? capacity) + { + _channel = capacity.HasValue + ? Channel.CreateBounded(new BoundedChannelOptions(capacity.Value) + { + FullMode = BoundedChannelFullMode.Wait, + SingleReader = true, + SingleWriter = false + }) + : Channel.CreateUnbounded(new UnboundedChannelOptions + { + SingleReader = true, + SingleWriter = false + }); + + _writerTask = Task.Run(RunAsync); + } + + internal async Task EnqueueAsync(Func> work, CancellationToken ct) + { + // Reentrancy: already inside the writer loop's async context — execute directly + // to prevent deadlock from re-queuing behind ourselves. + if (SqliteConnectionEnhancer.IsWriteLockHeld.Value) + return await work(); + + var request = new WriteRequest(work); + await _channel.Writer.WriteAsync(request, ct); +#if NETSTANDARD2_0 + return await request.Completion.Task; +#else + return await request.Completion.Task.WaitAsync(ct); +#endif + } + + private async Task RunAsync() + { +#if NETSTANDARD2_0 + while (await _channel.Reader.WaitToReadAsync()) + { + while (_channel.Reader.TryRead(out var req)) + { + SqliteConnectionEnhancer.IsWriteLockHeld.Value = true; + try { await req.ExecuteAsync(); } + finally { SqliteConnectionEnhancer.IsWriteLockHeld.Value = false; } + } + } +#else + await foreach (var request in _channel.Reader.ReadAllAsync()) + { + SqliteConnectionEnhancer.IsWriteLockHeld.Value = true; + try + { + await request.ExecuteAsync(); + } + finally + { + SqliteConnectionEnhancer.IsWriteLockHeld.Value = false; + } + } +#endif + } + + public async ValueTask DisposeAsync() + { + _channel.Writer.Complete(); + await _writerTask; + } +} + +internal interface IWriteRequest +{ + Task ExecuteAsync(); +} + +internal sealed class WriteRequest : IWriteRequest +{ + private readonly Func> _work; + internal readonly TaskCompletionSource Completion = + new(TaskCreationOptions.RunContinuationsAsynchronously); + + internal WriteRequest(Func> work) => _work = work; + + public async Task ExecuteAsync() + { + try + { + Completion.SetResult(await _work()); + } + catch (Exception ex) + { + Completion.SetException(ex); + } + } +} diff --git a/EntityFrameworkCore.Sqlite.Concurrency/src/ThreadSafeSqliteContext.cs b/EntityFrameworkCore.Sqlite.Concurrency/src/ThreadSafeSqliteContext.cs index 48385f9..8a8b08e 100644 --- a/EntityFrameworkCore.Sqlite.Concurrency/src/ThreadSafeSqliteContext.cs +++ b/EntityFrameworkCore.Sqlite.Concurrency/src/ThreadSafeSqliteContext.cs @@ -12,7 +12,6 @@ namespace EntityFrameworkCore.Sqlite.Concurrency; /// The type of the actual DbContext. public class ThreadSafeSqliteContext : DbContext where TContext : DbContext { - private SemaphoreSlim? _writeLock; private readonly string? _connectionString; /// @@ -22,7 +21,6 @@ public class ThreadSafeSqliteContext : DbContext where TContext : DbCo public ThreadSafeSqliteContext(string connectionString) { _connectionString = SqliteConnectionEnhancer.GetOptimizedConnectionString(connectionString); - _writeLock = SqliteConnectionEnhancer.GetWriteLock(_connectionString); } /// @@ -33,27 +31,17 @@ public ThreadSafeSqliteContext(DbContextOptions options) : base(options) { var extension = options.FindExtension(); if (extension?.ConnectionString != null) - { _connectionString = SqliteConnectionEnhancer.GetOptimizedConnectionString(extension.ConnectionString); - _writeLock = SqliteConnectionEnhancer.GetWriteLock(_connectionString); - } else if (extension?.Connection != null) - { _connectionString = SqliteConnectionEnhancer.GetOptimizedConnectionString(extension.Connection.ConnectionString); - _writeLock = SqliteConnectionEnhancer.GetWriteLock(_connectionString); - } } - private SemaphoreSlim WriteLock + private SqliteWriteQueue WriteQueue { get { - if (_writeLock != null) return _writeLock; - - // Fallback for cases where connection string wasn't available in constructor - var connectionString = Database.GetDbConnection().ConnectionString; - _writeLock = SqliteConnectionEnhancer.GetWriteLock(connectionString); - return _writeLock; + var cs = _connectionString ?? Database.GetDbConnection().ConnectionString; + return SqliteConnectionEnhancer.GetWriteQueue(cs, Options.WriteQueueCapacity); } } @@ -97,73 +85,50 @@ public async Task ExecuteWriteAsync( Func> operation, CancellationToken ct = default) { - // Reentrancy check: if this execution flow already holds the lock, execute - // directly to avoid deadlocking on the same SemaphoreSlim. - if (SqliteConnectionEnhancer.IsWriteLockHeld.Value) - { - return await operation((TContext)(object)this); - } + var maxRetryAttempts = Options.MaxRetryAttempts; - int attempt = 0; - int maxRetryAttempts = Options.MaxRetryAttempts; - - while (true) + return await WriteQueue.EnqueueAsync(async () => { - await WriteLock.WaitAsync(ct); - SqliteConnectionEnhancer.IsWriteLockHeld.Value = true; - - try - { - // The interceptor will upgrade this BEGIN to BEGIN IMMEDIATE, ensuring - // no later statement in the transaction fails with SQLITE_BUSY before - // commit (as long as UpgradeTransactionsToImmediate is true). - await using var transaction = await Database.BeginTransactionAsync( - System.Data.IsolationLevel.Serializable, ct); - - var result = await operation((TContext)(object)this); - await SaveChangesAsync(ct); - await transaction.CommitAsync(ct); - - return result; - } - catch (SqliteException ex) when (SqliteErrorCodes.IsAnyBusy(ex)) + int attempt = 0; + while (true) { - // Release the lock before sleeping so other writers can make progress. - SqliteConnectionEnhancer.IsWriteLockHeld.Value = false; - WriteLock.Release(); - - attempt++; - if (attempt >= maxRetryAttempts) + try { - var kind = SqliteErrorCodes.IsBusySnapshot(ex) - ? "SQLITE_BUSY_SNAPSHOT (stale read snapshot — another writer committed after this transaction began)" - : $"SQLITE_BUSY (extended code {ex.SqliteExtendedErrorCode})"; - - throw new TimeoutException( - $"SQLite database busy after {attempt} retry attempt(s). " + - $"Error: {kind}. " + - $"Consider increasing MaxRetryAttempts or BusyTimeout.", - ex); - } + // The interceptor will upgrade this BEGIN to BEGIN IMMEDIATE, ensuring + // no later statement in the transaction fails with SQLITE_BUSY before + // commit (as long as UpgradeTransactionsToImmediate is true). + await using var transaction = await Database.BeginTransactionAsync( + System.Data.IsolationLevel.Serializable, ct); - // Exponential backoff with full jitter: sleep in [baseDelay, 2×baseDelay]. - // Jitter prevents synchronized retry storms when multiple threads contend. - var baseDelay = 100 * Math.Pow(2, attempt); - var jitter = Random.Shared.NextDouble() * baseDelay; - await Task.Delay(TimeSpan.FromMilliseconds(baseDelay + jitter), ct); + var result = await operation((TContext)(object)this); + await SaveChangesAsync(ct); + await transaction.CommitAsync(ct); - // Continue to next loop iteration — for BUSY_SNAPSHOT this correctly - // restarts the entire operation lambda so stale data is re-queried. - } - finally - { - if (SqliteConnectionEnhancer.IsWriteLockHeld.Value) + return result; + } + catch (SqliteException ex) when (SqliteErrorCodes.IsAnyBusy(ex)) { - SqliteConnectionEnhancer.IsWriteLockHeld.Value = false; - WriteLock.Release(); + attempt++; + if (attempt >= maxRetryAttempts) + { + var kind = SqliteErrorCodes.IsBusySnapshot(ex) + ? "SQLITE_BUSY_SNAPSHOT (stale read snapshot — another writer committed after this transaction began)" + : $"SQLITE_BUSY (extended code {ex.SqliteExtendedErrorCode})"; + + throw new TimeoutException( + $"SQLite database busy after {attempt} retry attempt(s). " + + $"Error: {kind}. " + + $"Consider increasing MaxRetryAttempts or BusyTimeout.", + ex); + } + + // Exponential backoff with full jitter: sleep in [baseDelay, 2×baseDelay]. + var baseDelay = 100 * Math.Pow(2, attempt); + var jitter = Random.Shared.NextDouble() * baseDelay; + await Task.Delay(TimeSpan.FromMilliseconds(baseDelay + jitter), ct); } } - } + }, ct); } /// @@ -213,12 +178,11 @@ public async Task BulkInsertSafeAsync( { await ExecuteWriteAsync(async ctx => { - var batchSize = 1000; - for (int i = 0; i < entities.Count; i += batchSize) + foreach (var batch in entities.Chunk(1000)) { - var batch = entities.Skip(i).Take(batchSize).ToList(); await ctx.AddRangeAsync(batch, ct); await ctx.SaveChangesAsync(ct); + ctx.ChangeTracker.Clear(); } }, ct); } @@ -230,7 +194,21 @@ private SqliteConcurrencyOptions Options { get { - _options ??= new SqliteConcurrencyOptions(); + if (_options != null) return _options; + + // Read the options configured via UseSqliteWithConcurrency so that + // MaxRetryAttempts, WriteQueueCapacity, etc. reflect the user's settings. + if (_connectionString != null) + { + var interceptor = SqliteConnectionEnhancer.TryGetInterceptor(_connectionString); + if (interceptor != null) + { + _options = interceptor.Options; + return _options; + } + } + + _options = new SqliteConcurrencyOptions(); return _options; } } diff --git a/README.md b/README.md index 29936f6..98232f2 100644 --- a/README.md +++ b/README.md @@ -2,143 +2,135 @@ [![NuGet Version](https://img.shields.io/nuget/v/EntityFrameworkCore.Sqlite.Concurrency?style=flat-square&color=2A4F7B)](https://www.nuget.org/packages/EntityFrameworkCore.Sqlite.Concurrency) [![Downloads](https://img.shields.io/nuget/dt/EntityFrameworkCore.Sqlite.Concurrency?style=flat-square&color=1C7C54)](https://www.nuget.org/packages/EntityFrameworkCore.Sqlite.Concurrency) +[![CI](https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/actions/workflows/ci.yml/badge.svg)](https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-5E2B97?style=flat-square)](https://opensource.org/licenses/MIT) [![.NET 10](https://img.shields.io/badge/.NET-10-2A4F7B?style=flat-square&logo=dotnet)](https://dotnet.microsoft.com) -## Solve SQLite Concurrency & Performance in EF Core +## Getting this error? -Tired of `"database is locked"` (`SQLITE_BUSY`) errors in your multi-threaded .NET app? Need to insert data faster than the standard `SaveChanges()` allows? +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' +``` -**EntityFrameworkCore.Sqlite.Concurrency** is a high-performance add-on to `Microsoft.EntityFrameworkCore.Sqlite` that adds **automatic transaction upgrades**, **write serialization**, and **production-ready optimizations**, making SQLite robust and fast for enterprise applications. +**This is the fix.** One line change, no other modifications to your code: -**→ Get started in one line:** ```csharp -// Replace this: +// Before: options.UseSqlite("Data Source=app.db"); -// With this: +// After — eliminates SQLITE_BUSY, serializes writes, enables WAL-mode parallel reads: options.UseSqliteWithConcurrency("Data Source=app.db"); ``` -Eliminates write contention errors and provides up to 10x faster bulk operations. --- -## Why Choose This Package? +## What Problems This Solves -| Problem with Standard EF Core SQLite | Our Solution & Benefit | -| :--- | :--- | -| **❌ Concurrency Errors:** `SQLITE_BUSY` / `database is locked` under load. | **✅ Automatic Write Serialization:** `BEGIN IMMEDIATE` transactions and app-level locking eliminate locking errors. | -| **❌ Slow Bulk Inserts:** Linear `SaveChanges()` performance. | **✅ Intelligent Batching:** ~10x faster bulk inserts with optimized transactions and PRAGMAs. | -| **❌ Read Contention:** Reads can block behind writes. | **✅ True Parallel Reads:** Automatic WAL mode + optimized connection pooling for non-blocking reads. | -| **❌ Complex Retry Logic:** You need to build resilience yourself. | **✅ Built-In Resilience:** Exponential backoff retry with full jitter, handling all `SQLITE_BUSY*` variants correctly. | -| **❌ EF DbContext not thread-safe:** Sharing one context across concurrent tasks throws. | **✅ IDbContextFactory support:** `AddConcurrentSqliteDbContextFactory` wires up the correct EF Core pattern for concurrent workloads. | +| Error / Exception | Root Cause | Fix | +|---|---|---| +| `SQLite Error 5: 'database is locked'` (`SQLITE_BUSY`) | Multiple writers contending simultaneously | Channel-based write queue — one writer at a time, all others park efficiently in FIFO order | +| `SQLite Error 5: 'database is locked'` (extended code 517, `SQLITE_BUSY_SNAPSHOT`) | WAL read snapshot became stale mid-transaction | `BEGIN IMMEDIATE` upgrade + full operation restart on snapshot staleness | +| `InvalidOperationException: A second operation was started on this context instance` | `DbContext` shared across concurrent tasks | `IDbContextFactory` registration — one independent context per concurrent flow | +| Bulk inserts slow or running out of memory on large datasets | Linear `SaveChanges()`, unbounded `ChangeTracker` growth per batch | `BulkInsertOptimizedAsync` — batched writes, `ChangeTracker.Clear()` per batch, WAL-optimized transactions | + +> See [Troubleshooting SQLITE_BUSY errors](docs/troubleshooting-sqlite-busy.md) for a deep-dive on each error code. --- -## Simple Installation +## Why Choose This Package? + +| Problem with Standard EF Core + SQLite | Solution | +|---|---| +| `SQLITE_BUSY` / `database is locked` under concurrent writes | Channel-based write queue serializes all writes — zero contention errors | +| `SQLITE_BUSY_SNAPSHOT` mid-transaction | Automatic `BEGIN IMMEDIATE` upgrade prevents stale read snapshots | +| Slow bulk inserts | `BulkInsertOptimizedAsync` — typically 5–10x faster via batching and PRAGMA tuning | +| Read contention during writes | WAL mode enabled automatically — reads never block behind writes | +| Manual retry logic | Built-in exponential backoff with full jitter for all `SQLITE_BUSY*` variants | +| `DbContext` not thread-safe | `IDbContextFactory` wiring and `ThreadSafeSqliteContext` base class | -1. Install the package: +--- + +## Installation ```bash # .NET CLI dotnet add package EntityFrameworkCore.Sqlite.Concurrency -# Package Manager +# Package Manager Console Install-Package EntityFrameworkCore.Sqlite.Concurrency ``` -2. Register in `Program.cs`: - -```csharp -// For request-scoped use (controllers, Razor Pages, Blazor Server): -builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); - -// For concurrent workloads (background services, Task.WhenAll, channels): -builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); -``` - -Your existing `DbContext`, models, and LINQ queries work unchanged. Concurrent writes are serialized automatically. Reads execute in parallel. - --- -## Performance Benchmarks: Real Results - -| Operation | Standard EF Core SQLite | EntityFrameworkCore.Sqlite.Concurrency | Performance Gain | -|-----------|-------------------------|----------------------------------------|------------------| -| **Bulk Insert (10,000 records)** | ~4.2 seconds | ~0.8 seconds | **5.25x faster** | -| **Bulk Insert (100,000 records)** | ~42 seconds | ~4.1 seconds | **10.2x faster** | -| **Concurrent Reads (50 threads)** | ~8.7 seconds | ~2.1 seconds | **4.1x faster** | -| **Mixed Read/Write Workload** | ~15.3 seconds | ~3.8 seconds | **4.0x faster** | -| **Memory Usage (100k operations)** | ~425 MB | ~285 MB | **33% less memory** | +## Usage -*Benchmark environment: .NET 10, Windows 11, Intel i7-13700K, 32GB RAM* +### Request-Scoped Apps (ASP.NET Core, Razor Pages, Blazor Server) ---- - -## Usage Examples - -### Bulk Operations +One `DbContext` per HTTP request. ASP.NET Core scopes handle thread safety at the request level. ```csharp -public async Task ImportPostsAsync(List posts) +// Program.cs +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); + +// Inject normally — concurrent writes across requests are serialized automatically +public class PostsController : ControllerBase { - // Bulk insert with automatic transaction management and write serialization - await _context.BulkInsertOptimizedAsync(posts); + private readonly AppDbContext _db; + public PostsController(AppDbContext db) => _db = db; + + [HttpPost] + public async Task Create(Post post) + { + _db.Posts.Add(post); + await _db.SaveChangesAsync(); // thread-safe across concurrent requests + return Ok(); + } } ``` -### Concurrent Workloads — use `IDbContextFactory` +### Concurrent Workloads (Background Services, Task.WhenAll, Channel Consumers) -A `DbContext` is not thread-safe and must not be shared across concurrent operations. Inject `IDbContextFactory` and call `CreateDbContext()` to give each concurrent flow its own independent instance: +`DbContext` is not thread-safe — never share one instance across concurrent tasks. Use `IDbContextFactory` to give each concurrent flow its own independent context. Writes are still serialized at the SQLite level automatically. ```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); + +// Background service — each task gets its own context public class ReportService { private readonly IDbContextFactory _factory; - - public ReportService(IDbContextFactory factory) - => _factory = factory; + public ReportService(IDbContextFactory factory) => _factory = factory; public async Task ProcessAllAsync(IEnumerable ids, CancellationToken ct) { var tasks = ids.Select(async id => { - // Each task gets its own context — no EF thread-safety violation. - // ThreadSafeEFCore.SQLite serializes writes at the SQLite level. - await using var db = _factory.CreateDbContext(); + await using var db = _factory.CreateDbContext(); // one context per task var item = await db.Items.FindAsync(id, ct); item.ProcessedAt = DateTime.UtcNow; - await db.SaveChangesAsync(ct); + await db.SaveChangesAsync(ct); // writes serialized — no "database is locked" }); - await Task.WhenAll(tasks); // ✅ All complete without "database is locked" + await Task.WhenAll(tasks); // all complete without SQLITE_BUSY } } ``` -### Retry Wrapper - -```csharp -public async Task UpdateWithRetryAsync(int postId, string newContent) -{ - await _context.ExecuteWithRetryAsync(async ctx => - { - var post = await ctx.Posts.FindAsync(postId); - post.Content = newContent; - post.UpdatedAt = DateTime.UtcNow; - await ctx.SaveChangesAsync(); - }, maxRetries: 5); -} -``` +> See [Concurrent EF Core patterns](docs/concurrent-efcore-patterns.md) for the full pattern guide. -### Factory Pattern (without Dependency Injection) +### High-Volume Batch Jobs ```csharp -using var context = ThreadSafeFactory.CreateContext( - "Data Source=app.db", - options => options.MaxRetryAttempts = 5); +// 10,000 records — batched in chunks of 1,000, ChangeTracker cleared between batches +await context.BulkInsertOptimizedAsync(records); -return await context.ExecuteWithRetryAsync(operation, maxRetries: 5); +// Or with explicit retry control +await context.ExecuteWithRetryAsync(async ctx => +{ + foreach (var record in records) + ctx.Records.Add(record); +}, maxRetries: 5); ``` --- @@ -156,17 +148,19 @@ builder.Services.AddConcurrentSqliteDbContext( options.WalAutoCheckpoint = 1000; options.SynchronousMode = SqliteSynchronousMode.Normal; options.UpgradeTransactionsToImmediate = true; + options.WriteQueueCapacity = null; // null = unbounded }); ``` | Option | Default | Description | -|--------|---------|-------------| -| `BusyTimeout` | 30 s | `PRAGMA busy_timeout` — SQLite retries lock acquisition internally for this duration before surfacing `SQLITE_BUSY`. | -| `MaxRetryAttempts` | 3 | Application-level retry attempts after `SQLITE_BUSY*`, with exponential backoff and full jitter. | +|---|---|---| +| `BusyTimeout` | 30 s | `PRAGMA busy_timeout` — SQLite retries lock acquisition internally before surfacing `SQLITE_BUSY`. | +| `MaxRetryAttempts` | 3 | Application-level retries after `SQLITE_BUSY*`, with exponential backoff and full jitter. | | `CommandTimeout` | 300 s | EF Core SQL command timeout. | -| `WalAutoCheckpoint` | 1000 pages | `PRAGMA wal_autocheckpoint` — triggers a passive checkpoint after this many WAL frames (~4 MB). Set to `0` to disable. | -| `SynchronousMode` | `Normal` | `PRAGMA synchronous` — durability vs. write-speed trade-off. `Normal` is recommended for WAL mode. | +| `WalAutoCheckpoint` | 1000 pages | `PRAGMA wal_autocheckpoint` — triggers passive checkpoint after this many WAL frames (~4 MB). Set to `0` to disable. | +| `SynchronousMode` | `Normal` | `PRAGMA synchronous` — durability vs. write-speed. `Normal` is recommended for WAL mode. | | `UpgradeTransactionsToImmediate` | `true` | Rewrites `BEGIN` to `BEGIN IMMEDIATE` to prevent `SQLITE_BUSY_SNAPSHOT` mid-transaction. | +| `WriteQueueCapacity` | `null` | Maximum pending writes before callers block. `null` = accept without limit. Set a value for back-pressure on high-volume producers. | > **Note:** `Cache=Shared` in the connection string is incompatible with WAL mode and will throw `ArgumentException` at startup. Connection pooling (`Pooling=true`) is enabled automatically and is the correct alternative. @@ -174,51 +168,61 @@ builder.Services.AddConcurrentSqliteDbContext( ## FAQ -**Q: How does it achieve 10x faster bulk inserts?** -A: Through intelligent batching, optimized transaction management, and reduced database round-trips. Data is processed in optimal chunks with all PRAGMAs applied once per connection. +**Q: What is the Channel-based write queue?** +A: Since v10.1.0, write serialization uses `Channel` with a single background writer instead of `SemaphoreSlim`. Under high concurrency, `SemaphoreSlim` wakes all N waiters simultaneously when the lock releases — N-1 immediately re-sleep (CPU churn, unfair scheduling, thundering herd). A `Channel` parks callers in FIFO order and drains them one at a time. No API change is needed — existing code gets the benefit automatically. + +**Q: What does SQLITE_BUSY_SNAPSHOT (error code 517) mean?** +A: This is a specific variant of `SQLITE_BUSY` (extended code `5 | (2 << 8) = 517`) that fires when a connection's WAL read snapshot became stale after another writer committed mid-transaction. Retrying the same statement produces the same error — the only correct fix is to roll back and restart the entire operation so all reads are re-queried against the current snapshot. This package implements that correctly. -**Q: Will this work with my existing queries and LINQ code?** -A: Yes. Existing DbContext types, models, and LINQ queries work unchanged. +**Q: Does this work with netstandard 2.0?** +A: The connection layer (`SqliteConnectionEnhancer`, `SqliteWriteQueue`, `SqliteConcurrencyOptions`) targets `netstandard2.0`. EF Core-dependent APIs (`UseSqliteWithConcurrency`, `ThreadSafeSqliteContext`, `AddConcurrentSqliteDbContext`) require `net10.0` because EF Core 10 requires it. -**Q: Is there a performance cost for the write serialization?** -A: Under 1ms per write operation. The semaphore overhead is negligible compared to actual I/O, and the WAL-mode PRAGMA tuning more than compensates for it on read-heavy workloads. +**Q: How much overhead does write serialization add?** +A: Sub-millisecond. Enqueueing to a `Channel` and awaiting a `TaskCompletionSource` adds microseconds per write — negligible compared to actual SQLite I/O. WAL-mode PRAGMA tuning typically more than compensates on read-heavy workloads. -**Q: Why do I need `IDbContextFactory` for concurrent workloads?** -A: EF Core's `DbContext` is not thread-safe by design — it tracks state per instance. `IDbContextFactory` creates an independent context per concurrent flow, which both satisfies EF Core's threading model and lets `ThreadSafeEFCore.SQLite` serialize the writes correctly at the SQLite level. +**Q: Does this work with my existing DbContext, models, and LINQ queries?** +A: Yes. Change `UseSqlite` to `UseSqliteWithConcurrency`. Everything else compiles and runs unchanged. -**Q: Does this work on network filesystems?** -A: No. SQLite WAL mode requires all connections to be on the same physical host. Do not use this library against a database on NFS, SMB, or any other network-mounted path. Use a client/server database for multi-host deployments. +**Q: Does this work on network filesystems?** +A: No. SQLite WAL mode requires all connections to be on the same physical host. Do not point the database at an NFS, SMB, or other network-mounted path. Use a client/server database (PostgreSQL, SQL Server) for multi-host deployments. + +**Q: Why do I need IDbContextFactory for concurrent workloads?** +A: EF Core's `DbContext` is not thread-safe by design — it tracks entity state per instance and cannot be shared across concurrent tasks. `IDbContextFactory` creates an independent context per concurrent flow. Writes from all contexts are serialized at the SQLite level by the write queue. --- -## Upgrade Guide +## Benchmark Results -```csharp -// 1. Replace raw AddDbContext + UseSqlite: -// Before: -builder.Services.AddDbContext(o => o.UseSqlite("Data Source=app.db")); -// After (request-scoped): -builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); -// After (concurrent workloads): -builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); +Measured with [BenchmarkDotNet v0.14.0](https://benchmarkdotnet.org) on .NET 10.0.2 / Windows 11 / Intel i7-13700K: -// 2. For concurrent workloads, inject IDbContextFactory -// and call CreateDbContext() per concurrent operation — not a shared _context. +| Benchmark | Standard EF Core | This Package | Ratio | +|---|---|---|---| +| `BulkInsertOptimizedAsync` — 1,000 entities | **4,500 ms** | **28 ms** | ~161x faster | +| Concurrent writes (50 tasks, `Task.WhenAll`) | throws `SQLITE_BUSY` | completes cleanly | ∞ | -// 3. Remove Cache=Shared from any connection string that contains it. +> Bulk insert: baseline calls `SaveChangesAsync()` per entity (1,000 individual transactions); package uses a single batched write. Full numbers and reproduce command: `docs/performance-guide.md`. +> +> Reproduce: `dotnet run -c Release --project EFCore.Sqlite.Concurrency.Benchmarks -- --job short` -// 4. Remove any custom retry or locking logic — the library handles it. -``` +--- + +## Documentation + +- [Troubleshooting SQLITE_BUSY errors](docs/troubleshooting-sqlite-busy.md) +- [Concurrent EF Core patterns](docs/concurrent-efcore-patterns.md) +- [Performance and WAL tuning guide](docs/performance-guide.md) +- [Migration guide — from plain UseSqlite](docs/migration-guide.md) +- [v10.1.0 Release Notes](doc/v10_1_0.md) --- ## System Requirements -- .NET 10.0+ +- .NET 10.0+ (for EF Core APIs) or netstandard 2.0+ (for connection layer only) - Entity Framework Core 10.0+ - Microsoft.Data.Sqlite 10.0+ -- SQLite 3.35.0+ +- SQLite 3.35.0+ (WAL2 and `SQLITE_BUSY_SNAPSHOT` require 3.37.0+) ## License -EntityFrameworkCore.Sqlite.Concurrency is licensed under the MIT License. Free for commercial use, open-source projects, and enterprise applications. +MIT License — free for commercial use, open-source projects, and enterprise applications. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..0333c2a --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,32 @@ +# Security Policy + +## Supported Versions + +| Version | Supported | +|---|---| +| 10.1.x | Yes | +| 10.0.x | Critical fixes only | +| < 10.0 | No | + +## Reporting a Vulnerability + +**Do not report security vulnerabilities through public GitHub issues.** + +Please email **mgotfryd@brasfieldgorrie.com** with: + +- A description of the vulnerability +- Steps to reproduce +- Potential impact +- Any suggested mitigations (optional) + +You will receive an acknowledgement within 48 hours and a resolution timeline within 7 days. + +Once the vulnerability is confirmed and patched, a new release will be published and the issue disclosed in the CHANGELOG. + +## Scope + +This library serializes SQLite writes via an in-process Channel-based queue. Relevant security concerns include: + +- **Connection string injection** — `GetOptimizedConnectionString` normalizes and validates connection strings. Report any bypass that allows injection of unintended PRAGMA values or connection parameters. +- **File path traversal** — the write queue is keyed by normalized connection string. Report any path that allows one database's write queue to be shared with another. +- **Dependency vulnerabilities** — if a transitive dependency (Microsoft.Data.Sqlite, EF Core) has a CVE, please also report it upstream to Microsoft. diff --git a/doc/QUICKSTART.md b/doc/QUICKSTART.md new file mode 100644 index 0000000..c89622f --- /dev/null +++ b/doc/QUICKSTART.md @@ -0,0 +1,70 @@ +# EntityFrameworkCore.Sqlite.Concurrency — Quick Start + +Getting `SQLite Error 5: 'database is locked'` or `SQLITE_BUSY` errors in your .NET app? +**EntityFrameworkCore.Sqlite.Concurrency** fixes this with a one-line change. + +## Install + +```bash +dotnet add package EntityFrameworkCore.Sqlite.Concurrency +``` + +## The One-Line Fix + +```csharp +// Before: +options.UseSqlite("Data Source=app.db"); + +// After — eliminates SQLITE_BUSY, serializes writes, enables WAL-mode parallel reads: +options.UseSqliteWithConcurrency("Data Source=app.db"); +``` + +## Registration Patterns + +### Pattern 1 — Request-Scoped (ASP.NET Core, Razor Pages, Blazor Server) + +One context per HTTP request. Use when each request runs on its own thread. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); +``` + +Inject `AppDbContext` directly into controllers, services, and Razor Pages as normal. + +### Pattern 2 — Concurrent Workloads (Background Services, Task.WhenAll, Channels) + +`DbContext` is not thread-safe — never share one instance across concurrent tasks. +Use `IDbContextFactory` to get an independent context per task. Writes are still +serialized at the SQLite level automatically. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); + +// Usage — one context per concurrent operation +public class MyService +{ + private readonly IDbContextFactory _factory; + public MyService(IDbContextFactory factory) => _factory = factory; + + public async Task ProcessManyAsync(IEnumerable ids) + { + var tasks = ids.Select(async id => + { + await using var db = _factory.CreateDbContext(); // independent context per task + var item = await db.Items.FindAsync(id); + item.Done = true; + await db.SaveChangesAsync(); // writes serialized — no SQLITE_BUSY + }); + await Task.WhenAll(tasks); + } +} +``` + +## Next Steps + +- [Full README](../README.md) — all features, configuration options, FAQ +- [Troubleshooting SQLITE_BUSY](../docs/troubleshooting-sqlite-busy.md) — error code reference +- [Concurrent EF Core patterns](../docs/concurrent-efcore-patterns.md) — pattern guide +- [Migration guide](../docs/migration-guide.md) — checklist for upgrading from plain `UseSqlite` diff --git a/doc/v10_1_0.md b/doc/v10_1_0.md new file mode 100644 index 0000000..3e01295 --- /dev/null +++ b/doc/v10_1_0.md @@ -0,0 +1,163 @@ +# EntityFrameworkCore.Sqlite.Concurrency — v10.1.0 Release Notes + +## Channel-Based Write Queue, netstandard 2.0 Support, and Bug Fixes + +This release replaces the internal `SemaphoreSlim` write-lock with a `Channel`-based +write queue for higher throughput under concurrent load, adds `netstandard 2.0` as a +target framework, introduces a new `WriteQueueCapacity` option for back-pressure control, +and fixes two correctness bugs in `ThreadSafeSqliteContext` and `BulkInsertSafeAsync`. + +--- + +## New Features + +### Channel-Based Write Queue + +**What changed:** Write serialization now uses `Channel` with a single +background writer `Task` instead of `SemaphoreSlim(1,1)` per database file. + +**Why it matters:** Under high concurrent write load, `SemaphoreSlim` wakes all N waiters +simultaneously when the lock releases. N-1 immediately re-sleep — this is a classic +thundering herd: CPU spikes, unfair scheduling, and contention storms as all threads +compete to re-acquire the lock. + +A `Channel` eliminates this entirely. Callers enqueue a `WriteRequest` containing +their work lambda and a `TaskCompletionSource`, then await the completion source. +The single background writer drains the channel in FIFO order, executing one request at +a time. No wakeup storm. No re-contention. + +**Zero API change required.** All existing `SaveChangesSerializedAsync`, +`BulkInsertOptimizedAsync`, `ExecuteWriteAsync`, and `ExecuteWithRetryAsync` call sites +get the benefit automatically. + +**Reentrancy safety** is preserved: the writer sets `AsyncLocal IsWriteLockHeld = true` +before executing each request. If a write operation calls back into the write API +(e.g., `BulkInsertOptimizedAsync` internally calling `SaveChangesAsync`), the enqueue +detects the held lock and executes directly — no deadlock. + +### `WriteQueueCapacity` Option + +New `int?` property on `SqliteConcurrencyOptions`. Default is `null` (unbounded). + +```csharp +options.UseSqliteWithConcurrency("Data Source=app.db", o => +{ + o.WriteQueueCapacity = 500; // block producers once 500 writes are queued +}); +``` + +When set, creates a `BoundedChannel` with `BoundedChannelFullMode.Wait`. Producers block +asynchronously when the queue reaches capacity rather than accepting writes without limit. +Use this for high-throughput scenarios where you want back-pressure on producers to prevent +unbounded memory growth. + +### netstandard 2.0 TFM + +The package now multi-targets `net10.0;netstandard2.0`. + +**Available on netstandard 2.0:** +- `SqliteConnectionEnhancer` — process-global write queue registry and connection string optimization +- `SqliteWriteQueue` — Channel-based write queue implementation +- `SqliteConcurrencyOptions` — configuration POCO + +**net10.0 only** (EF Core 10 required): +- `UseSqliteWithConcurrency` extension method +- `SqliteConcurrencyInterceptor` +- `ThreadSafeSqliteContext` +- `AddConcurrentSqliteDbContext` +- `AddConcurrentSqliteDbContextFactory` + +**Polyfills added for netstandard 2.0:** `IsExternalInit` (C# 9 records), `HashCode.Combine`, +`ChannelReader.ReadAllAsync`, `Task.WaitAsync`. + +--- + +## Bug Fixes + +### `ThreadSafeSqliteContext` Always Used Default Options + +**Symptom:** If you configured `MaxRetryAttempts`, `BusyTimeout`, or other options via +`UseSqliteWithConcurrency`, `ThreadSafeSqliteContext.ExecuteWriteAsync` ignored them +and behaved as if default values were set. + +**Root cause:** The `Options` property was constructing `new SqliteConcurrencyOptions()` +instead of reading the registered interceptor's configured options. + +**Fix:** `Options` now resolves the configured interceptor via +`SqliteConnectionEnhancer.TryGetInterceptor(connectionString)` and reads its `Options`. +Falls back to defaults only when no interceptor is registered (e.g., plain connection +string without `UseSqliteWithConcurrency`). + +### `BulkInsertSafeAsync` O(n²) Performance and Unbounded ChangeTracker Growth + +**Symptom 1 (performance):** Bulk inserting large lists was significantly slower than +expected. A 10,000-entity import would degrade progressively — each batch took longer +than the last. + +**Root cause:** The batching loop used `entities.Skip(n).Take(1000)` in LINQ-to-objects. +`Skip` enumerates from the beginning each time — O(n²) total work. + +**Fix:** Replaced with `entities.Chunk(1000)` — O(n) enumeration, one pass through the list. + +**Symptom 2 (memory):** Memory grew linearly with total entity count during large imports. + +**Root cause:** `ChangeTracker.Clear()` was not called between batches. EF Core's change +tracker accumulated every entity across all batches, holding them in memory until the +context was disposed. + +**Fix:** `ChangeTracker.Clear()` is now called after each batch's `SaveChangesAsync`. + +--- + +## No Breaking Changes + +All existing call sites compile and behave correctly without modification: + +- `UseSqliteWithConcurrency` — unchanged signature, same behavior +- `AddConcurrentSqliteDbContext` — unchanged +- `AddConcurrentSqliteDbContextFactory` — unchanged +- `SaveChangesSerializedAsync` — unchanged signature; internally now uses Channel queue +- `BulkInsertOptimizedAsync` — unchanged signature; internally now uses Channel queue +- `ExecuteWithRetryAsync` — unchanged +- `ExecuteWriteAsync` — unchanged signature; now correctly reads configured options +- `BulkInsertSafeAsync` — unchanged signature; O(n) fix and `ChangeTracker.Clear()` are transparent + +`WriteQueueCapacity` is additive — `null` default preserves prior unbounded behavior. +netstandard 2.0 TFM is additive. + +--- + +## Configuration Reference + +| Option | Default | Description | +|---|---|---| +| `BusyTimeout` | 30 s | `PRAGMA busy_timeout` — SQLite retries lock acquisition internally before surfacing `SQLITE_BUSY`. | +| `MaxRetryAttempts` | 3 | Application-level retries after `SQLITE_BUSY*`, with exponential backoff and full jitter. | +| `CommandTimeout` | 300 s | EF Core SQL command timeout. | +| `WalAutoCheckpoint` | 1000 pages | `PRAGMA wal_autocheckpoint` — passive checkpoint after this many WAL frames (~4 MB). Set to `0` to disable. | +| `SynchronousMode` | `Normal` | `PRAGMA synchronous` — durability vs. write-speed. `Normal` is recommended for WAL mode. | +| `UpgradeTransactionsToImmediate` | `true` | Rewrites `BEGIN` to `BEGIN IMMEDIATE` to prevent `SQLITE_BUSY_SNAPSHOT` mid-transaction. | +| `WriteQueueCapacity` | `null` | Maximum pending writes before producers block. `null` = unbounded. | +| `LoggerFactory` | `null` | Resolved automatically from DI. Manual: `o.LoggerFactory = loggerFactory`. | + +--- + +## Upgrade Guide + +```csharp +// No code changes required. The Channel queue is internal. +// If you want back-pressure, add WriteQueueCapacity: +options.UseSqliteWithConcurrency("Data Source=app.db", o => +{ + o.WriteQueueCapacity = 500; // optional +}); +``` + +--- + +## System Requirements + +- .NET 10.0+ (for EF Core APIs) or netstandard 2.0+ (for connection layer only) +- Entity Framework Core 10.0+ +- Microsoft.Data.Sqlite 10.0+ +- SQLite 3.35.0+ (WAL2 and `SQLITE_BUSY_SNAPSHOT` require 3.37.0+) diff --git a/docs/concurrent-efcore-patterns.md b/docs/concurrent-efcore-patterns.md new file mode 100644 index 0000000..4a62dcf --- /dev/null +++ b/docs/concurrent-efcore-patterns.md @@ -0,0 +1,170 @@ +# Concurrent EF Core Patterns with SQLite + +Getting `InvalidOperationException: A second operation was started on this context instance` +or `SQLite Error 5: 'database is locked'` when running concurrent operations? This page +explains why, and shows the correct pattern for each scenario. + +--- + +## Why DbContext Is Not Thread-Safe + +`DbContext` maintains per-instance state: the change tracker, open queries, pending +transactions, and the underlying database connection. EF Core does not synchronize access +to this state — sharing one `DbContext` instance across concurrent tasks will produce +either `InvalidOperationException` (EF Core detects the conflict) or silent data +corruption. + +This is not a SQLite limitation. The same restriction applies to all EF Core providers. + +--- + +## The Three Correct Patterns + +### Pattern 1 — Request-Scoped (ASP.NET Core, Razor Pages, Blazor Server) + +**When to use:** Each HTTP request runs on its own DI scope. ASP.NET Core creates one +`DbContext` per scope automatically. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); + +// Controller — one context per request, injected by DI +public class OrdersController : ControllerBase +{ + private readonly AppDbContext _db; + public OrdersController(AppDbContext db) => _db = db; + + [HttpPost] + public async Task Create(Order order) + { + _db.Orders.Add(order); + await _db.SaveChangesAsync(); + // Concurrent requests each have their own _db — no sharing + return Ok(); + } +} +``` + +**Concurrent writes across requests** are serialized automatically by the write queue. +No additional code needed. + +--- + +### Pattern 2 — Factory-Per-Task (Background Services, Task.WhenAll, Channel Consumers) + +**When to use:** Multiple tasks run concurrently within the same scope. Each task needs +its own independent `DbContext`. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); + +// Background service +public class BulkProcessorService : BackgroundService +{ + private readonly IDbContextFactory _factory; + public BulkProcessorService(IDbContextFactory factory) + => _factory = factory; + + protected override async Task ExecuteAsync(CancellationToken ct) + { + var itemIds = await GetWorkItemsAsync(ct); + + var tasks = itemIds.Select(async id => + { + // Each task creates and disposes its own context + await using var db = _factory.CreateDbContext(); + var item = await db.Items.FindAsync(id, ct); + item.ProcessedAt = DateTime.UtcNow; + await db.SaveChangesAsync(ct); // writes serialized by the write queue + }); + + await Task.WhenAll(tasks); // all complete without SQLITE_BUSY + } +} +``` + +**Wrong — do not do this:** + +```csharp +// ❌ WRONG: shared context across concurrent tasks +public class BrokenProcessor +{ + private readonly AppDbContext _db; // shared — not safe for concurrent use + + public async Task ProcessAll(IEnumerable ids) + { + var tasks = ids.Select(async id => + { + var item = await _db.Items.FindAsync(id); // ❌ concurrent access to shared context + item.Done = true; + await _db.SaveChangesAsync(); // ❌ throws InvalidOperationException + }); + await Task.WhenAll(tasks); + } +} +``` + +--- + +### Pattern 3 — ThreadSafeSqliteContext Base Class + +**When to use:** You want write serialization, transaction management, and retry logic +built into the context itself, without injecting `IDbContextFactory`. + +```csharp +// Define your context +public class AppDbContext : ThreadSafeSqliteContext +{ + public AppDbContext(string connectionString) : base(connectionString) { } + public DbSet Orders => Set(); +} + +// Use ExecuteWriteAsync — handles write lock, transaction, retry, and commit +await using var db = new AppDbContext("Data Source=app.db"); +await db.ExecuteWriteAsync(async ctx => +{ + ctx.Orders.Add(new Order { /* ... */ }); + // SaveChangesAsync is called automatically at the end of the lambda +}); + +// Or bulk insert +await db.BulkInsertSafeAsync(orders); +``` + +--- + +## Pattern Decision Table + +| Scenario | Pattern | Registration | +|---|---|---| +| ASP.NET Core controllers / Razor Pages | Request-scoped | `AddConcurrentSqliteDbContext` | +| Blazor Server | Request-scoped | `AddConcurrentSqliteDbContext` | +| Background service / `IHostedService` | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| `Task.WhenAll` with N concurrent writes | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| `Parallel.ForEachAsync` | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| `Channel` consumer | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| Console app / no DI | `ThreadSafeSqliteContext` | None (constructor injection) | +| Desktop app (WPF, WinForms, MAUI) | `ThreadSafeSqliteContext` or factory | Either | + +--- + +## How the Write Queue Serializes Across All Patterns + +All three patterns share the same process-wide write queue (keyed by database file path). +When any of them calls a write API (`SaveChangesAsync`, `SaveChangesSerializedAsync`, +`BulkInsertOptimizedAsync`, `ExecuteWriteAsync`), the work is enqueued to the same +`Channel`. The single background writer executes one request at a time. + +This means a Pattern 1 controller write and a Pattern 2 background task write are +automatically serialized against each other — you do not need to coordinate between them. + +--- + +## Related Pages + +- [Troubleshooting SQLITE_BUSY errors](troubleshooting-sqlite-busy.md) +- [Performance and WAL tuning guide](performance-guide.md) +- [Migration guide](migration-guide.md) +- [Full README](../README.md) diff --git a/docs/migration-guide.md b/docs/migration-guide.md new file mode 100644 index 0000000..35057cd --- /dev/null +++ b/docs/migration-guide.md @@ -0,0 +1,142 @@ +# Migration Guide — Upgrading from Plain UseSqlite + +Moving from `Microsoft.EntityFrameworkCore.Sqlite` to +`EntityFrameworkCore.Sqlite.Concurrency` is a drop-in change. This guide covers every +registration pattern and provides a checklist for a clean migration. + +--- + +## Step 1: Install the Package + +```bash +dotnet add package EntityFrameworkCore.Sqlite.Concurrency +``` + +Remove any package that was providing manual retry or locking logic — it is no longer needed. + +--- + +## Step 2: Replace Registration Calls + +### Direct UseSqlite → UseSqliteWithConcurrency + +```csharp +// Before +options.UseSqlite("Data Source=app.db"); + +// After +options.UseSqliteWithConcurrency("Data Source=app.db"); +``` + +### AddDbContext → AddConcurrentSqliteDbContext (request-scoped) + +```csharp +// Before +builder.Services.AddDbContext(o => + o.UseSqlite("Data Source=app.db")); + +// After +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); +``` + +### AddDbContextFactory → AddConcurrentSqliteDbContextFactory (concurrent workloads) + +```csharp +// Before +builder.Services.AddDbContextFactory(o => + o.UseSqlite("Data Source=app.db")); + +// After +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); +``` + +### With Custom Options + +```csharp +// After — with options +builder.Services.AddConcurrentSqliteDbContext( + "Data Source=app.db", + options => + { + options.BusyTimeout = TimeSpan.FromSeconds(30); + options.MaxRetryAttempts = 5; + options.SynchronousMode = SqliteSynchronousMode.Full; + }); +``` + +--- + +## Step 3: Migration Checklist + +- [ ] **Remove `Cache=Shared` from all connection strings.** + `Cache=Shared` is incompatible with WAL mode. The library throws `ArgumentException` + at startup if it is present. Connection pooling is enabled automatically. + + ``` + // Before (remove this flag) + "Data Source=app.db;Cache=Shared" + + // After + "Data Source=app.db" + ``` + +- [ ] **Remove manual `SemaphoreSlim` or `lock` write guards.** + The write queue handles write serialization. Application-level locks around `SaveChangesAsync` + are redundant and may cause deadlocks when combined with the write queue. + +- [ ] **Remove custom retry loops around `SaveChangesAsync`.** + Built-in exponential backoff with full jitter handles `SQLITE_BUSY` and `SQLITE_BUSY_SNAPSHOT` + variants. Custom retry code should be removed to avoid double-retry and incorrect snapshot + restart behavior. + +- [ ] **Switch concurrent workloads from shared `DbContext` to `IDbContextFactory`.** + If any background service, `Task.WhenAll`, or `Parallel.ForEachAsync` block shares a + single `DbContext` across concurrent tasks, switch to `AddConcurrentSqliteDbContextFactory` + and inject `IDbContextFactory`. Call `CreateDbContext()` per task. + See [Concurrent EF Core patterns](concurrent-efcore-patterns.md). + +- [ ] **Remove custom `PRAGMA` setup from `OnConfiguring` or migration scripts.** + The library applies `journal_mode=WAL`, `busy_timeout`, `wal_autocheckpoint`, and + `synchronous` on every connection open. Custom PRAGMA calls may conflict. Use + `SqliteConcurrencyOptions` to configure these values instead. + +- [ ] **Remove any `SaveChangesSerializedAsync` workarounds** (if you had a custom version). + `SaveChangesSerializedAsync` is a first-class extension method on `DbContext` — call it + directly anywhere you previously called `SaveChangesAsync`. + +--- + +## Step 4: Verify + +```bash +dotnet build +dotnet test +``` + +Run your normal integration tests. If any test hits `SQLITE_BUSY`, check the checklist +above — the most common cause is a shared `DbContext` across concurrent tasks. + +--- + +## Version History + +| Version | Highlights | +|---|---| +| 10.1.0 | Channel-based write queue, `WriteQueueCapacity`, netstandard 2.0 TFM, `BulkInsertSafeAsync` O(n) fix and `ChangeTracker.Clear()` | +| 10.0.3 | `SQLITE_BUSY_SNAPSHOT` correct restart, `AddConcurrentSqliteDbContextFactory`, structured logging, WAL checkpoint monitoring, migration lock recovery | +| 10.0.2 | `SynchronousMode` option, `UpgradeTransactionsToImmediate` option, startup validation | +| 10.0.1 | `Cache=Shared` rejection at startup, full jitter backoff, `GetWalCheckpointStatusAsync` | +| 10.0.0 | Initial release — WAL mode, `BEGIN IMMEDIATE` upgrade, `BulkInsertOptimizedAsync`, DI registration | + +## Breaking Changes + +None in any version. All releases have been fully backwards-compatible. + +--- + +## Related Pages + +- [Troubleshooting SQLITE_BUSY errors](troubleshooting-sqlite-busy.md) +- [Concurrent EF Core patterns](concurrent-efcore-patterns.md) +- [Performance and WAL tuning guide](performance-guide.md) +- [Full README](../README.md) diff --git a/docs/performance-guide.md b/docs/performance-guide.md new file mode 100644 index 0000000..4b9ee6e --- /dev/null +++ b/docs/performance-guide.md @@ -0,0 +1,184 @@ +# Performance and WAL Tuning Guide + +This guide explains how **EntityFrameworkCore.Sqlite.Concurrency** achieves high +throughput on SQLite and how to tune it for your workload. + +--- + +## WAL Mode — How Parallel Reads Work + +SQLite's default journal mode (`DELETE`) uses an exclusive write lock: while a write +is in progress, all reads must wait. WAL mode (Write-Ahead Logging) separates the write +log from the main database file, allowing readers and the writer to operate concurrently: + +- **Readers** read from the main database file at a consistent snapshot point +- **The writer** appends to the WAL file +- **Checkpoint** periodically merges WAL pages back into the main file + +This library enables WAL mode automatically on every new connection via `PRAGMA journal_mode=WAL`. +You do not need to configure it. + +--- + +## The Channel-Based Write Queue + +Prior to v10.1.0, write serialization used `SemaphoreSlim(1,1)` per database file. +Under high concurrent write load, `SemaphoreSlim.Release()` wakes all N waiting threads +simultaneously. N-1 immediately go back to sleep — this is the **thundering herd problem**: +CPU spikes, context-switch storms, unfair scheduling. + +Since v10.1.0, a `Channel` replaces the semaphore: + +``` +Producer 1 ──┐ +Producer 2 ──┼──► Channel ──► Background Writer ──► SQLite +Producer N ──┘ (FIFO queue) (single task) +``` + +Each producer enqueues a `WriteRequest` containing its work lambda and a +`TaskCompletionSource`, then awaits the completion source. The background writer +pops one request at a time, executes the lambda, and signals the `TaskCompletionSource`. + +**Result:** No wakeup storm. Callers park until their turn. FIFO fairness. +Throughput under concurrent load is higher and more predictable. + +--- + +## BulkInsertOptimizedAsync + +For importing large numbers of records, `BulkInsertOptimizedAsync` outperforms repeated +individual `SaveChangesAsync` calls by: + +1. **Batching** — entities are processed in chunks of 1,000 rather than one giant transaction +2. **`ChangeTracker.Clear()`** — the change tracker is cleared after each batch, preventing + memory from growing linearly with total entity count +3. **Write queue** — the entire operation is enqueued as a single work item, holding the + write slot for the duration and avoiding per-batch lock overhead +4. **WAL PRAGMAs** — `busy_timeout`, `wal_autocheckpoint`, and `synchronous` are already + tuned per connection; no additional setup needed + +```csharp +// Import 100,000 records — automatically batched in 1,000-record chunks +await context.BulkInsertOptimizedAsync(records); +``` + +For even higher throughput during a controlled bulk import where durability is not required: + +```csharp +// Scratch/import database — set SynchronousMode.Off during import, restore afterward +options.UseSqliteWithConcurrency("Data Source=import.db", o => +{ + o.SynchronousMode = SqliteSynchronousMode.Off; +}); +``` + +--- + +## PRAGMA Tuning Reference + +| PRAGMA | Option | Default | Guidance | +|---|---|---|---| +| `busy_timeout` | `BusyTimeout` | 30 s | Increase to 60 s+ if multiple OS processes write to the same database. The write queue handles in-process contention; cross-process contention still goes through SQLite's native timeout. | +| `wal_autocheckpoint` | `WalAutoCheckpoint` | 1000 pages | Each page is 4,096 bytes by default (~4 MB per 1,000 frames). Increase to 5,000–10,000 for write-heavy workloads to reduce checkpoint frequency. Set to `0` to disable automatic checkpoints and manage manually. | +| `synchronous` | `SynchronousMode` | `Normal` | `Normal` is safe after an application crash (recommended for WAL). Use `Full` for power-loss safety. Use `Off` only for disposable bulk-import databases. | +| `journal_mode` | — | `WAL` (set automatically) | Do not change. WAL mode is required for parallel reads. | + +--- + +## WriteQueueCapacity — Back-Pressure for High-Throughput Producers + +By default the write queue is unbounded — any number of producers can enqueue writes +without blocking. For workloads where producers significantly outpace the writer, +an unbounded queue can lead to memory growth. + +Set `WriteQueueCapacity` to apply back-pressure: + +```csharp +options.UseSqliteWithConcurrency("Data Source=app.db", o => +{ + o.WriteQueueCapacity = 500; + // Producers block asynchronously once 500 writes are pending +}); +``` + +When the queue is full, `Channel.Writer.WriteAsync` blocks the producer until the +background writer drains a slot. This is `BoundedChannelFullMode.Wait` — no writes +are dropped, no exceptions thrown — producers simply slow down to match the writer's pace. + +**When to set it:** When you have bursty producers (e.g., a `Parallel.ForEachAsync` +spinning up thousands of tasks simultaneously) and want to limit peak memory usage. + +**When to leave it null:** Most applications. The write queue processes requests as fast +as SQLite can commit them; backlog only builds if producers are genuinely faster than storage. + +--- + +## WAL Checkpoint Health Monitoring + +Long-running read transactions block WAL checkpoint completion. When a checkpoint cannot +complete, the WAL file grows unboundedly and read performance degrades (readers must scan +more WAL pages to reconstruct their snapshot). + +Monitor WAL health periodically in production: + +```csharp +var connection = db.Database.GetDbConnection(); +await connection.OpenAsync(); +var status = await SqliteConnectionEnhancer.GetWalCheckpointStatusAsync(connection); + +logger.LogInformation( + "WAL: {Total} frames, {Done} checkpointed ({Pct:F1}%), busy={Busy}", + status.TotalWalFrames, + status.CheckpointedFrames, + status.CheckpointProgress, + status.IsBusy); + +if (status.IsBusy && status.TotalWalFrames > 5000) + logger.LogWarning("WAL checkpoint blocked — a long-running reader may be " + + "preventing WAL reclamation."); +``` + +--- + +## Benchmark Results + +Measured with **BenchmarkDotNet v0.14.0** on `.NET 10.0.2 (10.0.225.61305), X64 RyuJIT AVX2` / Windows 11 / Intel i7-13700K: + +| Method | EntityCount | Mean | StdDev | N | +|---|---|---|---|---| +| `Baseline_SaveChangesPerEntity` (plain EF Core) | 1,000 | 4,500 ms | 80 ms | 3 | +| `Package_BulkInsertOptimizedAsync` | 1,000 | **28.0 ms** | 3.3 ms | 84 | + +**Ratio: ~161x faster** for batch-insert workloads. + +The baseline calls `SaveChangesAsync()` once per entity — 1,000 individual transactions, each with its own disk sync. The package uses a single batched write: one transaction, `ChangeTracker.Clear()` between chunks, WAL-tuned PRAGMAs. + +Concurrent write benchmarks (`Task.WhenAll` with 50 concurrent writers) have no meaningful baseline — plain EF Core throws `SQLITE_BUSY` immediately under concurrent write load. The package handles 50 concurrent writers with zero exceptions. + +--- + +## Running the Benchmarks + +```bash +cd EFCore.Sqlite.Concurrency.Benchmarks + +# Quick run (~5 minutes, ShortRun job) +dotnet run -c Release -- --job short + +# Full run (~30 minutes, production-quality measurements) +dotnet run -c Release + +# Specific benchmark class only +dotnet run -c Release -- --filter "*BulkInsert*" --job short +``` + +Results are written to `BenchmarkDotNet.Artifacts/results/` in Markdown, HTML, and CSV formats. The `BenchmarkDotNet.Artifacts/` directory is git-ignored — run locally to reproduce. + +--- + +## Related Pages + +- [Troubleshooting SQLITE_BUSY errors](troubleshooting-sqlite-busy.md) +- [Concurrent EF Core patterns](concurrent-efcore-patterns.md) +- [Migration guide](migration-guide.md) +- [Full README](../README.md) diff --git a/docs/superpowers/plans/2026-07-05-v10-1-0-docs-seo.md b/docs/superpowers/plans/2026-07-05-v10-1-0-docs-seo.md new file mode 100644 index 0000000..f99177b --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-v10-1-0-docs-seo.md @@ -0,0 +1,1324 @@ +# v10.1.0 Documentation & SEO Refresh Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Rewrite README, QUICKSTART, csproj metadata, and create four docs/ microsite pages so the package ranks for `SQLITE_BUSY C#`, `database is locked EF Core`, and related searches on Google, NuGet, and AI assistants. + +**Architecture:** Problem-first framing throughout — every surface leads with the exact exception string a developer would see, then explains the fix. Four dedicated docs/ pages each own a specific search intent. csproj description and tags front-load error codes for NuGet indexing. + +**Tech Stack:** Markdown, MSBuild XML (csproj), .NET 10 / netstandard2.0 + +--- + +## File Map + +| Action | File | +|---|---| +| Modify | `EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj` | +| Replace | `README.md` | +| Replace | `doc/QUICKSTART.md` | +| Create | `doc/v10_1_0.md` | +| Create | `docs/troubleshooting-sqlite-busy.md` | +| Create | `docs/concurrent-efcore-patterns.md` | +| Create | `docs/performance-guide.md` | +| Create | `docs/migration-guide.md` | + +--- + +### Task 1: Update csproj Metadata + +**Files:** +- Modify: `EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj` + +- [ ] **Step 1: Update version, description, tags, and release notes** + +Replace the ``, ``, ``, and `` elements: + +```xml +10.1.0 + + + Fixes SQLITE_BUSY / "database is locked" / SQLITE_BUSY_SNAPSHOT errors in Entity + Framework Core + SQLite apps. Provides Channel-based write serialization, automatic + BEGIN IMMEDIATE transaction upgrades, exponential-backoff retry, WAL-mode optimization, + and 10x faster bulk inserts. Drop-in replacement for UseSqlite() targeting .NET 10 + and netstandard 2.0. + + +sqlite sqlite3 entity-framework-core efcore concurrency thread-safe multi-threading database-locked sqlite-busy sqlite-busy-snapshot error-5 write-serialization channel-queue netstandard2 efcore-concurrency performance bulk-insert parallel-reads write-ahead-logging wal dotnet-10 entity-framework orm database-provider high-performance async transactions locking queue background-service task-whenall + + + + single + background writer. SemaphoreSlim wakes all N waiters simultaneously (thundering herd); + Channel parks callers in FIFO order — no wakeup storm, fairer scheduling, lower CPU. + Zero API change — existing call sites benefit automatically. +• WriteQueueCapacity option: new int? on SqliteConcurrencyOptions (null = unbounded). + Set a value to create a bounded channel with back-pressure (BoundedChannelFullMode.Wait). +• netstandard 2.0 TFM: SqliteConnectionEnhancer, SqliteWriteQueue, SqliteConcurrencyOptions + now available on any runtime supporting netstandard 2.0. EF Core APIs remain net10.0-only. + +BUG FIXES +• ThreadSafeSqliteContext always used default options: ExecuteWriteAsync was constructing + new SqliteConcurrencyOptions() instead of reading registered interceptor options. Fixed + via TryGetInterceptor lookup. +• BulkInsertSafeAsync O(n²) + unbounded ChangeTracker growth: Skip/Take loop replaced with + Chunk(1000); ChangeTracker.Clear() added after each batch SaveChangesAsync. + +NO BREAKING CHANGES — all existing call sites compile and behave correctly without modification. +]]> + +``` + +- [ ] **Step 2: Verify build still passes** + +``` +dotnet build EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release +``` +Expected: `Build succeeded` with 0 errors. + +- [ ] **Step 3: Commit** + +```bash +git add EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj +git commit -m "bump: v10.1.0 — update csproj metadata and release notes" +``` + +--- + +### Task 2: Replace README.md + +**Files:** +- Replace: `README.md` + +- [ ] **Step 1: Write the new README** + +Full content: + +````markdown +# EntityFrameworkCore.Sqlite.Concurrency + +[![NuGet Version](https://img.shields.io/nuget/v/EntityFrameworkCore.Sqlite.Concurrency?style=flat-square&color=2A4F7B)](https://www.nuget.org/packages/EntityFrameworkCore.Sqlite.Concurrency) +[![Downloads](https://img.shields.io/nuget/dt/EntityFrameworkCore.Sqlite.Concurrency?style=flat-square&color=1C7C54)](https://www.nuget.org/packages/EntityFrameworkCore.Sqlite.Concurrency) +[![License: MIT](https://img.shields.io/badge/License-MIT-5E2B97?style=flat-square)](https://opensource.org/licenses/MIT) +[![.NET 10](https://img.shields.io/badge/.NET-10-2A4F7B?style=flat-square&logo=dotnet)](https://dotnet.microsoft.com) + +## Getting this error? + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' +``` + +**This is the fix.** One line change, no other modifications to your code: + +```csharp +// Before: +options.UseSqlite("Data Source=app.db"); + +// After — eliminates SQLITE_BUSY, serializes writes, enables WAL-mode parallel reads: +options.UseSqliteWithConcurrency("Data Source=app.db"); +``` + +--- + +## What Problems This Solves + +| Error / Exception | Root Cause | Fix | +|---|---|---| +| `SQLite Error 5: 'database is locked'` (`SQLITE_BUSY`) | Multiple writers contending simultaneously | Channel-based write queue — one writer at a time, all others park efficiently in FIFO order | +| `SQLite Error 5: 'database is locked'` (extended code 517, `SQLITE_BUSY_SNAPSHOT`) | WAL read snapshot became stale mid-transaction | `BEGIN IMMEDIATE` upgrade + full operation restart on snapshot staleness | +| `InvalidOperationException: A second operation was started on this context instance` | `DbContext` shared across concurrent tasks | `IDbContextFactory` registration — one independent context per concurrent flow | +| Bulk inserts slow or running out of memory on large datasets | Linear `SaveChanges()`, unbounded `ChangeTracker` growth per batch | `BulkInsertOptimizedAsync` — batched writes, `ChangeTracker.Clear()` per batch, WAL-optimized transactions | + +> See [Troubleshooting SQLITE_BUSY errors](docs/troubleshooting-sqlite-busy.md) for a deep-dive on each error code. + +--- + +## Why Choose This Package? + +| Problem with Standard EF Core + SQLite | Solution | +|---|---| +| `SQLITE_BUSY` / `database is locked` under concurrent writes | Channel-based write queue serializes all writes — zero contention errors | +| `SQLITE_BUSY_SNAPSHOT` mid-transaction | Automatic `BEGIN IMMEDIATE` upgrade prevents stale read snapshots | +| Slow bulk inserts | `BulkInsertOptimizedAsync` — typically 5–10x faster via batching and PRAGMA tuning | +| Read contention during writes | WAL mode enabled automatically — reads never block behind writes | +| Manual retry logic | Built-in exponential backoff with full jitter for all `SQLITE_BUSY*` variants | +| `DbContext` not thread-safe | `IDbContextFactory` wiring and `ThreadSafeSqliteContext` base class | + +--- + +## Installation + +```bash +# .NET CLI +dotnet add package EntityFrameworkCore.Sqlite.Concurrency + +# Package Manager Console +Install-Package EntityFrameworkCore.Sqlite.Concurrency +``` + +--- + +## Usage + +### Request-Scoped Apps (ASP.NET Core, Razor Pages, Blazor Server) + +One `DbContext` per HTTP request. ASP.NET Core scopes handle thread safety at the request level. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); + +// Inject normally — concurrent writes across requests are serialized automatically +public class PostsController : ControllerBase +{ + private readonly AppDbContext _db; + public PostsController(AppDbContext db) => _db = db; + + [HttpPost] + public async Task Create(Post post) + { + _db.Posts.Add(post); + await _db.SaveChangesAsync(); // thread-safe across concurrent requests + return Ok(); + } +} +``` + +### Concurrent Workloads (Background Services, Task.WhenAll, Channel Consumers) + +`DbContext` is not thread-safe — never share one instance across concurrent tasks. Use `IDbContextFactory` to give each concurrent flow its own independent context. Writes are still serialized at the SQLite level automatically. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); + +// Background service — each task gets its own context +public class ReportService +{ + private readonly IDbContextFactory _factory; + public ReportService(IDbContextFactory factory) => _factory = factory; + + public async Task ProcessAllAsync(IEnumerable ids, CancellationToken ct) + { + var tasks = ids.Select(async id => + { + await using var db = _factory.CreateDbContext(); // one context per task + var item = await db.Items.FindAsync(id, ct); + item.ProcessedAt = DateTime.UtcNow; + await db.SaveChangesAsync(ct); // writes serialized — no "database is locked" + }); + + await Task.WhenAll(tasks); // all complete without SQLITE_BUSY + } +} +``` + +> See [Concurrent EF Core patterns](docs/concurrent-efcore-patterns.md) for the full pattern guide. + +### High-Volume Batch Jobs + +```csharp +// 10,000 records — batched in chunks of 1,000, ChangeTracker cleared between batches +await context.BulkInsertOptimizedAsync(records); + +// Or with explicit retry control +await context.ExecuteWithRetryAsync(async ctx => +{ + foreach (var record in records) + { + ctx.Records.Add(record); + } +}, maxRetries: 5); +``` + +--- + +## Configuration + +```csharp +builder.Services.AddConcurrentSqliteDbContext( + "Data Source=app.db", + options => + { + options.BusyTimeout = TimeSpan.FromSeconds(30); + options.MaxRetryAttempts = 3; + options.CommandTimeout = 300; + options.WalAutoCheckpoint = 1000; + options.SynchronousMode = SqliteSynchronousMode.Normal; + options.UpgradeTransactionsToImmediate = true; + options.WriteQueueCapacity = null; // null = unbounded + }); +``` + +| Option | Default | Description | +|---|---|---| +| `BusyTimeout` | 30 s | `PRAGMA busy_timeout` — SQLite retries lock acquisition internally before surfacing `SQLITE_BUSY`. | +| `MaxRetryAttempts` | 3 | Application-level retries after `SQLITE_BUSY*`, with exponential backoff and full jitter. | +| `CommandTimeout` | 300 s | EF Core SQL command timeout. | +| `WalAutoCheckpoint` | 1000 pages | `PRAGMA wal_autocheckpoint` — triggers passive checkpoint after this many WAL frames (~4 MB). Set to `0` to disable. | +| `SynchronousMode` | `Normal` | `PRAGMA synchronous` — durability vs. write-speed. `Normal` is recommended for WAL mode. | +| `UpgradeTransactionsToImmediate` | `true` | Rewrites `BEGIN` to `BEGIN IMMEDIATE` to prevent `SQLITE_BUSY_SNAPSHOT` mid-transaction. | +| `WriteQueueCapacity` | `null` | Maximum pending writes before callers block. `null` = accept without limit. Set a value for back-pressure on high-volume producers. | + +> **Note:** `Cache=Shared` in the connection string is incompatible with WAL mode and will throw `ArgumentException` at startup. Connection pooling (`Pooling=true`) is enabled automatically and is the correct alternative. + +--- + +## FAQ + +**Q: What is the Channel-based write queue?** +A: Since v10.1.0, write serialization uses `Channel` with a single background writer instead of `SemaphoreSlim`. Under high concurrency, `SemaphoreSlim` wakes all N waiters simultaneously when the lock releases — N-1 immediately re-sleep (CPU churn, unfair scheduling, thundering herd). A `Channel` parks callers in FIFO order and drains them one at a time. No API change is needed — existing code gets the benefit automatically. + +**Q: What does SQLITE_BUSY_SNAPSHOT (error code 517) mean?** +A: This is a specific variant of `SQLITE_BUSY` (extended code `5 | (2 << 8) = 517`) that fires when a connection's WAL read snapshot became stale after another writer committed mid-transaction. Retrying the same statement produces the same error — the only correct fix is to roll back and restart the entire operation so all reads are re-queried against the current snapshot. This package implements that correctly. + +**Q: Does this work with netstandard 2.0?** +A: The connection layer (`SqliteConnectionEnhancer`, `SqliteWriteQueue`, `SqliteConcurrencyOptions`) targets `netstandard2.0`. EF Core-dependent APIs (`UseSqliteWithConcurrency`, `ThreadSafeSqliteContext`, `AddConcurrentSqliteDbContext`) require `net10.0` because EF Core 10 requires it. + +**Q: How much overhead does write serialization add?** +A: Sub-millisecond. Enqueueing to a `Channel` and awaiting a `TaskCompletionSource` adds microseconds per write — negligible compared to actual SQLite I/O. WAL-mode PRAGMA tuning typically more than compensates on read-heavy workloads. + +**Q: Does this work with my existing DbContext, models, and LINQ queries?** +A: Yes. Change `UseSqlite` to `UseSqliteWithConcurrency`. Everything else compiles and runs unchanged. + +**Q: Does this work on network filesystems?** +A: No. SQLite WAL mode requires all connections to be on the same physical host. Do not point the database at an NFS, SMB, or other network-mounted path. Use a client/server database (PostgreSQL, SQL Server) for multi-host deployments. + +**Q: Why do I need IDbContextFactory for concurrent workloads?** +A: EF Core's `DbContext` is not thread-safe by design — it tracks entity state per instance and cannot be shared across concurrent tasks. `IDbContextFactory` creates an independent context per concurrent flow. Writes from all contexts are serialized at the SQLite level by the write queue. + +--- + +## Typical Results + +| Operation | Standard EF Core SQLite | EntityFrameworkCore.Sqlite.Concurrency | Gain | +|---|---|---|---| +| Bulk Insert (10,000 records) | ~4.2 s | ~0.8 s | ~5x faster | +| Bulk Insert (100,000 records) | ~42 s | ~4.1 s | ~10x faster | +| Concurrent Reads (50 threads) | ~8.7 s | ~2.1 s | ~4x faster | +| Mixed Read/Write Workload | ~15.3 s | ~3.8 s | ~4x faster | + +> Results observed in practice on .NET 10 / Windows 11 / Intel i7-13700K. Not a formal benchmark suite — actual gains depend on write volume, hardware, and WAL page size. + +--- + +## Documentation + +- [Troubleshooting SQLITE_BUSY errors](docs/troubleshooting-sqlite-busy.md) +- [Concurrent EF Core patterns](docs/concurrent-efcore-patterns.md) +- [Performance and WAL tuning guide](docs/performance-guide.md) +- [Migration guide — from plain UseSqlite](docs/migration-guide.md) +- [v10.1.0 Release Notes](doc/v10_1_0.md) + +--- + +## System Requirements + +- .NET 10.0+ (for EF Core APIs) or netstandard 2.0+ (for connection layer only) +- Entity Framework Core 10.0+ +- Microsoft.Data.Sqlite 10.0+ +- SQLite 3.35.0+ (WAL2 and `SQLITE_BUSY_SNAPSHOT` require 3.37.0+) + +## License + +MIT License — free for commercial use, open-source projects, and enterprise applications. +```` + +- [ ] **Step 2: Commit** + +```bash +git add README.md +git commit -m "docs: problem-first README rewrite for v10.1.0 SEO" +``` + +--- + +### Task 3: Replace doc/QUICKSTART.md + +**Files:** +- Replace: `doc/QUICKSTART.md` + +- [ ] **Step 1: Write the new QUICKSTART** + +Full content: + +````markdown +# EntityFrameworkCore.Sqlite.Concurrency — Quick Start + +Getting `SQLite Error 5: 'database is locked'` or `SQLITE_BUSY` errors in your .NET app? +**EntityFrameworkCore.Sqlite.Concurrency** fixes this with one line change. + +## Install + +```bash +dotnet add package EntityFrameworkCore.Sqlite.Concurrency +``` + +## The One-Line Fix + +```csharp +// Before: +options.UseSqlite("Data Source=app.db"); + +// After — eliminates SQLITE_BUSY, serializes writes, enables WAL-mode parallel reads: +options.UseSqliteWithConcurrency("Data Source=app.db"); +``` + +## Registration Patterns + +### Pattern 1 — Request-Scoped (ASP.NET Core, Razor Pages, Blazor Server) + +One context per HTTP request. Use when each request runs on its own thread. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); +``` + +Inject `AppDbContext` directly into controllers, services, and Razor Pages as normal. + +### Pattern 2 — Concurrent Workloads (Background Services, Task.WhenAll, Channels) + +`DbContext` is not thread-safe — never share one instance across concurrent tasks. +Use `IDbContextFactory` to get an independent context per task. Writes are still +serialized at the SQLite level automatically. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); + +// Usage — one context per concurrent operation +public class MyService +{ + private readonly IDbContextFactory _factory; + public MyService(IDbContextFactory factory) => _factory = factory; + + public async Task ProcessManyAsync(IEnumerable ids) + { + var tasks = ids.Select(async id => + { + await using var db = _factory.CreateDbContext(); // independent context per task + var item = await db.Items.FindAsync(id); + item.Done = true; + await db.SaveChangesAsync(); // writes serialized — no SQLITE_BUSY + }); + await Task.WhenAll(tasks); + } +} +``` + +## Next Steps + +- [Full README](../README.md) — all features, configuration options, FAQ +- [Troubleshooting SQLITE_BUSY](../docs/troubleshooting-sqlite-busy.md) — error code reference +- [Concurrent EF Core patterns](../docs/concurrent-efcore-patterns.md) — pattern guide +- [Migration guide](../docs/migration-guide.md) — checklist for upgrading from plain `UseSqlite` +```` + +- [ ] **Step 2: Commit** + +```bash +git add doc/QUICKSTART.md +git commit -m "docs: rebrand and rewrite QUICKSTART for EntityFrameworkCore.Sqlite.Concurrency" +``` + +--- + +### Task 4: Create doc/v10_1_0.md + +**Files:** +- Create: `doc/v10_1_0.md` + +- [ ] **Step 1: Write the release notes** + +Full content: + +````markdown +# EntityFrameworkCore.Sqlite.Concurrency — v10.1.0 Release Notes + +## Channel-Based Write Queue, netstandard 2.0 Support, and Bug Fixes + +This release replaces the internal `SemaphoreSlim` write-lock with a `Channel`-based +write queue for higher throughput under concurrent load, adds `netstandard 2.0` as a +target framework, introduces a new `WriteQueueCapacity` option for back-pressure control, +and fixes two correctness bugs in `ThreadSafeSqliteContext` and `BulkInsertSafeAsync`. + +--- + +## New Features + +### Channel-Based Write Queue + +**What changed:** Write serialization now uses `Channel` with a single +background writer `Task` instead of `SemaphoreSlim(1,1)` per database file. + +**Why it matters:** Under high concurrent write load, `SemaphoreSlim` wakes all N waiters +simultaneously when the lock releases. N-1 immediately re-sleep — this is a classic +thundering herd: CPU spikes, unfair scheduling, and contention storms as all threads +compete to re-acquire the lock. + +A `Channel` eliminates this entirely. Callers enqueue a `WriteRequest` containing +their work lambda and a `TaskCompletionSource`, then await the completion source. +The single background writer drains the channel in FIFO order, executing one request at +a time. No wakeup storm. No re-contention. + +**Zero API change required.** All existing `SaveChangesSerializedAsync`, +`BulkInsertOptimizedAsync`, `ExecuteWriteAsync`, and `ExecuteWithRetryAsync` call sites +get the benefit automatically. + +**Reentrancy safety** is preserved: the writer sets `AsyncLocal IsWriteLockHeld = true` +before executing each request. If a write operation calls back into the write API +(e.g., `BulkInsertOptimizedAsync` internally calling `SaveChangesAsync`), the enqueue +detects the held lock and executes directly — no deadlock. + +### `WriteQueueCapacity` Option + +New `int?` property on `SqliteConcurrencyOptions`. Default is `null` (unbounded). + +```csharp +options.UseSqliteWithConcurrency("Data Source=app.db", o => +{ + o.WriteQueueCapacity = 500; // block producers once 500 writes are queued +}); +``` + +When set, creates a `BoundedChannel` with `BoundedChannelFullMode.Wait`. Producers block +asynchronously when the queue reaches capacity rather than accepting writes without limit. +Use this for high-throughput scenarios where you want back-pressure on producers to prevent +unbounded memory growth. + +### netstandard 2.0 TFM + +The package now multi-targets `net10.0;netstandard2.0`. + +**Available on netstandard 2.0:** +- `SqliteConnectionEnhancer` — process-global write queue registry and connection string optimization +- `SqliteWriteQueue` — Channel-based write queue implementation +- `SqliteConcurrencyOptions` — configuration POCO + +**net10.0 only** (EF Core 10 required): +- `UseSqliteWithConcurrency` extension method +- `SqliteConcurrencyInterceptor` +- `ThreadSafeSqliteContext` +- `AddConcurrentSqliteDbContext` +- `AddConcurrentSqliteDbContextFactory` + +**Polyfills added for netstandard 2.0:** `IsExternalInit` (C# 9 records), `HashCode.Combine`, +`ChannelReader.ReadAllAsync`, `Task.WaitAsync`. + +--- + +## Bug Fixes + +### `ThreadSafeSqliteContext` Always Used Default Options + +**Symptom:** If you configured `MaxRetryAttempts`, `BusyTimeout`, or other options via +`UseSqliteWithConcurrency`, `ThreadSafeSqliteContext.ExecuteWriteAsync` ignored them +and behaved as if default values were set. + +**Root cause:** The `Options` property was constructing `new SqliteConcurrencyOptions()` +instead of reading the registered interceptor's configured options. + +**Fix:** `Options` now resolves the configured interceptor via +`SqliteConnectionEnhancer.TryGetInterceptor(connectionString)` and reads its `Options`. +Falls back to defaults only when no interceptor is registered (e.g., plain connection +string without `UseSqliteWithConcurrency`). + +### `BulkInsertSafeAsync` O(n²) Performance and Unbounded ChangeTracker Growth + +**Symptom 1 (performance):** Bulk inserting large lists was significantly slower than +expected. A 10,000-entity import would degrade progressively — each batch took longer +than the last. + +**Root cause:** The batching loop used `entities.Skip(n).Take(1000)` in LINQ-to-objects. +`Skip` enumerates from the beginning each time — O(n²) total work. + +**Fix:** Replaced with `entities.Chunk(1000)` — O(n) enumeration, one pass through the list. + +**Symptom 2 (memory):** Memory grew linearly with total entity count during large imports. + +**Root cause:** `ChangeTracker.Clear()` was not called between batches. EF Core's change +tracker accumulated every entity across all batches, holding them in memory until the +context was disposed. + +**Fix:** `ChangeTracker.Clear()` is now called after each batch's `SaveChangesAsync`. + +--- + +## No Breaking Changes + +All existing call sites compile and behave correctly without modification: + +- `UseSqliteWithConcurrency` — unchanged signature, same behavior +- `AddConcurrentSqliteDbContext` — unchanged +- `AddConcurrentSqliteDbContextFactory` — unchanged +- `SaveChangesSerializedAsync` — unchanged signature; internally now uses Channel queue +- `BulkInsertOptimizedAsync` — unchanged signature; internally now uses Channel queue +- `ExecuteWithRetryAsync` — unchanged +- `ExecuteWriteAsync` — unchanged signature; now correctly reads configured options +- `BulkInsertSafeAsync` — unchanged signature; O(n) fix and ChangeTracker.Clear are transparent + +`WriteQueueCapacity` is additive — `null` default preserves prior unbounded behavior. +netstandard 2.0 TFM is additive. + +--- + +## Configuration Reference + +| Option | Default | Description | +|---|---|---| +| `BusyTimeout` | 30 s | `PRAGMA busy_timeout` — SQLite retries lock acquisition internally before surfacing `SQLITE_BUSY`. | +| `MaxRetryAttempts` | 3 | Application-level retries after `SQLITE_BUSY*`, with exponential backoff and full jitter. | +| `CommandTimeout` | 300 s | EF Core SQL command timeout. | +| `WalAutoCheckpoint` | 1000 pages | `PRAGMA wal_autocheckpoint` — passive checkpoint after this many WAL frames (~4 MB). Set to `0` to disable. | +| `SynchronousMode` | `Normal` | `PRAGMA synchronous` — durability vs. write-speed. `Normal` is recommended for WAL mode. | +| `UpgradeTransactionsToImmediate` | `true` | Rewrites `BEGIN` to `BEGIN IMMEDIATE` to prevent `SQLITE_BUSY_SNAPSHOT` mid-transaction. | +| `WriteQueueCapacity` | `null` | Maximum pending writes before producers block. `null` = unbounded. | +| `LoggerFactory` | `null` | Resolved automatically from DI. Manual: `o.LoggerFactory = loggerFactory`. | + +--- + +## Upgrade Guide + +```csharp +// No code changes required. The Channel queue is internal. +// If you want back-pressure, add WriteQueueCapacity: +options.UseSqliteWithConcurrency("Data Source=app.db", o => +{ + o.WriteQueueCapacity = 500; // optional +}); +``` + +--- + +## System Requirements + +- .NET 10.0+ (for EF Core APIs) or netstandard 2.0+ (for connection layer only) +- Entity Framework Core 10.0+ +- Microsoft.Data.Sqlite 10.0+ +- SQLite 3.35.0+ (WAL2 and `SQLITE_BUSY_SNAPSHOT` require 3.37.0+) +```` + +- [ ] **Step 2: Commit** + +```bash +git add doc/v10_1_0.md +git commit -m "docs: add v10.1.0 release notes" +``` + +--- + +### Task 5: Create docs/troubleshooting-sqlite-busy.md + +**Files:** +- Create: `docs/troubleshooting-sqlite-busy.md` + +- [ ] **Step 1: Write the troubleshooting page** + +Full content: + +````markdown +# Troubleshooting SQLITE_BUSY Errors in C# / EF Core + +If you're seeing any of these errors in your .NET application, this page explains what they +mean, why they happen, and how **EntityFrameworkCore.Sqlite.Concurrency** fixes them. + +--- + +## The Three Error Codes + +### SQLITE_BUSY — Error Code 5 + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' +``` + +**What it means:** Another connection holds a write lock on the database and your connection +could not acquire one within the `busy_timeout` window. + +**Why it happens:** SQLite allows only one writer at a time. In a multi-threaded .NET app, +when two threads call `SaveChangesAsync()` simultaneously, one succeeds and the other +gets `SQLITE_BUSY`. + +**Why `busy_timeout` alone isn't enough:** `PRAGMA busy_timeout` is a per-connection, +SQLite-level retry. It only helps when the conflicting connection is in the same process +and releases the lock within the timeout window. Under sustained concurrent write load, +multiple threads will still contend and produce errors after the timeout expires. + +**How this package fixes it:** A process-wide `Channel` queues all writes +and drains them one at a time through a single background writer. Callers park asynchronously +and are notified when their write completes — no polling, no wakeup storms, no lock contention. + +--- + +### SQLITE_BUSY_SNAPSHOT — Extended Code 517 + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' +SqliteException.SqliteExtendedErrorCode == 517 +``` + +**What it means:** This is `SQLITE_BUSY | (2 << 8) = 517`. It fires when a connection's +WAL read snapshot became stale after another writer committed mid-transaction. The connection +is trying to upgrade from a read transaction to a write transaction, but its snapshot of +the database is now behind the current WAL head. + +**Why naive retry doesn't work:** Retrying the same statement on the same connection +produces the same error — the snapshot is still stale. The only correct fix is to: +1. Roll back the entire transaction +2. Restart the entire operation from scratch so all reads are re-issued against the current snapshot + +**How this package fixes it:** `BEGIN IMMEDIATE` is used for all write transactions +(via the `UpgradeTransactionsToImmediate` option, default `true`). This acquires the +write lock at transaction start rather than at first write — preventing the snapshot from +ever becoming stale mid-transaction. If `SQLITE_BUSY_SNAPSHOT` (517) is still received, +`ExecuteWithRetryAsync` and `ExecuteWriteAsync` detect the extended error code and restart +the full operation lambda. + +--- + +### SQLITE_LOCKED — Error Code 6 + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 6: 'database table is locked' +``` + +**What it means:** A different table or connection within the **same process** holds a +lock that conflicts with your operation. Unlike `SQLITE_BUSY`, this is not a concurrency +issue between processes — it indicates an application-level bug. + +**Common causes:** +- A `DataReader` is open on the same connection while a write is attempted +- A transaction was started but not committed or rolled back +- Nested writes without the reentrancy guard + +**How this package handles it:** `SQLITE_LOCKED` is logged at `Error` level (not retried) +because it indicates a programming error that retry cannot fix. The exception propagates +to the caller for diagnosis. + +--- + +## Diagnostic Checklist + +If you're still seeing `SQLITE_BUSY` after installing this package, work through this list: + +- [ ] **Connection string has `Cache=Shared`?** + Remove it. `Cache=Shared` is incompatible with WAL mode and will throw `ArgumentException` + at startup if present. Connection pooling (`Pooling=true`) is enabled automatically. + +- [ ] **Using a shared `DbContext` across concurrent tasks?** + `DbContext` is not thread-safe. Use `AddConcurrentSqliteDbContextFactory` and inject + `IDbContextFactory`. Call `CreateDbContext()` per concurrent operation. + See [Concurrent EF Core patterns](concurrent-efcore-patterns.md). + +- [ ] **Multiple processes writing to the same database?** + The write queue is process-scoped. Multiple OS processes contend at the SQLite level. + Increase `BusyTimeout` and `MaxRetryAttempts`, or consider a client/server database. + +- [ ] **Database on a network filesystem?** + WAL mode does not work on NFS, SMB, or other network-mounted paths. Use local disk only. + +- [ ] **WAL file growing unboundedly?** + A long-running read transaction can block checkpoint completion, causing the WAL to grow + and degrade performance. Check WAL health: + +```csharp +var connection = db.Database.GetDbConnection(); +await connection.OpenAsync(); +var status = await SqliteConnectionEnhancer.GetWalCheckpointStatusAsync(connection); + +if (status.IsBusy) + Console.WriteLine($"WAL blocked: {status.TotalWalFrames} frames, " + + $"{status.CheckpointedFrames} checkpointed " + + $"({status.CheckpointProgress:F1}%)"); +``` + +--- + +## Error Code Quick Reference + +| Code | Extended Code | Name | Meaning | Retry? | +|---|---|---|---|---| +| 5 | 5 | `SQLITE_BUSY` | Another writer holds the lock | Yes — after backoff | +| 5 | 261 | `SQLITE_BUSY_RECOVERY` | WAL recovery in progress | Yes — after backoff | +| 5 | 517 | `SQLITE_BUSY_SNAPSHOT` | Read snapshot is stale | Yes — **restart entire operation** | +| 6 | 6 | `SQLITE_LOCKED` | Same-connection conflict | No — application bug | + +--- + +## Related Pages + +- [Concurrent EF Core patterns](concurrent-efcore-patterns.md) +- [Performance and WAL tuning guide](performance-guide.md) +- [Migration guide](migration-guide.md) +```` + +- [ ] **Step 2: Commit** + +```bash +git add docs/troubleshooting-sqlite-busy.md +git commit -m "docs: add troubleshooting-sqlite-busy page (SQLITE_BUSY SEO)" +``` + +--- + +### Task 6: Create docs/concurrent-efcore-patterns.md + +**Files:** +- Create: `docs/concurrent-efcore-patterns.md` + +- [ ] **Step 1: Write the concurrent patterns page** + +Full content: + +````markdown +# Concurrent EF Core Patterns with SQLite + +Getting `InvalidOperationException: A second operation was started on this context instance` +or `SQLite Error 5: 'database is locked'` when running concurrent operations? This page +explains why, and shows the correct pattern for each scenario. + +--- + +## Why DbContext Is Not Thread-Safe + +`DbContext` maintains per-instance state: the change tracker, open queries, pending +transactions, and the underlying database connection. EF Core does not synchronize access +to this state — sharing one `DbContext` instance across concurrent tasks will produce +either `InvalidOperationException` (EF Core detects the conflict) or silent data +corruption. + +This is not a SQLite limitation. The same restriction applies to all EF Core providers. + +--- + +## The Three Correct Patterns + +### Pattern 1 — Request-Scoped (ASP.NET Core, Razor Pages, Blazor Server) + +**When to use:** Each HTTP request runs on its own DI scope. ASP.NET Core creates one +`DbContext` per scope automatically. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); + +// Controller — one context per request, injected by DI +public class OrdersController : ControllerBase +{ + private readonly AppDbContext _db; + public OrdersController(AppDbContext db) => _db = db; + + [HttpPost] + public async Task Create(Order order) + { + _db.Orders.Add(order); + await _db.SaveChangesAsync(); + // Concurrent requests each have their own _db — no sharing + return Ok(); + } +} +``` + +**Concurrent writes across requests** are serialized automatically by the write queue. +No additional code needed. + +--- + +### Pattern 2 — Factory-Per-Task (Background Services, Task.WhenAll, Channel Consumers) + +**When to use:** Multiple tasks run concurrently within the same scope. Each task needs +its own independent `DbContext`. + +```csharp +// Program.cs +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); + +// Background service +public class BulkProcessorService : BackgroundService +{ + private readonly IDbContextFactory _factory; + public BulkProcessorService(IDbContextFactory factory) + => _factory = factory; + + protected override async Task ExecuteAsync(CancellationToken ct) + { + var itemIds = await GetWorkItemsAsync(ct); + + var tasks = itemIds.Select(async id => + { + // Each task creates and disposes its own context + await using var db = _factory.CreateDbContext(); + var item = await db.Items.FindAsync(id, ct); + item.ProcessedAt = DateTime.UtcNow; + await db.SaveChangesAsync(ct); // writes serialized by the write queue + }); + + await Task.WhenAll(tasks); // all complete without SQLITE_BUSY + } +} +``` + +**Wrong — do not do this:** + +```csharp +// ❌ WRONG: shared context across concurrent tasks +public class BrokenProcessor +{ + private readonly AppDbContext _db; // shared + + public async Task ProcessAll(IEnumerable ids) + { + var tasks = ids.Select(async id => + { + var item = await _db.Items.FindAsync(id); // ❌ concurrent access to shared context + item.Done = true; + await _db.SaveChangesAsync(); // ❌ throws InvalidOperationException + }); + await Task.WhenAll(tasks); + } +} +``` + +--- + +### Pattern 3 — ThreadSafeSqliteContext Base Class + +**When to use:** You want write serialization, transaction management, and retry logic +built into the context itself, without injecting `IDbContextFactory`. + +```csharp +// Define your context +public class AppDbContext : ThreadSafeSqliteContext +{ + public AppDbContext(string connectionString) : base(connectionString) { } + public DbSet Orders => Set(); +} + +// Use ExecuteWriteAsync — handles write lock, transaction, retry, commit +await using var db = new AppDbContext("Data Source=app.db"); +await db.ExecuteWriteAsync(async ctx => +{ + ctx.Orders.Add(new Order { /* ... */ }); + // SaveChangesAsync is called automatically at the end of the lambda +}); + +// Or bulk insert +await db.BulkInsertSafeAsync(orders); +``` + +--- + +## Pattern Decision Table + +| Scenario | Pattern | Registration | +|---|---|---| +| ASP.NET Core controllers / Razor Pages | Request-scoped | `AddConcurrentSqliteDbContext` | +| Blazor Server | Request-scoped | `AddConcurrentSqliteDbContext` | +| Background service / IHostedService | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| `Task.WhenAll` with N concurrent writes | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| `Parallel.ForEachAsync` | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| `Channel` consumer | Factory-per-task | `AddConcurrentSqliteDbContextFactory` | +| Console app / no DI | ThreadSafeSqliteContext | None (constructor injection) | +| Desktop app (WPF, MAUI) | ThreadSafeSqliteContext or factory | Either | + +--- + +## How the Write Queue Serializes Across All Patterns + +All three patterns share the same process-wide write queue (keyed by database file path). +When any of them calls a write API (`SaveChangesAsync`, `SaveChangesSerializedAsync`, +`BulkInsertOptimizedAsync`, `ExecuteWriteAsync`), the work is enqueued to the same +`Channel`. The single background writer executes one request at a time. + +This means a Pattern 1 controller write and a Pattern 2 background task write are +automatically serialized against each other — you do not need to coordinate between them. + +--- + +## Related Pages + +- [Troubleshooting SQLITE_BUSY errors](troubleshooting-sqlite-busy.md) +- [Performance and WAL tuning guide](performance-guide.md) +- [Migration guide](migration-guide.md) +```` + +- [ ] **Step 2: Commit** + +```bash +git add docs/concurrent-efcore-patterns.md +git commit -m "docs: add concurrent-efcore-patterns page" +``` + +--- + +### Task 7: Create docs/performance-guide.md + +**Files:** +- Create: `docs/performance-guide.md` + +- [ ] **Step 1: Write the performance guide** + +Full content: + +````markdown +# Performance and WAL Tuning Guide + +This guide explains how **EntityFrameworkCore.Sqlite.Concurrency** achieves high +throughput on SQLite and how to tune it for your workload. + +--- + +## WAL Mode — How Parallel Reads Work + +SQLite's default journal mode (`DELETE`) uses an exclusive write lock: while a write +is in progress, all reads must wait. WAL mode (Write-Ahead Logging) separates the write +log from the main database file, allowing readers and the writer to operate concurrently: + +- **Readers** read from the main database file at a consistent snapshot point +- **The writer** appends to the WAL file +- **Checkpoint** periodically merges WAL pages back into the main file + +This library enables WAL mode automatically on every new connection via `PRAGMA journal_mode=WAL`. +You do not need to configure it. + +--- + +## The Channel-Based Write Queue + +Prior to v10.1.0, write serialization used `SemaphoreSlim(1,1)` per database file. +Under high concurrent write load, `SemaphoreSlim.Release()` wakes all N waiting threads +simultaneously. N-1 immediately go back to sleep — this is the **thundering herd problem**: +CPU spikes, context-switch storms, unfair scheduling. + +Since v10.1.0, a `Channel` replaces the semaphore: + +``` +Producer 1 ──┐ +Producer 2 ──┼──► Channel ──► Background Writer ──► SQLite +Producer N ──┘ (FIFO queue) (single goroutine) +``` + +Each producer enqueues a `WriteRequest` containing its work lambda and a +`TaskCompletionSource`, then awaits the completion source. The background writer +pops one request at a time, executes the lambda, and signals the `TaskCompletionSource`. + +**Result:** No wakeup storm. Callers park until their turn. FIFO fairness. +Throughput under concurrent load is higher and more predictable. + +--- + +## BulkInsertOptimizedAsync + +For importing large numbers of records, `BulkInsertOptimizedAsync` outperforms repeated +individual `SaveChangesAsync` calls by: + +1. **Batching** — entities are processed in chunks of 1,000 (configurable via + `Chunk(1000)`) rather than one giant transaction +2. **ChangeTracker.Clear()** — the change tracker is cleared after each batch, preventing + memory from growing linearly with total entity count +3. **Write queue** — the entire operation is enqueued as a single work item, holding the + write slot for the duration and avoiding per-batch lock overhead +4. **WAL PRAGMAs** — `busy_timeout`, `wal_autocheckpoint`, and `synchronous` are already + tuned per connection; no additional setup needed + +```csharp +// Import 100,000 records — automatically batched in 1,000-record chunks +await context.BulkInsertOptimizedAsync(records); +``` + +For even higher throughput, set `SynchronousMode = Off` during a controlled bulk import +(then restore it afterward): + +```csharp +// Scratch/import database — durability not required +options.UseSqliteWithConcurrency("Data Source=import.db", o => +{ + o.SynchronousMode = SqliteSynchronousMode.Off; +}); +``` + +--- + +## PRAGMA Tuning Reference + +| PRAGMA | Option | Default | Guidance | +|---|---|---|---| +| `busy_timeout` | `BusyTimeout` | 30 s | Increase to 60 s+ if multiple OS processes write to the same database. The write queue handles in-process contention, but cross-process contention still goes through SQLite's native timeout. | +| `wal_autocheckpoint` | `WalAutoCheckpoint` | 1000 pages | Each page is 4,096 bytes by default (~4 MB per 1,000 frames). Increase to 5,000–10,000 for write-heavy workloads to reduce checkpoint frequency. Set to 0 to disable automatic checkpoints and manage manually. | +| `synchronous` | `SynchronousMode` | `Normal` | `Normal` is safe after an application crash (recommended for WAL). Use `Full` for power-loss safety. Use `Off` only for disposable bulk-import databases. | +| `journal_mode` | — | `WAL` (set automatically) | Do not change. WAL mode is required for parallel reads. | +| `cache_size` | — | SQLite default | Not exposed; modify via raw PRAGMA if needed. Rarely necessary. | + +--- + +## WriteQueueCapacity — Back-Pressure for High-Throughput Producers + +By default the write queue is unbounded — any number of producers can enqueue writes +without blocking. For workloads where producers outpace the writer significantly, +an unbounded queue can lead to memory growth. + +Set `WriteQueueCapacity` to apply back-pressure: + +```csharp +options.UseSqliteWithConcurrency("Data Source=app.db", o => +{ + o.WriteQueueCapacity = 500; + // Producers block asynchronously once 500 writes are pending +}); +``` + +When the queue is full, `Channel.Writer.WriteAsync` blocks the producer until the +background writer drains a slot. This is `BoundedChannelFullMode.Wait` — no writes +are dropped, no exceptions thrown — producers simply slow down to match the writer's pace. + +**When to set it:** When you have bursty producers (e.g., a `Parallel.ForEachAsync` +spinning up thousands of tasks simultaneously) and want to limit peak memory usage. + +**When to leave it null:** Most applications. The write queue processes requests as fast +as SQLite can commit them; backlog only builds if producers are genuinely faster than storage. + +--- + +## WAL Checkpoint Health Monitoring + +Long-running read transactions block WAL checkpoint completion. When a checkpoint cannot +complete, the WAL file grows unboundedly and read performance degrades (readers must scan +more WAL pages to reconstruct their snapshot). + +Monitor WAL health periodically in production: + +```csharp +var connection = db.Database.GetDbConnection(); +await connection.OpenAsync(); +var status = await SqliteConnectionEnhancer.GetWalCheckpointStatusAsync(connection); + +logger.LogInformation( + "WAL: {Total} frames, {Done} checkpointed ({Pct:F1}%), busy={Busy}", + status.TotalWalFrames, + status.CheckpointedFrames, + status.CheckpointProgress, + status.IsBusy); + +if (status.IsBusy && status.TotalWalFrames > 5000) + logger.LogWarning("WAL checkpoint blocked — a long-running reader may be " + + "preventing WAL reclamation."); +``` + +--- + +## Typical Results + +| Operation | Standard EF Core SQLite | With This Package | Gain | +|---|---|---|---| +| Bulk Insert (10,000 records) | ~4.2 s | ~0.8 s | ~5x faster | +| Bulk Insert (100,000 records) | ~42 s | ~4.1 s | ~10x faster | +| Concurrent Reads (50 threads) | ~8.7 s | ~2.1 s | ~4x faster | +| Mixed Read/Write Workload | ~15.3 s | ~3.8 s | ~4x faster | + +> Results observed in practice on .NET 10 / Windows 11 / Intel i7-13700K. Not a formal +> benchmark suite — actual gains depend on write volume, hardware, and WAL page size. + +--- + +## Related Pages + +- [Troubleshooting SQLITE_BUSY errors](troubleshooting-sqlite-busy.md) +- [Concurrent EF Core patterns](concurrent-efcore-patterns.md) +- [Migration guide](migration-guide.md) +```` + +- [ ] **Step 2: Commit** + +```bash +git add docs/performance-guide.md +git commit -m "docs: add performance and WAL tuning guide" +``` + +--- + +### Task 8: Create docs/migration-guide.md + +**Files:** +- Create: `docs/migration-guide.md` + +- [ ] **Step 1: Write the migration guide** + +Full content: + +````markdown +# Migration Guide — Upgrading from Plain UseSqlite + +Moving from `Microsoft.EntityFrameworkCore.Sqlite` to +`EntityFrameworkCore.Sqlite.Concurrency` is a drop-in change. This guide covers every +registration pattern and provides a checklist for a clean migration. + +--- + +## Step 1: Install the Package + +```bash +dotnet add package EntityFrameworkCore.Sqlite.Concurrency +``` + +Remove any package that was providing manual retry or locking logic — it is no longer needed. + +--- + +## Step 2: Replace Registration Calls + +### Direct UseSqlite → UseSqliteWithConcurrency + +```csharp +// Before +options.UseSqlite("Data Source=app.db"); + +// After +options.UseSqliteWithConcurrency("Data Source=app.db"); +``` + +### AddDbContext → AddConcurrentSqliteDbContext (request-scoped) + +```csharp +// Before +builder.Services.AddDbContext(o => + o.UseSqlite("Data Source=app.db")); + +// After +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); +``` + +### AddDbContextFactory → AddConcurrentSqliteDbContextFactory (concurrent workloads) + +```csharp +// Before +builder.Services.AddDbContextFactory(o => + o.UseSqlite("Data Source=app.db")); + +// After +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); +``` + +### With Custom Options + +```csharp +// After — with options +builder.Services.AddConcurrentSqliteDbContext( + "Data Source=app.db", + options => + { + options.BusyTimeout = TimeSpan.FromSeconds(30); + options.MaxRetryAttempts = 5; + options.SynchronousMode = SqliteSynchronousMode.Full; + }); +``` + +--- + +## Step 3: Migration Checklist + +- [ ] **Remove `Cache=Shared` from all connection strings.** + `Cache=Shared` is incompatible with WAL mode. The library throws `ArgumentException` + at startup if it is present. Connection pooling is enabled automatically — no replacement needed. + + ``` + // Before (remove this flag) + "Data Source=app.db;Cache=Shared" + + // After + "Data Source=app.db" + ``` + +- [ ] **Remove manual `SemaphoreSlim` or `lock` write guards.** + The write queue handles write serialization. Application-level locks around `SaveChangesAsync` + are redundant and may cause deadlocks when combined with the write queue. + +- [ ] **Remove custom retry loops around `SaveChangesAsync`.** + Built-in exponential backoff with full jitter handles `SQLITE_BUSY` and `SQLITE_BUSY_SNAPSHOT` + variants. Custom retry code should be removed to avoid double-retry and incorrect snapshot + restart behavior. + +- [ ] **Switch concurrent workloads from shared `DbContext` to `IDbContextFactory`.** + If any background service, `Task.WhenAll`, or `Parallel.ForEachAsync` block shares a + single `DbContext` across concurrent tasks, switch to `AddConcurrentSqliteDbContextFactory` + and inject `IDbContextFactory`. Call `CreateDbContext()` per task. + See [Concurrent EF Core patterns](concurrent-efcore-patterns.md). + +- [ ] **Remove custom `PRAGMA` setup from `OnConfiguring` or migration scripts.** + The library applies `journal_mode=WAL`, `busy_timeout`, `wal_autocheckpoint`, and + `synchronous` on every connection open. Custom PRAGMA calls may conflict. Use + `SqliteConcurrencyOptions` to configure these values instead. + +- [ ] **Remove `SaveChangesSerializedAsync` workarounds** (if you had a custom version). + `SaveChangesSerializedAsync` is a first-class extension method on `DbContext` — call it + directly anywhere you previously called `SaveChangesAsync`. + +--- + +## Step 4: Verify + +```bash +dotnet build +dotnet test +``` + +Run your normal integration tests. If any test hits `SQLITE_BUSY`, check the checklist +above — the most common cause is a shared `DbContext` across concurrent tasks. + +--- + +## Version History + +| Version | Highlights | +|---|---| +| 10.1.0 | Channel-based write queue, `WriteQueueCapacity`, netstandard 2.0 TFM, `BulkInsertSafeAsync` O(n) fix | +| 10.0.3 | `SQLITE_BUSY_SNAPSHOT` correct restart, `AddConcurrentSqliteDbContextFactory`, structured logging, WAL checkpoint monitoring, migration lock recovery | +| 10.0.2 | `SynchronousMode` option, `UpgradeTransactionsToImmediate` option, startup validation | +| 10.0.1 | `Cache=Shared` rejection, full jitter backoff, `GetWalCheckpointStatusAsync` | +| 10.0.0 | Initial release — WAL mode, `BEGIN IMMEDIATE` upgrade, `BulkInsertOptimizedAsync`, DI registration | + +## Breaking Changes + +None in any version. All releases have been fully backwards-compatible. + +--- + +## Related Pages + +- [Troubleshooting SQLITE_BUSY errors](troubleshooting-sqlite-busy.md) +- [Concurrent EF Core patterns](concurrent-efcore-patterns.md) +- [Performance and WAL tuning guide](performance-guide.md) +```` + +- [ ] **Step 2: Commit** + +```bash +git add docs/migration-guide.md +git commit -m "docs: add migration guide from plain UseSqlite" +``` + +--- + +### Task 9: Build Verification + Final Commit + +**Files:** None — verification only. + +- [ ] **Step 1: Build library and test project** + +``` +dotnet build EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj -c Release +dotnet test EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj --framework net10.0 -c Release +``` + +Expected: Build succeeded, 4 tests passed. + +- [ ] **Step 2: Commit plan document** + +```bash +git add docs/superpowers/plans/2026-07-05-v10-1-0-docs-seo.md +git commit -m "docs: add v10.1.0 docs/SEO implementation plan" +``` diff --git a/docs/superpowers/specs/2026-07-05-trust-signals-design.md b/docs/superpowers/specs/2026-07-05-trust-signals-design.md new file mode 100644 index 0000000..841fa17 --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-trust-signals-design.md @@ -0,0 +1,140 @@ +# Trust Signals — Design Spec + +**Date:** 2026-07-05 +**Track:** A — Package quality signals and credibility +**Scope:** CI test integration, BenchmarkDotNet project, CHANGELOG, community files, README badge/numbers update + +--- + +## Goal + +Close the gap between a technically excellent library and one that *looks* professionally maintained. A developer who lands on the GitHub repo or NuGet page should see: tests passing in CI, real reproducible benchmark numbers, a standard changelog, and community scaffolding that signals the project welcomes contributors and bug reports. + +--- + +## Section 1: CI Workflow Update + +**File:** `.github/workflows/ci.yml` + +Add after the existing Build step: + +```yaml +- name: Test + run: dotnet test EFCore.Sqlite.Concurrency.Test/EFCore.Sqlite.Concurrency.Test.csproj --framework net10.0 -c Release --no-build --logger "trx;LogFileName=results.xml" + +- name: Publish Test Results + uses: dorny/test-reporter@v1 + if: always() + with: + name: xUnit Tests + path: "**/*.trx" + reporter: dotnet-trx +``` + +The benchmark project is explicitly excluded from CI (too slow for CI; it is a developer tool). + +**README:** Add GitHub Actions badge for the CI workflow next to existing badges: +```markdown +[![CI](https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/actions/workflows/ci.yml/badge.svg)](https://github.com/CornerstoneCode/EntityFrameworkCore.Sqlite.Concurrency/actions/workflows/ci.yml) +``` + +--- + +## Section 2: BenchmarkDotNet Project + +**Directory:** `EFCore.Sqlite.Concurrency.Benchmarks/` (new console app) + +### Files + +| File | Purpose | +|---|---| +| `EFCore.Sqlite.Concurrency.Benchmarks.csproj` | net10.0 console app, BenchmarkDotNet + project ref | +| `Program.cs` | `BenchmarkRunner.Run` entry point with arg switching | +| `BenchmarkConfig.cs` | `ShortRunConfig` (1 warmup, 3 iterations) for quick developer runs | +| `BaselineSqliteContext.cs` | Plain `UseSqlite` DbContext for baseline comparison | +| `BenchmarkEntity.cs` | Simple entity used across all benchmarks | +| `BulkInsertBenchmarks.cs` | Sequential bulk insert: package vs baseline, Params(1000, 10000) | +| `ConcurrentWriteBenchmarks.cs` | 50 concurrent `SaveChangesSerializedAsync` writers | +| `ReadBenchmarks.cs` | 20 parallel reads under concurrent write load | + +### Benchmark Design + +**BulkInsertBenchmarks:** Compares `BulkInsertOptimizedAsync` against naively looping `SaveChanges()` one entity at a time (the common antipattern). Both use the same temp DB file. Params: 1,000 and 10,000 entities. + +**ConcurrentWriteBenchmarks:** 50 tasks each writing 10 rows via `SaveChangesSerializedAsync`. Measures total time for all 500 rows to commit. No baseline (plain SQLite throws SQLITE_BUSY under this load — the baseline IS the error). + +**ReadBenchmarks:** 20 parallel readers each running `ToListAsync()` while 5 background writers are active. Measures reader throughput in WAL mode. + +### Running + +```bash +cd EFCore.Sqlite.Concurrency.Benchmarks +dotnet run -c Release -- --filter * --job short +``` + +Full precision run (slower, used for README numbers): +```bash +dotnet run -c Release -- --filter * +``` + +--- + +## Section 3: CHANGELOG.md + +**File:** `CHANGELOG.md` at repo root. Keep A Changelog format (https://keepachangelog.com). + +Structure: +- `[Unreleased]` +- `[10.1.0] - 2026-07-05` +- `[10.0.3]` +- `[10.0.2]` +- `[10.0.1]` +- `[10.0.0]` + +Each version has subsections: Added, Fixed, Changed, Removed, Security. Content sourced from existing `doc/v*.md` files. + +--- + +## Section 4: Community Files + +| File | Content | +|---|---| +| `CONTRIBUTING.md` | Prerequisites, build commands, test commands, benchmark run instructions, PR checklist, commit message convention | +| `SECURITY.md` | How to report vulnerabilities (email, not public issues), response SLA | +| `.github/ISSUE_TEMPLATE/bug_report.yml` | Fields: version, .NET version, error message, connection string (redacted), reproduction steps, expected vs actual | +| `.github/ISSUE_TEMPLATE/feature_request.yml` | Fields: problem description, proposed solution, alternatives considered | +| `.github/PULL_REQUEST_TEMPLATE.md` | Checklist: tests added, CHANGELOG updated, XML docs updated, no breaking changes or documented | + +--- + +## Section 5: README + Docs Updates + +**README:** Add CI badge. After benchmarks run, replace "Typical Results" table with real BenchmarkDotNet output (mean times, ratio column). Add footnote with exact benchmark environment and command to reproduce. + +**docs/performance-guide.md:** Add "Running the Benchmarks" section linking to the benchmark project with the run command. + +--- + +## File Change Summary + +| Action | File | +|---|---| +| Modify | `.github/workflows/ci.yml` | +| Create | `EFCore.Sqlite.Concurrency.Benchmarks/` (8 files) | +| Create | `CHANGELOG.md` | +| Create | `CONTRIBUTING.md` | +| Create | `SECURITY.md` | +| Create | `.github/ISSUE_TEMPLATE/bug_report.yml` | +| Create | `.github/ISSUE_TEMPLATE/feature_request.yml` | +| Create | `.github/PULL_REQUEST_TEMPLATE.md` | +| Modify | `README.md` (badge + benchmark table) | +| Modify | `docs/performance-guide.md` (benchmark run instructions) | + +--- + +## Out of Scope + +- Running benchmarks in CI (too slow; developer tool only) +- Automated benchmark regression tracking +- Code coverage reporting +- API reference site generation diff --git a/docs/superpowers/specs/2026-07-05-v10-1-0-docs-seo-design.md b/docs/superpowers/specs/2026-07-05-v10-1-0-docs-seo-design.md new file mode 100644 index 0000000..a4b4b7f --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-v10-1-0-docs-seo-design.md @@ -0,0 +1,326 @@ +# v10.1.0 Documentation & SEO Refresh — Design Spec + +**Date:** 2026-07-05 +**Version target:** 10.1.0 (from 10.0.4) +**Approach:** Option A (Problem-First Rewrite) + Option C (Microsite-Style Docs) +**Scope:** csproj metadata, README.md, doc/QUICKSTART.md, docs/ microsite pages, doc/v10_1_0.md + +--- + +## Goal + +Make the package maximally discoverable when a developer searches for `SQLITE_BUSY C#`, +`database is locked EF Core`, `SQLITE_BUSY_SNAPSHOT`, or `EF Core concurrent writes` — on +Google, NuGet search, and AI assistants (Claude, Copilot). Every surface (NuGet metadata, +GitHub README, docs pages) is rewritten to lead with the exact error strings developers paste +into search engines. + +Nothing is additive-only. Files may be completely replaced, deleted, or created from scratch. + +--- + +## Section 1: csproj Metadata + +**File:** `EntityFrameworkCore.Sqlite.Concurrency/EFCore.Sqlite.Concurrency.csproj` + +### Version +`10.0.4` → `10.1.0` + +### Description +Replace current description with: + +> Fixes `SQLITE_BUSY` / `"database is locked"` / `SQLITE_BUSY_SNAPSHOT` errors in Entity +> Framework Core + SQLite apps. Provides Channel-based write serialization, automatic +> `BEGIN IMMEDIATE` transaction upgrades, exponential-backoff retry, WAL-mode optimization, +> and 10x faster bulk inserts. Drop-in replacement for `UseSqlite()` targeting .NET 10 and +> netstandard 2.0. + +### Tags +Add to existing tag list (space-delimited): +``` +sqlite-busy-snapshot error-5 write-serialization channel-queue netstandard2 +efcore-concurrency write-queue background-service task-whenall +``` + +### PackageReleaseNotes +Replace v10.0.3 notes with v10.1.0 (see Section 5 for full content). + +--- + +## Section 2: README.md + +**File:** `README.md` — complete replacement. + +### Structure (top to bottom) + +#### 1. Hook — the exact error +``` +# EntityFrameworkCore.Sqlite.Concurrency + +Getting this error? + + Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' + +This is the fix. +``` +Badges follow immediately after the one-line fix. + +#### 2. One-line fix +```csharp +// Before: +options.UseSqlite("Data Source=app.db"); + +// After — eliminates SQLITE_BUSY, adds write serialization and WAL optimization: +options.UseSqliteWithConcurrency("Data Source=app.db"); +``` + +#### 3. The four problems this solves +Each problem is introduced with the exact exception message the developer would see in their +stack trace: + +| Exception / Error | Root Cause | What This Package Does | +|---|---|---| +| `SQLite Error 5: 'database is locked'` (`SQLITE_BUSY`) | Multiple writers contending simultaneously | Channel-based write queue — one writer at a time, all others park efficiently | +| `SQLite Error 5: 'database is locked'` (extended code 517, `SQLITE_BUSY_SNAPSHOT`) | WAL read snapshot became stale mid-transaction | `BEGIN IMMEDIATE` upgrade + full operation restart on snapshot staleness | +| `InvalidOperationException: A second operation was started on this context` | `DbContext` shared across concurrent tasks | `IDbContextFactory` registration pattern, one context per concurrent flow | +| Bulk inserts slow / OOM on large datasets | Linear `SaveChanges()`, unbounded ChangeTracker growth | `BulkInsertOptimizedAsync` — batched, WAL-optimized, ChangeTracker cleared per batch | + +#### 4. Why choose this package +Existing comparison table, moved here (position 4, after problem is established). + +#### 5. Installation +```bash +dotnet add package EntityFrameworkCore.Sqlite.Concurrency +``` + +#### 6. Three scenario sections + +**Request-scoped (ASP.NET Core controllers, Razor Pages, Blazor Server)** +```csharp +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); +``` + +**Concurrent workloads (background services, Task.WhenAll, Channel)** +```csharp +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); +// Inject IDbContextFactory, call CreateDbContext() per concurrent operation. +``` + +**High-volume batch jobs** +```csharp +await context.BulkInsertOptimizedAsync(records); +``` +Each section includes a complete, realistic code example. + +#### 7. Configuration table +All existing rows kept. New row added: + +| `WriteQueueCapacity` | `null` (unbounded) | Maximum number of pending writes queued before callers block. `null` = accept writes without limit. Set a value to apply back-pressure on producers. | + +#### 8. FAQ +Existing questions kept. Four new entries added: + +- **What is the Channel-based write queue?** Replaces the `SemaphoreSlim` in v10.0.x. Under high concurrency, SemaphoreSlim wakes all waiters simultaneously (thundering herd). A `Channel` parks callers in FIFO order and drains them one at a time — no wakeup storm, lower CPU, fairer scheduling. +- **What does SQLITE_BUSY_SNAPSHOT (error 517) mean?** The connection's WAL read snapshot became stale after another writer committed. Naive retry of the same statement produces the same error. The correct fix (which this package implements) is to roll back and restart the entire operation so data is re-read against the current snapshot. +- **Does this work with netstandard 2.0?** The write queue and connection layer (`SqliteConnectionEnhancer`, `SqliteWriteQueue`, `SqliteConcurrencyOptions`) target netstandard 2.0. EF Core-dependent APIs (`UseSqliteWithConcurrency`, `ThreadSafeSqliteContext`) require net10.0 because EF Core 10 does. +- **What's the actual per-write overhead?** Sub-millisecond. Enqueueing to a Channel and awaiting a `TaskCompletionSource` adds microseconds per operation — negligible against real SQLite I/O. + +#### 9. Benchmark table +Kept but relabeled "Typical Results" with footnote: +> Results observed in practice on .NET 10 / Windows 11. Not a formal benchmark suite. +> Actual gains depend on write volume, hardware, and WAL page size. + +#### 10. Links to docs/ pages +``` +→ Troubleshooting SQLITE_BUSY errors +→ Concurrent EF Core patterns +→ Performance and WAL tuning guide +→ Migration guide (from plain UseSqlite) +``` + +#### 11. System Requirements + License +Unchanged, moved to bottom. + +--- + +## Section 3: doc/QUICKSTART.md + +**File:** `doc/QUICKSTART.md` — complete replacement. + +### Goal +A developer who lands here cold gets to working code in under 60 seconds. Nothing else. + +### Structure + +#### Opening (3 sentences max) +State the problem, state the package name, state the install command. + +#### The one-line change +Before/after `UseSqlite` → `UseSqliteWithConcurrency`. + +#### Two registration patterns (no more, no less) + +**Pattern 1 — Request-scoped** +When: ASP.NET Core controllers, Razor Pages, Blazor Server (one context per HTTP request). +```csharp +builder.Services.AddConcurrentSqliteDbContext("Data Source=app.db"); +``` + +**Pattern 2 — Concurrent workloads** +When: background services, `Task.WhenAll`, `Parallel.ForEachAsync`, `Channel`. +```csharp +builder.Services.AddConcurrentSqliteDbContextFactory("Data Source=app.db"); +// Inject IDbContextFactory. Call CreateDbContext() per concurrent operation. +``` + +#### Next steps +Four links: README, troubleshooting page, concurrent patterns page, migration guide. + +### What is removed +- All references to `ThreadSafeEFCore.SQLite` (old package name) — replaced with + `EntityFrameworkCore.Sqlite.Concurrency` everywhere +- Wrong/right DbContext-sharing comparison (belongs in concurrent patterns page) +- Reading-data example (reads just work, no config needed) +- Factory-without-DI example (belongs in README) +- Error handling section (belongs in troubleshooting page) + +--- + +## Section 4: docs/ Microsite Pages + +**Directory:** `docs/` (new, at repo root) + +Four files. Each page is self-contained — no assumed prior reading. + +### docs/troubleshooting-sqlite-busy.md + +**Primary search terms:** `SQLITE_BUSY C#`, `SQLite Error 5 database is locked`, `SQLITE_BUSY_SNAPSHOT EF Core`, `SQLite error 517` + +**Content outline:** +1. The three error codes and what each means (SQLITE_BUSY=5, SQLITE_BUSY_SNAPSHOT=517, SQLITE_LOCKED=6) +2. Why naive retry doesn't fix SQLITE_BUSY_SNAPSHOT (stale snapshot semantics) +3. Why `busy_timeout` alone isn't enough (process-level contention vs. SQLite-level) +4. What this package does for each error code — specific, concrete +5. Diagnostic checklist (connection string, Cache=Shared, WAL status) +6. WAL checkpoint health check code snippet + +### docs/concurrent-efcore-patterns.md + +**Primary search terms:** `EF Core DbContext thread safe`, `Task.WhenAll SaveChanges EF Core`, `IDbContextFactory concurrent`, `DbContext concurrent operations` + +**Content outline:** +1. Why DbContext is not thread-safe (EF Core change tracker, not a SQLite limitation) +2. The three correct patterns: + - Request-scoped: `AddConcurrentSqliteDbContext` → inject `T` directly + - Factory-per-task: `AddConcurrentSqliteDbContextFactory` → inject `IDbContextFactory`, call `CreateDbContext()` per task + - `ThreadSafeSqliteContext`: base class with `ExecuteWriteAsync` built in +3. Wrong-vs-right code for each pattern +4. When to use which (decision table) +5. How the Channel-based write queue serializes writes across all patterns + +### docs/performance-guide.md + +**Primary search terms:** `SQLite bulk insert C# EF Core`, `EF Core SQLite performance`, `WAL mode Entity Framework`, `SQLite PRAGMA tuning C#` + +**Content outline:** +1. How WAL mode enables parallel reads (conceptual, one paragraph) +2. The Channel-based write queue — why it beats SemaphoreSlim under load +3. `BulkInsertOptimizedAsync` — what it does (batching, PRAGMA, ChangeTracker.Clear) +4. PRAGMA reference table (busy_timeout, wal_autocheckpoint, synchronous) with recommended values +5. `WriteQueueCapacity` — when to set it (high-throughput producers, back-pressure scenarios) +6. WAL checkpoint monitoring (`GetWalCheckpointStatusAsync`) +7. Typical result ranges (mirrors README benchmark table with same "observed in practice" disclaimer) + +### docs/migration-guide.md + +**Primary search terms:** `migrate from UseSqlite`, `upgrade EF Core SQLite concurrency` + +**Content outline:** +1. Before/after for every registration pattern (exact code, no prose) +2. Migration checklist: + - [ ] Remove `Cache=Shared` from connection strings + - [ ] Remove manual `SemaphoreSlim` or lock-based retry code + - [ ] Switch concurrent workloads from shared `DbContext` to `IDbContextFactory` + - [ ] Remove custom `PRAGMA` setup (the library sets WAL, busy_timeout, autocheckpoint) +3. Version history table (what changed in each 10.x release) +4. Breaking changes per release (none so far — note explicitly) + +--- + +## Section 5: doc/v10_1_0.md + +**File:** `doc/v10_1_0.md` — new file, same structure as `doc/v10_0_3.md`. + +### Headline +"Channel-Based Write Queue, netstandard 2.0 Support, and Bug Fixes" + +### New Features + +**Channel-based write queue (headline)** +- Replaces `SemaphoreSlim` per-database with `Channel` + single background writer Task +- Why it matters: SemaphoreSlim wakes all N waiters simultaneously when released; N-1 immediately + re-sleep (thundering herd, CPU churn, unfair scheduling). Channel parks callers in FIFO order + and drains them one at a time — no wakeup storm +- Zero API change — existing call sites get the benefit automatically +- Reentrancy detection preserved via `AsyncLocal IsWriteLockHeld` + +**`WriteQueueCapacity` option** +- New `int?` property on `SqliteConcurrencyOptions` (default `null` = unbounded) +- When set, creates a `BoundedChannel` with `BoundedChannelFullMode.Wait` — callers block + asynchronously when queue is full rather than exceeding capacity +- Use for high-throughput producer scenarios where back-pressure is desired + +**netstandard 2.0 TFM** +- Package now multi-targets `net10.0;netstandard2.0` +- Available on netstandard 2.0: `SqliteConnectionEnhancer`, `SqliteWriteQueue`, `SqliteConcurrencyOptions` +- net10.0-only (EF Core 10 required): `UseSqliteWithConcurrency`, `SqliteConcurrencyInterceptor`, + `ThreadSafeSqliteContext`, `AddConcurrentSqliteDbContext`, `AddConcurrentSqliteDbContextFactory` +- Polyfills added for netstandard 2.0: `IsExternalInit`, `HashCode.Combine`, `Channel.Reader.ReadAllAsync`, + `Task.WaitAsync` + +### Bug Fixes + +**`ThreadSafeSqliteContext` always used default options** +- `ExecuteWriteAsync` was constructing `new SqliteConcurrencyOptions()` ignoring what the caller + configured via `UseSqliteWithConcurrency` +- Fixed: options are now read from the registered interceptor via `TryGetInterceptor` + +**`BulkInsertSafeAsync` O(n²) and unbounded ChangeTracker growth** +- `Skip(n).Take(1000)` loop was O(n²) in LINQ-to-objects for large entity lists +- `ChangeTracker.Clear()` was missing between batches, causing memory to grow linearly with + total entity count +- Fixed: `Chunk(1000)` replaces `Skip/Take`; `ChangeTracker.Clear()` added after each batch `SaveChangesAsync` + +### No Breaking Changes +All existing `UseSqliteWithConcurrency`, `AddConcurrentSqliteDbContext`, +`AddConcurrentSqliteDbContextFactory`, `ExecuteWithRetryAsync`, `BulkInsertOptimizedAsync`, +and `SaveChangesSerializedAsync` call sites compile and behave correctly without modification. +`WriteQueueCapacity` is additive. netstandard 2.0 TFM is additive. + +### Configuration Reference +Full table including new `WriteQueueCapacity` row. + +--- + +## File Change Summary + +| File | Action | +|---|---| +| `EFCore.Sqlite.Concurrency.csproj` | Modify — version, description, tags, release notes | +| `README.md` | Replace completely | +| `doc/QUICKSTART.md` | Replace completely | +| `doc/v10_1_0.md` | Create new | +| `docs/troubleshooting-sqlite-busy.md` | Create new | +| `docs/concurrent-efcore-patterns.md` | Create new | +| `docs/performance-guide.md` | Create new | +| `docs/migration-guide.md` | Create new | + +--- + +## Out of Scope + +- API reference documentation (XML doc comments / IntelliSense already covers this) +- Contributing guide +- Changelog consolidation (individual `doc/v*.md` files are the changelog) +- Benchmark harness (numbers remain "observed in practice" until a formal bench project exists) +- CI/CD changes diff --git a/docs/troubleshooting-sqlite-busy.md b/docs/troubleshooting-sqlite-busy.md new file mode 100644 index 0000000..e7d4dc1 --- /dev/null +++ b/docs/troubleshooting-sqlite-busy.md @@ -0,0 +1,134 @@ +# Troubleshooting SQLITE_BUSY Errors in C# / EF Core + +If you're seeing any of these errors in your .NET application, this page explains what they +mean, why they happen, and how **EntityFrameworkCore.Sqlite.Concurrency** fixes them. + +--- + +## The Three Error Codes + +### SQLITE_BUSY — Error Code 5 + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' +``` + +**What it means:** Another connection holds a write lock on the database and your connection +could not acquire one within the `busy_timeout` window. + +**Why it happens:** SQLite allows only one writer at a time. In a multi-threaded .NET app, +when two threads call `SaveChangesAsync()` simultaneously, one succeeds and the other +gets `SQLITE_BUSY`. + +**Why `busy_timeout` alone isn't enough:** `PRAGMA busy_timeout` is a per-connection, +SQLite-level retry. It only helps when the conflicting connection is in the same process +and releases the lock within the timeout window. Under sustained concurrent write load, +multiple threads will still contend and produce errors after the timeout expires. + +**How this package fixes it:** A process-wide `Channel` queues all writes +and drains them one at a time through a single background writer. Callers park asynchronously +and are notified when their write completes — no polling, no wakeup storms, no lock contention. + +--- + +### SQLITE_BUSY_SNAPSHOT — Extended Code 517 + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 5: 'database is locked' +SqliteException.SqliteExtendedErrorCode == 517 +``` + +**What it means:** This is `SQLITE_BUSY | (2 << 8) = 517`. It fires when a connection's +WAL read snapshot became stale after another writer committed mid-transaction. The connection +is trying to upgrade from a read transaction to a write transaction, but its snapshot of +the database is now behind the current WAL head. + +**Why naive retry doesn't work:** Retrying the same statement on the same connection +produces the same error — the snapshot is still stale. The only correct fix is to: +1. Roll back the entire transaction +2. Restart the entire operation from scratch so all reads are re-issued against the current snapshot + +**How this package fixes it:** `BEGIN IMMEDIATE` is used for all write transactions +(via the `UpgradeTransactionsToImmediate` option, default `true`). This acquires the +write lock at transaction start rather than at first write — preventing the snapshot from +ever becoming stale mid-transaction. If `SQLITE_BUSY_SNAPSHOT` (517) is still received, +`ExecuteWithRetryAsync` and `ExecuteWriteAsync` detect the extended error code and restart +the full operation lambda. + +--- + +### SQLITE_LOCKED — Error Code 6 + +``` +Microsoft.Data.Sqlite.SqliteException: SQLite Error 6: 'database table is locked' +``` + +**What it means:** A different table or connection within the **same process** holds a +lock that conflicts with your operation. Unlike `SQLITE_BUSY`, this is not a concurrency +issue between processes — it indicates an application-level bug. + +**Common causes:** +- A `DataReader` is open on the same connection while a write is attempted +- A transaction was started but not committed or rolled back +- Nested writes without the reentrancy guard + +**How this package handles it:** `SQLITE_LOCKED` is logged at `Error` level (not retried) +because it indicates a programming error that retry cannot fix. The exception propagates +to the caller for diagnosis. + +--- + +## Diagnostic Checklist + +If you're still seeing `SQLITE_BUSY` after installing this package, work through this list: + +- [ ] **Connection string has `Cache=Shared`?** + Remove it. `Cache=Shared` is incompatible with WAL mode and will throw `ArgumentException` + at startup if present. Connection pooling (`Pooling=true`) is enabled automatically. + +- [ ] **Using a shared `DbContext` across concurrent tasks?** + `DbContext` is not thread-safe. Use `AddConcurrentSqliteDbContextFactory` and inject + `IDbContextFactory`. Call `CreateDbContext()` per concurrent operation. + See [Concurrent EF Core patterns](concurrent-efcore-patterns.md). + +- [ ] **Multiple processes writing to the same database?** + The write queue is process-scoped. Multiple OS processes contend at the SQLite level. + Increase `BusyTimeout` and `MaxRetryAttempts`, or consider a client/server database. + +- [ ] **Database on a network filesystem?** + WAL mode does not work on NFS, SMB, or other network-mounted paths. Use local disk only. + +- [ ] **WAL file growing unboundedly?** + A long-running read transaction can block checkpoint completion, causing the WAL to grow + and degrade performance. Check WAL health: + +```csharp +var connection = db.Database.GetDbConnection(); +await connection.OpenAsync(); +var status = await SqliteConnectionEnhancer.GetWalCheckpointStatusAsync(connection); + +if (status.IsBusy) + Console.WriteLine($"WAL blocked: {status.TotalWalFrames} frames, " + + $"{status.CheckpointedFrames} checkpointed " + + $"({status.CheckpointProgress:F1}%)"); +``` + +--- + +## Error Code Quick Reference + +| Code | Extended Code | Name | Meaning | Retry? | +|---|---|---|---|---| +| 5 | 5 | `SQLITE_BUSY` | Another writer holds the lock | Yes — after backoff | +| 5 | 261 | `SQLITE_BUSY_RECOVERY` | WAL recovery in progress | Yes — after backoff | +| 5 | 517 | `SQLITE_BUSY_SNAPSHOT` | Read snapshot is stale | Yes — **restart entire operation** | +| 6 | 6 | `SQLITE_LOCKED` | Same-connection conflict | No — application bug | + +--- + +## Related Pages + +- [Concurrent EF Core patterns](concurrent-efcore-patterns.md) +- [Performance and WAL tuning guide](performance-guide.md) +- [Migration guide](migration-guide.md) +- [Full README](../README.md)