feat(json): add a converter that stores a TimeSpan as a number of ticks

CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+* `TimeSpanTicksConverter`, a `System.Text.Json` converter that writes a `TimeSpan` as a JSON number of ticks. Cosmos DB sorts and compares a number in time order, but not the text that `System.Text.Json` writes (`"2.00:00:00"` sorts after `"10.00:00:00"`). The converter also reads that text, so old documents stay readable. See [How To store a duration](https://fsprojects.github.io/FSharp.Azure.Cosmos/How_Tos/Store_Durations.html)
+
 ### Changed
 * BREAKING: the package targets `net10.0` only and no longer supports `net8.0`, which makes this a major release
 * BREAKING: `Microsoft.Azure.Cosmos` updated from 3.60.0 to 3.62.0, which becomes the minimum version for consumers. Its release notes list a breaking change of their own: thin client mode is enabled by default and does not support resource-token authentication (opt out with `AZURE_COSMOS_THIN_CLIENT_ENABLED=false`)

docsSrc/How_Tos/Store_Durations.md

@@ -0,0 +1,87 @@
+---
+title: How To store a duration
+category: How To Guides
+categoryindex: 2
+index: 4
+---
+
+# How To store a duration so that a query sorts it
+
+Cosmos DB has no type for a duration. System.Text.Json writes a `TimeSpan` as text, for example `"2.00:00:00"`.
+Cosmos DB compares text by character code. Thus `"2.00:00:00"` sorts after `"10.00:00:00"`, and `"23:00:00"` sorts
+after both. A query with `ORDER BY` or with `>` on such a field gives the wrong result.
+
+`TimeSpanTicksConverter` writes the number of ticks instead. One tick is 100 nanoseconds. Cosmos DB sorts and
+compares a number in time order.
+
+## Use the converter
+
+Add the converter to the options of System.Text.Json. Then give the options to the client. The camel case naming
+policy writes the field `Id` as `id`, which Cosmos DB requires:
+
+``` F#
+open System.Text.Json
+open Microsoft.Azure.Cosmos
+open FSharp.Azure.Cosmos
+
+let serializerOptions = JsonSerializerOptions (PropertyNamingPolicy = JsonNamingPolicy.CamelCase)
+serializerOptions.Converters.Add (TimeSpanTicksConverter ())
+
+let client =
+    new CosmosClient (
+        endpoint,
+        key,
+        CosmosClientOptions (UseSystemTextJsonSerializerWithOptions = serializerOptions)
+    )
+```
+
+Each `TimeSpan` that the client writes is now a number. This record is stored as the JSON below it:
+
+``` F#
+type Job = { Id : string; PartitionKey : string; Duration : System.TimeSpan }
+
+let job = { Id = "job-1"; PartitionKey = "jobs"; Duration = System.TimeSpan.FromDays 2.0 }
+```
+
+``` JSON
+{ "id": "job-1", "partitionKey": "jobs", "duration": 1728000000000 }
+```
+
+The client writes a parameter with the same converter. Thus a query can compare the field with a `TimeSpan`:
+
+``` F#
+let longJobsAsync (container : Container) (minimum : System.TimeSpan) = task {
+    let query =
+        QueryDefinition("SELECT VALUE c FROM c WHERE c.duration >= @minimum ORDER BY c.duration")
+            .WithParameter ("@minimum", minimum)
+
+    use iterator = container.GetItemQueryIterator<Job> query
+    let jobs = ResizeArray<Job> ()
+
+    while iterator.HasMoreResults do
+        let! page = iterator.ReadNextAsync ()
+        jobs.AddRange page
+
+    return jobs.ToArray ()
+}
+```
+
+The converter also handles `Nullable<TimeSpan>` and a `TimeSpan` as the key of a dictionary.
+
+## Read the documents that you wrote before
+
+The converter reads the text that System.Text.Json writes, for example `"1.02:00:00"`. Thus a container with old
+documents stays readable. An old document keeps its text until you write it again. Cosmos DB sorts a number before a
+text. Thus a query with `ORDER BY` returns the new documents in time order, then the old documents in the order of
+their text. With `DESC` the two groups change places.
+
+To move the old documents, read each one and write it again with a client that uses the converter.
+
+## Know the limits
+
+- The converter works with System.Text.Json only. Set it in `UseSystemTextJsonSerializerWithOptions`.
+- Cosmos DB defines a JSON number as a 64-bit floating-point number. Such a number holds each whole number up to 2^53
+  exactly, which is about 28 years of ticks. A longer `TimeSpan` can lose digits in a client that reads the number as
+  a floating-point number.
+- The number has no unit in the document. A query written by hand must use ticks. Write
+  `c.duration >= 36000000000` for one hour, or use a `TimeSpan` parameter as above.

src/Cosmos/FSharp.Azure.Cosmos.fsproj

@@ -26,6 +26,7 @@
     <Compile Include="TaskSeq.fs" />
     <Compile Include="CosmosResponse.fs" />
     <Compile Include="SubStatusCodes.fs" />
+    <Compile Include="JsonConverters.fs" />
     <Compile Include="Cosmos.fs" />
     <Compile Include="Create.fs" />
     <Compile Include="Upsert.fs" />

src/Cosmos/JsonConverters.fs

@@ -0,0 +1,100 @@
+namespace FSharp.Azure.Cosmos
+
+open System
+open System.Globalization
+open System.Text.Json
+open System.Text.Json.Serialization
+
+/// <summary>
+/// Writes a <see cref="TimeSpan"/> as a JSON number, its number of ticks, and reads the same number.
+/// </summary>
+/// <remarks>
+/// <para>
+/// The converter of System.Text.Json writes a <see cref="TimeSpan"/> as text, for example <c>"2.00:00:00"</c>.
+/// Cosmos DB compares text by character code. Thus <c>"2.00:00:00"</c> sorts after <c>"10.00:00:00"</c>.
+/// A number sorts and compares in time order. Use this converter for each <see cref="TimeSpan"/> that a query
+/// compares or sorts.
+/// </para>
+/// <para>
+/// One tick is 100 nanoseconds. Cosmos DB defines a JSON number as a 64-bit floating-point number. Such a number holds
+/// each whole number up to 2^53 exactly, which is about 28 years of ticks. A longer <see cref="TimeSpan"/> can lose
+/// digits in a client that reads the number as a floating-point number.
+/// </para>
+/// <para>
+/// The converter also reads the text that System.Text.Json writes, so a container that holds old documents stays
+/// readable. An old document keeps its text, and the text does not sort in time order, until you write the document
+/// again.
+/// </para>
+/// <para>
+/// To use the converter, add it to <see cref="JsonSerializerOptions.Converters"/>. Then set the options in
+/// <see cref="Microsoft.Azure.Cosmos.CosmosClientOptions.UseSystemTextJsonSerializerWithOptions"/>. The converter
+/// also handles <see cref="Nullable{T}"/> values and the keys of a dictionary.
+/// </para>
+/// </remarks>
+[<Sealed>]
+type TimeSpanTicksConverter () =
+    inherit JsonConverter<TimeSpan> ()
+
+    // A text that holds a whole number is a number of ticks. This is the form of a dictionary key.
+    // Any other text must have the form of System.Text.Json: [-][d.]hh:mm:ss[.fffffff]. It always has a colon.
+    // The parser of the "c" format reads a text without a colon as a number of days, so the check keeps such a text out.
+    static let tryParseText (text : string) : TimeSpan voption =
+        match Int64.TryParse (text, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture) with
+        | true, ticks -> ValueSome (TimeSpan.FromTicks ticks)
+        | false, _ when text.Contains (':', StringComparison.Ordinal) ->
+            match TimeSpan.TryParseExact (text, "c", CultureInfo.InvariantCulture) with
+            | true, value -> ValueSome value
+            | false, _ -> ValueNone
+        | false, _ -> ValueNone
+
+    static let parseText (text : string) : TimeSpan =
+        match tryParseText text with
+        | ValueSome value -> value
+        | ValueNone ->
+            raise (
+                JsonException
+                    $"The text '{text}' is not a number of ticks and not a TimeSpan in the form [-][d.]hh:mm:ss[.fffffff]."
+            )
+
+    /// <summary>
+    /// Reads a <see cref="TimeSpan"/> from a whole number of ticks. Reads the text form of System.Text.Json as well.
+    /// </summary>
+    /// <exception cref="JsonException">
+    /// The token is not a number or a text, the number is not a whole number that fits in 64 bits, or the text has an
+    /// unknown form.
+    /// </exception>
+    override _.Read (reader : byref<Utf8JsonReader>, typeToConvert : Type, options : JsonSerializerOptions) : TimeSpan =
+        match reader.TokenType with
+        | JsonTokenType.Number ->
+            match reader.TryGetInt64 () with
+            | true, ticks -> TimeSpan.FromTicks ticks
+            | false, _ -> raise (JsonException "A TimeSpan must be a whole number of ticks that fits in 64 bits.")
+        | JsonTokenType.String ->
+            match reader.GetString () with
+            | null -> raise (JsonException "A TimeSpan must not be null.")
+            | text -> parseText text
+        | token -> raise (JsonException $"A TimeSpan must be a number or a text, but the JSON token is {token}.")
+
+    /// <summary>
+    /// Writes the number of ticks of <paramref name="value"/> as a JSON number.
+    /// </summary>
+    override _.Write (writer : Utf8JsonWriter, value : TimeSpan, options : JsonSerializerOptions) : unit =
+        writer.WriteNumberValue value.Ticks
+
+    /// <summary>
+    /// Reads the key of a dictionary: the number of ticks as text. Reads the text form of System.Text.Json as well.
+    /// </summary>
+    /// <exception cref="JsonException">The key has an unknown form.</exception>
+    override _.ReadAsPropertyName
+        (reader : byref<Utf8JsonReader>, typeToConvert : Type, options : JsonSerializerOptions)
+        : TimeSpan
+        =
+        match reader.GetString () with
+        | null -> raise (JsonException "A TimeSpan key must not be null.")
+        | text -> parseText text
+
+    /// <summary>
+    /// Writes the key of a dictionary: the number of ticks of <paramref name="value"/> as text.
+    /// </summary>
+    override _.WriteAsPropertyName (writer : Utf8JsonWriter, value : TimeSpan, options : JsonSerializerOptions) : unit =
+        writer.WritePropertyName (value.Ticks.ToString CultureInfo.InvariantCulture)

tests/Cosmos.Tests.Infrastructure/Emulator.fs

@@ -10,6 +10,7 @@ open System
 open System.Globalization
 open System.Net.Http
 open System.Net.Security
+open System.Text.Json
 open System.Threading
 open System.Threading.Tasks
 
@@ -172,20 +173,28 @@ let private createHttpMessageHandler () =
             )
     )
 
+// The shared settings are in a static member, not in a function, because F# sets the named properties of the object
+// that a method returns. Thus each client adds its own settings in a class initializer.
+type private ClientOptions =
+    static member Create () =
+        CosmosClientOptions (
+            ConnectionMode = ConnectionMode.Gateway,
+            HttpClientFactory = Func<HttpClient>(fun () -> new HttpClient (createHttpMessageHandler (), true))
+        )
+
 /// <summary>
 /// Creates a <see cref="CosmosClient"/> for the emulator in <see cref="ConnectionMode.Gateway"/> mode, the only mode
 /// the Linux vNext emulator serves, whose <see cref="HttpClient"/> accepts the emulator's self-signed certificate from
 /// local hosts only.
 /// </summary>
-let createClient () : CosmosClient =
-    new CosmosClient (
-        endpoint,
-        key,
-        CosmosClientOptions (
-            ConnectionMode = ConnectionMode.Gateway,
-            HttpClientFactory = Func<HttpClient>(fun () -> new HttpClient (createHttpMessageHandler (), true))
-        )
-    )
+let createClient () : CosmosClient = new CosmosClient (endpoint, key, ClientOptions.Create ())
+
+/// <summary>
+/// Creates a <see cref="CosmosClient"/> like <see cref="createClient"/>, whose serializer is System.Text.Json with
+/// <paramref name="serializerOptions"/>. The client of a test that checks how a converter stores a type.
+/// </summary>
+let createClientWithJson (serializerOptions : JsonSerializerOptions) : CosmosClient =
+    new CosmosClient (endpoint, key, ClientOptions.Create (UseSystemTextJsonSerializerWithOptions = serializerOptions))
 
 // How long the sweep waits for an answer before it concludes that no emulator runs. Without this check, a run of
 // the unit tests alone on a machine without an emulator would wait for the SDK to give up (seconds per request).

tests/Cosmos.Tests/FSharp.Azure.Cosmos.Tests.fsproj

@@ -24,6 +24,7 @@
     <Compile Include="ValueCollectionsTests.fs" />
     <Compile Include="CosmosNameTests.fs" />
     <Compile Include="ResponseMessageTests.fs" />
+    <Compile Include="TimeSpanTicksConverterTests.fs" />
     <Compile Include="DatabaseIdentifierTests.fs" />
     <Compile Include="ContainerSetTests.fs" />
     <Compile Include="PartitionBudgetTests.fs" />
@@ -32,6 +33,7 @@
     <Compile Include="OperationTestInfrastructure.fs" />
     <Compile Include="Scenarios\SingleItemScenario.fs" />
     <Compile Include="Scenarios\MultipleItemsScenario.fs" />
+    <Compile Include="Scenarios\DurationsScenario.fs" />
     <Compile Include="CreateOperationTests.fs" />
     <Compile Include="ReadOperationTests.fs" />
     <Compile Include="UpsertOperationTests.fs" />
@@ -41,6 +43,7 @@
     <Compile Include="ReadManyOperationTests.fs" />
     <Compile Include="ReadExtensionsTests.fs" />
     <Compile Include="IterationExtensionsTests.fs" />
+    <Compile Include="TimeSpanTicksConverterIntegrationTests.fs" />
     <Compile Include="BuilderUnitTests.fs" />
   </ItemGroup>
 

tests/Cosmos.Tests/Scenarios/DurationsScenario.fs

@@ -0,0 +1,112 @@
+namespace FSharp.Azure.Cosmos.Tests.Integration.DurationsScenario
+
+open System
+open System.Collections.Immutable
+open System.Text.Json
+open System.Threading
+open System.Threading.Tasks
+
+open Microsoft.VisualStudio.TestTools.UnitTesting
+
+open FSharp.Azure.Cosmos
+open FSharp.Azure.Cosmos.Tests
+open FSharp.Azure.Cosmos.Tests.Integration
+
+// The record is public because System.Text.Json does not write the properties of an internal record. Only the files
+// that open this namespace see its labels. Thus, the record type inference of the other test files does not use them.
+/// A document that holds one duration.
+[<CLIMutable>]
+type DurationItem = { Id : string; PartitionKey : string; Duration : TimeSpan }
+
+/// The serializer options of the clients that write and read the durations.
+[<RequireQualifiedAccess>]
+module SerializerOptions =
+
+    /// <summary>
+    /// The options with the default converter of <see cref="TimeSpan"/>, which writes text. The queries use camel case
+    /// names, and Cosmos DB requires the name <c>id</c>.
+    /// </summary>
+    let text = JsonSerializerOptions (PropertyNamingPolicy = JsonNamingPolicy.CamelCase)
+
+    /// <summary>
+    /// The text options with <see cref="TimeSpanTicksConverter"/>, which writes a number of ticks.
+    /// </summary>
+    let ticks =
+        let options = JsonSerializerOptions text
+        options.Converters.Add (TimeSpanTicksConverter ())
+        options
+
+/// The values that the tests of this scenario read.
+[<Struct>]
+type TestData = {
+    /// The durations of each partition, in the order of their creation. They differ in the length of their text and in
+    /// their sign. Thus, no accident puts the text in time order.
+    Durations : ImmutableArray<TimeSpan>
+    /// <summary>
+    /// The partition key of the documents that <see cref="TimeSpanTicksConverter"/> wrote.
+    /// </summary>
+    TicksPartition : string
+    /// The partition key of the documents that the default converter wrote, as old documents hold them.
+    TextPartition : string
+}
+
+/// <summary>
+/// Writes the same durations two times into the container of <see cref="OperationTestBase{T}"/>: through
+/// <see cref="TimeSpanTicksConverter"/>, which writes ticks, into <see cref="TestData.TicksPartition"/>, and through the
+/// default converter, which writes text, into <see cref="TestData.TextPartition"/>.
+/// </summary>
+type TestApplicationFactory (testContext : TestContext) =
+    inherit OperationTestApplicationFactory (testContext)
+
+    let durations =
+        seq {
+            TimeSpan.FromDays 2.0
+            TimeSpan.FromDays 10.0
+            TimeSpan.FromHours 23.0
+            TimeSpan.FromHours 1.0
+            TimeSpan.FromDays 1.0
+            TimeSpan.FromMinutes (-5.0)
+        }
+        |> ImmutableArray.CreateRange
+
+    let ticksPartition = "ticks"
+    let textPartition = "text"
+
+    let documentsIn (partition : string) =
+        durations
+        |> Seq.mapi (fun index duration -> { Id = $"{partition}-{index}"; PartitionKey = partition; Duration = duration })
+        |> ImmutableArray.CreateRange
+
+    let ticksDocuments = documentsIn ticksPartition
+    let textDocuments = documentsIn textPartition
+
+    /// The values that the tests read.
+    member val TestData : TestData =
+        {
+            Durations = durations
+            TicksPartition = ticksPartition
+            TextPartition = textPartition
+        }
+
+    // A client of its own writes the documents, because its serializer decides how Cosmos DB stores a TimeSpan
+    member private this.CreateAsync (options, documents : ImmutableArray<DurationItem>, cancellationToken) : Task = task {
+        use client = Emulator.createClientWithJson options
+        let container = this.GetContainerOf client
+
+        for document in documents do
+            let! response =
+                container.ExecuteAsync (
+                    create {
+                        item document
+                        partitionKey document.PartitionKey
+                    },
+                    cancellationToken
+                )
+
+            CosmosAssert.IsOk (response.Result, $"Seed create should succeed for the document '{document.Id}'.")
+    }
+
+    override this.SeedDataAsync (cancellationToken : CancellationToken) : Task = task {
+        do! this.CreateAsync (SerializerOptions.ticks, ticksDocuments, cancellationToken)
+        do! this.CreateAsync (SerializerOptions.text, textDocuments, cancellationToken)
+    }