Skip to content

agent, ntf server: add lookupOrInsertIO, use it for locks and stats - #1989

Merged
epoberezkin merged 12 commits into
simplex-chat:masterfrom
evgeny-simplex:ep/lookup-or-insert-io
Oct 26, 2026
Merged

epoberezkin merged 12 commits into
simplex-chat:masterfrom
evgeny-simplex:ep/lookup-or-insert-io

Conversation

@evgeny-simplex

Copy link
Copy Markdown
Contributor

TMap

  • lookupOrInsertIO: looks the key up without a transaction; if absent, looks it up again and inserts in one transaction. Returns Right for a found value, Left for an inserted one.

Agent

  • Map locks (connection, invitation, chat entity and service locks via withLockMap/withLocksMap): an existing lock is found without a transaction and then taken; a new lock is inserted already taken. A thread waiting for a lock no longer re-runs its transaction on every insert into the lock map.
  • createLock is removed (unused).

Ntf server

  • incServerStat uses lookupOrInsertIO instead of its own double lookup.

Comment thread src/Simplex/Messaging/Notifications/Server/Stats.hs Outdated
Comment thread src/Simplex/Messaging/Agent.hs Outdated
@epoberezkin
epoberezkin merged commit c0c416d into simplex-chat:master Oct 26, 2026
8 checks passed
@evgeny-simplex

Copy link
Copy Markdown
Contributor Author

Benchmark: A = 3818284 (base) vs B = c0c416d (this PR, merged)

Method

  • One harness source (server-bench, client-bench) was built twice with the same flags (-O2); the two cabal plans differ only in paths. The only other source change between A and B is in the XFTP server (xftp server: release storage of interrupted uploads #1977), which is absent from both binaries.
  • Server: one SMP server (in-memory store, TLS, 4 cores) and one load process (4 other cores), with 10,000 queues (50,000 in one storm configuration) prefilled with one message each. service: one SUBS for all queues. individual: SUB in batches of 1,000. storm: 8 connections send SUBs at once. 32 threads send 256-byte messages, unlimited or capped at 300/s: for 10 s after subscribing in service and individual, and during the SUBs in storm. With unlimited sending, few ACKs are completed within the window, so delivered/s is the throughput metric there.
  • Client: an agent with 5,000 or 20,000 connections on 1 or 4 local servers, restored from a snapshot. subscribe: from subscribeAllConnections until all connections are UP. resubscribe: from the server restart until all are UP again. traffic: another agent sends 100 messages/s on the same connections. SQLite runs with synchronous = OFF.
  • A and B alternate run by run, and the order flips each repetition. p is an exact two-sided Mann-Whitney test.
  • The 8-CPU host was shared with other workloads. Load average at measurement start was 6.3-13.7 in the server runs and 3.6-7.5 in the client runs. Hypervisor steal during the server runs was a median 0.56 CPU (max 2.0).

Server (2026-10-26, 22:38-23:43 UTC)

configuration metric A median [min-max] B median [min-max] B vs A p
service 10k unlimited (6+6) SUBS response, s 0.10 [0.09-0.17] 0.11 [0.10-0.17] +9% 0.56
delivered/s 389 [299-456] 370 [324-467] -5% 1.00
server CPU, us/delivered 6883 [5725-7198] 6712 [4715-8005] -2% 0.70
individual 10k unlimited (6+6) SUB response, s 6.84 [4.84-7.90] 5.36 [4.77-8.51] -22% 0.59
delivered/s 308 [278-356] 372 [359-419] +21% 0.002
server CPU, us/delivered 7702 [6591-8477] 6577 [5360-7146] -15% 0.06
storm 10k unlimited (6+6) SUB storm window, s 4.99 [4.49-5.85] 6.33 [4.92-6.82] +27% 0.04
ACK p50, ms 439 [364-500] 527 [417-614] +20% 0.24
server CPU, us/delivered 946 [897-1063] 1041 [939-1184] +10% 0.13
service 10k 300/s (6+6) SUBS response, s 0.10 [0.08-0.16] 0.11 [0.08-0.12] +10% 0.94
ACK p50, ms 19.3 [15.6-20.3] 18.1 [15.1-20.9] -6% 0.70
server CPU, us/delivered 2358 [2185-2558] 2316 [2232-2608] -2% 0.70
individual 10k 300/s (6+6) SUB response, s 6.70 [5.23-7.88] 6.04 [4.53-7.24] -10% 0.24
ACK p50, ms 19.8 [16.6-22.1] 18.6 [15.3-21.4] -6% 0.94
server CPU, us/delivered 2500 [2214-2662] 2348 [2125-2486] -6% 0.18
storm 10k 300/s (6+6) SUB storm window, s 5.62 [3.74-6.47] 5.04 [4.40-5.64] -10% 0.48
ACK p50, ms 486 [316-585] 405 [311-495] -17% 0.31
server CPU, us/delivered 976 [892-1174] 980 [944-1080] 0% 0.82
storm 50k unlimited (5+5) SUB storm window, s 31.8 [27.8-34.9] 29.7 [26.8-33.4] -7% 0.69
ACK p50, ms 531 [460-566] 510 [447-530] -4% 0.42
server CPU, us/delivered 1097 [1081-1200] 1075 [1055-1136] -2% 0.22
individual 10k unlimited, replication (5+5) SUB response, s 7.16 [5.13-7.99] 6.62 [4.44-6.77] -8% 0.10
delivered/s 342 [293-450] 362 [336-397] +6% 0.42
server CPU, us/delivered 7313 [6292-8324] 6922 [6468-7100] -5% 0.69

Client (2026-10-26 23:53 to 10-27 00:12 UTC; the run stopped during repetition 8)

configuration metric A median [min-max] B median [min-max] B vs A p
5k, 1 server, quiet (8+8) subscribe, s 1.23 [1.06-1.50] 1.23 [1.08-1.42] -1% 0.88
resubscribe, s 1.07 [0.96-1.55] 1.06 [0.94-1.47] -1% 0.78
5k, 1 server, traffic (8+8) subscribe, s 1.76 [1.57-2.50] 1.85 [1.62-2.56] +5% 0.44
resubscribe, s 1.59 [1.46-2.04] 1.60 [1.22-3.54] +1% 0.80
5k, 4 servers, quiet (8+8) subscribe, s 0.66 [0.60-0.82] 0.66 [0.56-0.88] 0% 0.65
resubscribe, s 0.41 [0.38-0.59] 0.43 [0.36-0.85] +5% 0.55
5k, 4 servers, traffic (8+8) subscribe, s 1.00 [0.76-1.50] 1.03 [0.84-1.50] +4% 0.57
resubscribe, s 0.65 [0.49-0.81] 0.65 [0.52-0.77] -1% 0.80
20k, 1 server, quiet (7+7) subscribe, s 4.82 [4.22-5.64] 4.76 [4.53-6.62] -1% 0.80
resubscribe, s 4.24 [3.76-4.97] 4.10 [3.93-4.78] -3% 0.46
20k, 1 server, traffic (7+7) subscribe, s 7.47 [6.61-8.48] 8.52 [7.42-32.4] +14% 0.02
resubscribe, s 6.95 [5.80-8.30] 7.34 [5.84-16.9] +6% 0.38
20k, 4 servers, quiet (7+7) subscribe, s 2.61 [2.40-3.49] 2.50 [2.44-2.85] -4% 0.62
resubscribe, s 1.25 [1.14-2.19] 1.32 [1.19-1.52] +6% 0.60
20k, 4 servers, traffic (7+7) subscribe, s 4.21 [3.71-5.17] 4.03 [3.40-4.60] -4% 0.62
resubscribe, s 2.47 [2.03-3.12] 2.78 [2.02-6.85] +13% 0.38

Re-runs (2026-10-27, 03:29-03:40 UTC; same scripts and arguments)

In the tables recomputed from the raw logs, storm 10k unlimited also has p < 0.05 (B slower), so it was re-run too. Host load was higher than in the first runs: load average 14-22, with 1.4-2.2 CPUs in I/O wait (0.0-0.2 in the first server runs). With 3+3 runs alone, p cannot be below 0.10. In the "all runs" rows, the first runs and the re-runs are pooled.

configuration metric A median [min-max] B median [min-max] B vs A p
individual 10k unlimited, re-run (3+3) delivered/s 340 [288-366] 371 [315-372] +9% 0.40
server CPU, us/delivered 7224 [5918-7880] 7118 [6849-7327] -1% 1.00
individual 10k unlimited, all runs (14+14) delivered/s 340 [278-450] 370 [315-419] +9% 0.002
server CPU, us/delivered 7477 [5918-8477] 6820 [5360-7327] -9% 0.08
storm 10k unlimited, re-run (3+3) SUB storm window, s 7.14 [5.80-7.43] 6.08 [5.31-6.75] -15% 0.40
storm 10k unlimited, all runs (9+9) SUB storm window, s 5.31 [4.49-7.43] 6.08 [4.92-6.82] +15% 0.30
client 20k, 1 server, traffic, re-run (3+3) subscribe, s 12.5 [8.62-13.7] 11.2 [10.5-11.9] -11% 0.70
resubscribe, s 13.4 [11.5-13.6] 17.3 [10.7-50.5] +29% 0.70
client 20k, 1 server, traffic, all runs (10+10) subscribe, s 7.85 [6.61-13.7] 9.17 [7.42-32.4] +17% 0.11
resubscribe, s 7.06 [5.80-13.6] 7.89 [5.84-50.5] +12% 0.39

Conclusion

  • One difference is confirmed. In individual with unlimited sending, B delivers 9% more messages per second (370 vs 340, 14+14 runs, p = 0.002). B was higher in all three re-run pairs. Server CPU per delivered message is 9% lower in B, but p = 0.08.
  • The storm slowdown was not confirmed. In the re-run, B was 15% faster; over all 9+9 runs, B vs A is +15% with p = 0.30. With 50k queues, storm times do not differ (-7%, p = 0.69).
  • The client slowdown (20k connections, 1 server, traffic) was not confirmed. In the re-run, B was 11% faster; over all 10+10 runs, B vs A is +17% with p = 0.11. Client CPU per second of the window was the same in A and B (1.34 vs 1.30 CPU-seconds per second in the first runs), so client CPU time is proportional to the window length.
  • All other differences are within run-to-run spread: the A and B ranges overlap, and p > 0.05. The host carried external load (steal time in every sampled run, heavy I/O wait in the re-runs), and the spread is wider under that load.
  • Observation: all 6 values above twice their configuration median are from B (5 of 63 B runs, 0 of 63 A runs; Fisher p = 0.06). Five of them are resubscribe times; the largest is 50.5 s against the re-run median of 13.5 s. This threshold was chosen after seeing the data, so a dedicated run of the reconnect path is needed to confirm or reject it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants