fix(keys): compact the key index so dead slots do not accumulate

[bwangll]

Problem

KeyIndex never reuses slots. A key that expires or is removed leaves its slot number behind, so key_count only grows. Every full sync walks 0..key_count synchronously, and full syncs happen:

• whenever a worker starts (nginx reload, worker respawn);
• in every worker after any delete_count bump, which includes add() re-registering a key whose slot expired.

With label churn and an exptime, key_count therefore grows for the lifetime of the shared dict, and so does the cost of starting a worker or of any delete. #18 reclaims the memory of expired entries, but not the slot numbers.

Fix

remove_expired_keys() compacts the index once key_count reaches 10000 slots and at least twice the live keys:

1. take a lock in the shared dict (dict:add with a ttl, refreshed after every batch);
2. copy the live keys, with their remaining ttl, into a new generation of slots (<prefix><gen>_key_N), yielding between batches;
3. switch the generation (<prefix>gen);
4. reconcile with what other workers did to the old generation until they switched: carry over expiries renewed after the copy, drop copies of keys whose old slot no longer holds them, and register slots written past the copied range;
5. delete the old slots; the old key_count gets a ttl rather than a delete.

Other workers switch on their next sync(). The copy is complete before the switch, so no worker lists a partial index. add() checks the generation after writing a new slot and re-registers the key if the slot landed in the old generation after the switch. When add() renews an existing key and the generation has switched since its sync, it renews the key again in the current generation, since the reconciliation may already have run. Only a higher generation counts as a switch, and add() gives up on a key after following 3 switches. remove() now syncs first, so it acts on the current generation.

Worker exit: ngx.sleep(0) is a posted event and does not keep an exiting worker alive, so a compaction yielding that way during a reload would be abandoned with its lock held. Compaction therefore yields with ngx.sleep(0.001), is not started by the premature timer run or by an exiting worker, gives up before the switch if the worker starts exiting, and finishes without yielding after the switch.

Generation 0 keeps the existing key names, so workers running an older version share the index during a reload.

Hot path cost: sync() reads one more dict key, and add() reads one more key when it allocates a new slot.

Tests

Unit tests (prometheus_test.lua) cover: dead slots dropped with ttl preserved, no compaction below the threshold or when most slots are live, lock held by another worker, other workers following the switch, remove() from a worker that has not synced since the switch, a slot written past key_count, a key removed or renewed during the copy, add() racing the switch, a renewal racing the switch, repeated switches during add(), leftovers from an interrupted compaction, an evicted generation node, the premature timer run, a worker exiting before or during the copy, and unchanged metric_data() output. Each fix point was checked by disabling it and confirming the corresponding test fails.

Stress test on OpenResty 1.27.1.2 with 4 workers: 40000 requests, each registering 5 counter and 5 histogram series with a 1 s exptime (30 keys) plus one of 50 long-lived series; remove_expired_keys_interval = 1, compact_min_slots lowered to 300.

	before	after
final key_count	1335612	14932
full sync of a fresh KeyIndex (live keys: 52)	1708 ms	27 ms
compactions	-	13
request counter / long-lived series sum	40000 / 40000	40000 / 40000
duplicate series, library errors, error log lines	0	0

The same run with nginx -s reload after 20 s:

	before	after
final key_count	1328640	13115
full sync of a fresh KeyIndex	2462 ms	17 ms
compactions	-	12, continuing after the reload
request counter / long-lived series sum	39996 / 39746	39996 / 39996
duplicate series, library errors, error log lines	0	0

The reload loses the same 4 requests in both versions (unsynced per-worker counters of the old workers).

Notes

• While copying, live slots exist twice, so compaction needs free space in the dict for the live keys. If a write reports forcible, the compaction aborts, deletes what it wrote and logs an error.
• During a reload from a version without compaction, a compaction started by a new worker deletes the generation-0 slots that old workers still track, and keys that old workers register afterwards stay in generation 0 until a new worker registers them. This only affects the reload that introduces this change.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 9e290e3f-b96b-42b5-94d2-8d76a6559f36

---

_{Comment @coderabbitai help to get the list of available commands.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

bwang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}