feat(eventing): add typed signal-to-event projection architecture

[robbiemu]

Summary

This PR introduces a host-neutral typed signal-to-event projection architecture for Maple’s hosted and Local runtimes, and adds a durable named-consumer boundary to Maple Local.

It separates three concerns:

• source adapters normalize authenticated input into typed signals;
• bounded selectors and pure versioned projectors create deterministic CloudEvents; and
• Maple Local stages, commits, leases, acknowledges, checkpoints, and prunes projected events durably.

It also extracts the scheduled-alert decision and delivery policy into a reusable host-neutral package while preserving existing alert behavior.

Related to #222.

Event paths

Immediate per-occurrence path

authenticated input → source adapter → typed normalized signal → bounded selector → pure projector → durable outbox

The original telemetry continues through the existing warehouse encoder. A matched event is staged before the warehouse write and marked ready only after that write succeeds. Retrying the same source occurrence recomputes the same event identity.

Scheduled aggregate path

warehouse query → observation → alert lifecycle evaluation → factual alert event → existing delivery outbox

Rates, thresholds, percentiles, absence, recovery, flap suppression, and renotification remain scheduled conclusions over a window. They are not modeled as individual ingest-time facts.

Core architecture

A source definition publishes a typed field catalog, including allowed operators, sensitivity, and replay capability. Projection configuration stores a bounded typed predicate AST.

Projection revisions compile into immutable registry snapshots only after source fields, operators, activation time, and closed projector configuration are validated. Evaluation runs every matching projection from one snapshot and isolates failures so one malformed projector does not suppress successful siblings.

Projectors are pure, versioned functions. They declare an ID/version, accepted source kinds, output type/schema, and closed configuration decoder. They perform no I/O or external side effects.

Canonical CloudEvents and identity

Projected events use a common versioned CloudEvents envelope.

Event IDs are SHA-256 hashes over a length-delimited tuple of tenant, source kind, source, source occurrence ID, projection ID, and projection revision. Two optional backward-compatible extensions expose source occurrence identity and its quality. Historical envelopes without those extensions remain valid.

This lets downstream consumers correlate source occurrence → immutable Maple event → deterministic transport transaction without parsing event data.

Durable Local outbox and consumers

Maple Local stores projection revisions, active pointers, bounded failures, staged/ready events, and consumer state in a private SQLite control database.

Named consumers support:

• explicit registration at the beginning or current tail;
• whole-batch claims under bounded leases;
• exact acknowledgement;
• replay after lease expiry;
• rejection of stale, wrong, or partial acknowledgements; and
• pruning only through the lowest active-consumer acknowledgement.

Staged events are never pruned. Ready ordering is stable across restart and schema migration. Checkpoint manifests bind the control snapshot alongside the existing data backup.

Alert-core extraction

The new alerting-core package owns host-neutral observation evaluation, trigger/resolve/renotify planning, flap suppression, no-data recovery safety, scheduling helpers, delivery idempotency, and bounded retry policy.

Existing alert queries, persistence, queue behavior, and delivery payloads remain compatible. The factual event envelope is additive.

Existing producer convergence

The existing verified provider-webhook path now creates its factual event through the common projection seam while retaining queue compatibility, including jobs queued before deployment.

This demonstrates the architecture without making any provider-specific vocabulary part of the projection core.

Safety and boundedness

The implementation enforces:

• bounded predicate depth, clause count, and string-literal bytes;
• exact scalar typing without implicit coercion;
• bounded CloudEvent size and schema validation;
• immutable revision identity;
• source/tenant isolation;
• bounded low-cardinality telemetry; and
• no payloads, URLs, credentials, identifiers, or arbitrary field values in eventing metrics.

Deliberate boundaries

This PR does not:

• add a provider-specific lifecycle vocabulary or projector family;
• implement transport delivery, destination topology, or agent policy;
• add environment-specific paths, destination identities, credentials, or deployment configuration;
• make projectors call external providers;
• add a required broker;
• replace scheduled aggregate alerts with ingest selectors;
• expose arbitrary SQL or executable projection configuration;
• claim exactly-once external side effects; or
• activate projections automatically.

Provider adapters, deployment policy, transport delivery, and live credentials remain separate integrations built on the generic contracts introduced here.

Review guide

Primary surfaces:

• packages/eventing-core: typed model, predicates, source/projector registries, deterministic identity, schemas, and fixtures;
• packages/alerting-core: alert evaluation, lifecycle planning, idempotency, scheduling, and retry policy;
• apps/cli/src/server/eventing: source-neutral normalization, telemetry, runtime, SQLite state, outbox, and consumer protocol;
• apps/cli/src/server/serve.ts: decode-once integration and authenticated control/consumer endpoints;
• apps/cli/src/server/checkpoints.ts: eventing-control checkpoint participation;
• hosted alert services and the existing provider-webhook runtime;
• docs/signal-to-event-projection.md and docs/local-event-consumers.md; and
• docs/eventing-extension-guide.md: a complete compile-time source adapter and projector walkthrough with host wiring, versioning, testing, and review checklists.

Review status

Ready for review. Current upstream main is merged into the branch, and GitHub reports it mergeable.

Validation

Against the clean provider-neutral tree:

• 249 focused post-merge tests pass across alerting core, eventing core, Local runtime, ingest, consumer-control, and hosted API suites;
• generated-schema and checkpoint compatibility are covered;
• migration, source-tuple collision handling, staged-retry recovery, pre-swap snapshot restoration, locale-stable source fingerprints, restart, lease expiry, stale acknowledgement, pruning, and snapshot restore are exercised;
• existing provider queue compatibility and alert behavior remain covered;
• after merging current upstream main, 146 host-neutral core/Local tests and 103 hosted API tests were rerun successfully;
• eventing-core, alerting-core, and CLI typechecks pass; the repository-wide API test typecheck still reports unrelated errors inherited from current upstream main; and
• git diff --check passes.

Review change ledger

• Typed failures and validation: replaced throw/instanceof paths with tagged errors and schema/Result/Effect decoding; predicate depth and total-node bounds now live in the root schema.
• Event contract: clarified subject inheritance/clearing, shortened extensions, documented length-delimited IDs and supported runtimes, and made generated schemas deterministic.
• Alert lifecycle: derives shared domain types, uses one hysteresis fold for scheduler and preview, converts projection defects into typed delivery failures, and keeps stored event payloads forward-compatible.
• PlanetScale compatibility: retained timestamp fallback and inline ignore/log handling, isolated queue failures per message, made legacy jobs readable, changed deleted-issue redelivery to skipped, and added receipt retention/indexing. Migration metadata was regenerated and rollout ordering is explicit.
• Local outbox: collapsed the new control store to schema v1 and added it to the schema gate. Transactional usage counters replace full scans; projection overflow preserves warehouse ingest and records a durable delivery gap with explicit abandon/accept recovery.
• Local HTTP and OTLP: request bodies are strictly decoded, OTLP batch normalization reuses resource/scope work, zero-projection ingest avoids the outbox lookup, and maintenance CORS exposure was removed.
• Checkpoints and lifecycle: kept the warehouse/control snapshot atomic, moved asynchronous control-file writing outside the admission gate, documented the native backup availability cost and forward-only checkpoint compatibility, and typed startup/shutdown failures.
• Shared plumbing: consolidated local token-file helpers, confirmed CLI metric export, added SQLite-backed integration coverage, registered new packages with knip, and called out the unrelated hostname-fixture cleanup.
• Validation: 579 CLI and 460 API tests pass, along with eventing/alerting core tests, relevant typechecks, Effect lint, generated-schema checks, and migration/schema-control checks.

[robbiemu]

I now have a working, mostly tested and verified version of this. Just putting a final review / finishing touches on it

[Makisuo]

This is pretty cool and I think this moves alerts into a cool direction and agree with most patterns you build here already tbh!

This is quite a massive PR and from looking through it there are quite a lot of small nitpicks/patterns etc.
Do you mind if I go through it and fix those up, or would you rather me go through and comment all of them for you to fix? Fine with both!

Anyways this looks great and want to get this in asap

[robbiemu]

Do you mind if I go through it and fix those up, or would you rather me go through and comment all of them for you to fix?

Yes some architectural changes can grow into a bit of a problem, we had to move where and how signals are made into events. One thing we could do if you are not wanting to go through so much manually is to pick archetypical examples to call out, then I can take that input to guide my agent to find and repair similar issues throughout.

Not to say I object to you just doing it; it might be the most direct way to get what you want.

I'm just happy i don't need to maintain a parallel fork! Happy to get this in.

[Makisuo]

Went through the whole thing properly now (had a worktree with the PR head, typecheck + all the new tests are green btw, lint isnt). Still think the direction is right and want this in, but there is one systemic thing and a couple of real bugs that need to happen before merge. Ill try to give archetypes like you suggested so you can let your agent sweep the rest.

1. never try/catch, no plain Error, no instanceof — this is the big one

The whole PR is written in throw land. Across the non-test diff its ~143 throw new Error, ~39 try blocks, 6 classes extending native Error, and zero Schema.TaggedError. In effect land a thrown error is invisible to the type system, it escapes the typed error channel and one catch block flattens every failure into a single branch. That is exactly what breaks here in prod:

• apps/api/src/planetscale-webhook-runtime.ts:54 calls projectPlanetScaleWebhookEvent / planetScaleWebhookPayloadFromEvent synchronously inside the queue consumer. any throw there is a defect and kills the whole Effect.forEach batch, so sibling messages get neither ack nor retry.
• AlertDestinationDelivery.ts:135 calls projectAlertLifecycleEvent inside Effect.gen, its throws become defects instead of AlertDeliveryError.
• planetscale-webhook.http.ts maps every projection throw to a 503 and drops the cause, so PlanetScale retries a request that is deterministically bad. should be a 400 with a tag.

How this should look, using your own code as the examples:

errors are Schema.TaggedError with a reverse domain tag, a message and the context a caller needs to act. not class X extends Error {}

// packages/eventing-core/src/predicate.ts:28 — today
export class SignalPredicateValidationError extends Error { ... }

// instead
export class SignalPredicateInvalid extends Schema.TaggedError<SignalPredicateInvalid>()(
  "@maple/eventing-core/SignalPredicateInvalid",
  {
    message: Schema.String,
    issues: Schema.Array(Schema.Struct({ path: Schema.String, message: Schema.String })),
  },
) {}

// apps/cli/src/server/eventing/control-store.ts:321 — today
export class EventConsumerInputError extends Error {}
export class EventConsumerNotFoundError extends Error {}
export class EventConsumerConflictError extends Error {}

// instead (and give them the fields, thats the whole point)
export class EventConsumerNotFound extends Schema.TaggedError<EventConsumerNotFound>()(
  "@maple/cli/eventing/EventConsumerNotFound",
  { message: Schema.String, consumerId: Schema.String },
) {}

export class EventConsumerLeaseConflict extends Schema.TaggedError<EventConsumerLeaseConflict>()(
  "@maple/cli/eventing/EventConsumerLeaseConflict",
  { message: Schema.String, consumerId: Schema.String, leaseId: Schema.String, expiresAtMs: Schema.Number },
) {}

validation is a schema, not a chain of throws. event.ts:112-125 is five if (...) throw new Error("...") in a row on the projector input. that is a Schema.Struct with Schema.NonEmptyString / checks, decoded with Schema.decodeUnknownEffect (or decodeUnknownResult if it has to stay sync), and the error is a SchemaError you can mapError into one tagged error. same for registry.ts:32-35, source.ts, and the whole webhook-events.ts:84-251 block.

dont catch to return null/false, use the primitive.

// predicate.ts:107 — today
const parseInt64 = (value: string): bigint | null => {
  try { const parsed = BigInt(value); return ... } catch { return null }
}

// instead: Schema.BigInt (or BigIntFromString) + a check for the int64 range, then
Schema.decodeUnknownOption(Int64FromString)(value)

// PlanetScaleWebhookQueue.ts:40 — today: try/catch inside Schema.makeFilter
// instead: planetScaleWebhookPayloadFromEvent returns Result/Option and the filter is
Schema.makeFilter((job) => !("event" in job) || Result.isSuccess(payloadFromEvent(job)), ...)

never instanceof / regex on message to decide what an error is. serve.ts picks HTTP status with instanceof, control-store.ts:1080 does error instanceof EventConsumerConflictError && /lease/.test(error.message) to pick a metric outcome. with tagged errors this is

Effect.catchTags({
  "@maple/cli/eventing/EventConsumerNotFound": (e) => Effect.succeed(text(e.message, 404)),
  "@maple/cli/eventing/EventConsumerLeaseConflict": (e) => Effect.succeed(text(e.message, 409)),
})

and the metric outcome is just error._tag.

request bodies are Schema.Struct decodes. serve.ts:530-535 validates the consumer endpoints with Object.keys(record).sort().join(",") === "consumerId,leaseSeconds,limit" + Schema.is(Schema.String). limit: 1.5 gets through that today and blows up inside the store. one Schema.Struct({ consumerId: Schema.NonEmptyString, leaseSeconds: Schema.Int.check(...), limit: Schema.Int... }) per endpoint, decoded with Schema.decodeUnknownEffect, and the SchemaError maps to a 400. readBoundedJson (serve.ts:617) hand rolling a stream reader + JSON.parse("") to force a SyntaxError can go the same way (Schema.fromJsonString).

casts are a hack imo. 13 inline as X and 8 non-null ! in the diff (otlp.ts:323,491 (request ?? {}) as OtlpLogsRequest on an unknown body is the worst one, a key: 123 reaches Buffer.byteLength(123) and is a 500 not a 400). AlertsService.ts:268 replaced one cast with row.payloadJson as Record<string, unknown> and throws away the decoded value. satisfies is fine, decode is better.

Repo has the lint for this btw, oxlint -c .oxlintrc.effect.json on the touched dirs gives 17 errors (8 non-null assertions, 6 extends-native-error, 3 no-try-catch) so CI would fail once its approved on the fork.

2. blockers (actual bugs)

• poison message. webhook-events.ts:598 throws when a receipt exists without its issue. issues get hard deleted elsewhere (ErrorsService.ts:1446), receipts have no FK and are never pruned, so a redelivery after a delete retries forever until the queue DLQs it. should be a typed skipped outcome.
• hand typed migration timestamp. packages/db/drizzle/meta/_journal.json 0055 has when: 1788825600000 which is exactly midnight UTC. drizzle only applies entries newer than the last applied one, so any migration generated on another branch before that instant and merged after this one gets silently skipped. please regenerate with bun run --cwd packages/db db:generate.
• local outbox fails closed on ingest. control-store.ts:746 throws on maxOutboxEvents/Bytes before the chDB insert, serve.ts:290 turns that into a 503 with accepted: 0, so the warehouse rows are dropped too. ready events only prune through consumer acks and staged ones have no abandon path, so one dead consumer eventually blocks all local ingest. needs a bounded / abandon path, the doc mentions one but it doesnt exist.
• backup is now exclusive. serve.ts:593 moved the checkpoint backup from admitted to gate.exclusive, so every request 503s for the whole BACKUP DATABASE. i get why (control snapshot consistent with the chDB backup) but its an availability regression for the whole listener that isnt mentioned anywhere. want to talk about this one.

3. other stuff

• planAlertLifecycle in alerting-core re-implements the fold incident-hysteresis.ts was created to unify. scheduler now runs alerting-core, the preview path (alert-firing-spans.ts) still runs foldObservation. parity test proves they agree today but its the same mechanic in two places again, pick one.
• alerting-core redeclares AlertComparator / AlertEventType as string unions instead of deriving from @maple/domain. it is maple alert knowledge so packages/ is right, just dont keep a second copy in sync by hand.
• predicate decoder bounds clauses per level but not depth or total nodes, the budget guard is a separate pre-decode call that compile and both control-store decode sites skip. put the bound in the schema (a check on the root) or in compile.
• docs say the event id is sha256 over the tuple, code hashes a length delimited tuple with a maple-event-v1 prefix. anyone implementing from the doc gets different ids.
• control DB ships a 1→4 migration chain + schema-3 preflight + 4 tests for versions that never existed on main. collapse to v1 and please hook it into the local schema gate.
• MAX_DELIVERY_ATTEMPTS in AlertsService is dead now, consumer-auth.ts duplicates archives/retention.ts token helpers, otlp.ts redeclares AnyValue/KeyValue from otlp/encode.ts, maple.eventing.* metrics are recorded but nothing exports metrics from the cli.
• StoredDeliveryPayloadSchema.event should be Schema.Unknown passthrough, api never reads it back and a strict decode of already queued rows turns into a non retryable AlertValidationError on the next tightening.
• x-maple-maintenance-token got added to local CORS allow-headers, nothing in local-ui uses it. drop it.
• narrative comment blocks (otlp.ts:479, control-store.ts:1254, registry.ts:14) → 1-3 lines, the story goes in the PR body.
• docs: "Status: implemented on codex/issue-222-alerting-core" and "packages/alerting-core (existing)" are both wrong, its new in this PR.

4. behaviour changes I want to be explicit about

Did a separate pass on what this changes for stuff thats already running, since a lot of it isnt in the PR body. Some of these are fine, some I want to decide on explicitly:

• PlanetScale webhooks without timestamp now 400 (planetscale-webhook.http.ts:171, used to fall back to receivedAt) and >120KB bodies 413. PlanetScale retries a few times and then drops, so thats permanent loss if they ever omit it. every existing route test had to gain a timestamp field to keep passing which is a hint. I'd keep the fallback.
• every non-test event is now queued, including ignore/log ones that were acked inline before (old else branch at :228 is gone, test at .http.test.ts:319 got inverted). thats 100% of webhook traffic through the queue for the consumer to ack without persisting. dont think we want that.
• redelivery semantics: byte identical redelivery is now skipped instead of occurrenceCount+1, but a redelivery where PlanetScale reserialises resource differently still counts, and timeline inserts arent receipt guarded so those still duplicate. fine with skipped, but the timeline should be guarded too then.
• legacy queue jobs without timestamp get acked and discarded at deploy (planetscale-webhook-runtime.ts:46-60) instead of processed. small window but silent.
• migration 0055 is a hard ordering dependency: consumer inserts receipts inside the issue tx, so if the worker rolls before the migration every issue worthy message retries until DLQ. prod migrations are manual here so I'll handle that, but I'd rather ship the table in its own PR first. also no TTL on receipts, low volume but unbounded.
• customer alert webhook + hazel bodies gain the ~1KB event object with tenantid, projectionid, sha256 id. additive and fine, just needs a docs line. side note makeCloudEvent throws on a non finite value (event.ts:47) so a NaN ratio that used to serialise as null is now a defect in the delivery fiber.
• alert evaluation: diffed planAlertLifecycle against the old machine, open/resolve/renotify/flap/no-data/retry are all equivalent and pg round trips per tick are unchanged. good.
• maple local for users with zero projections: startup now creates control/eventing.sqlite + a .event-consumer-token secret file and refuses to start if either fails, one count(*) per OTLP request, and the exclusive backup 503 from above. checkpoint format bumps to v2 so the old CLI cant list/restore new checkpoints and old reset refuses a data dir with control/ in it. forward compatible only, needs a note in the release.
• deps: bun.lock changes are all workspace links, no new third party packages. new packages arent in knip.json yet.

Happy to pair on any of this, and once the error model is flipped ill go through it again quickly.

[Makisuo]

inline version of the comment above so the agent has anchors, top comment has the archetypes and examples

[robbiemu]

Review pass is in f75ee2b44.

The systemic Effect work is done across the new eventing surfaces: tagged errors, schema/Result/Effect decoding, bounded predicates at the root schema, and removal of the reviewed casts, non-null assertions, and catch-based control flow. Alert lifecycle now shares one hysteresis implementation and domain types.

PlanetScale keeps its old timestamp fallback and inline ignore/log behavior, legacy queue jobs remain readable, poison redeliveries are skipped, sibling queue messages are isolated, and receipts now have indexed retention. The migration metadata was regenerated and the rollout dependency is documented.

Local eventing now starts at control schema v1 under the normal schema gate. Outbox accounting is constant-time; capacity loss no longer drops warehouse rows and is exposed as a durable delivery gap with explicit recovery. HTTP/OTLP decoding, metrics, token helpers, CORS, checkpoint compatibility, startup/shutdown handling, knip, schemas, and real SQLite coverage were also tightened.

I kept the checkpoint capture atomic rather than allowing control state to get ahead of warehouse state, but moved the asynchronous file write outside the admission gate and documented the remaining native-backup availability cost.

Validation is green: 579 CLI tests, 460 API tests, eventing/alerting core tests, relevant typechecks, Effect lint, generated schemas, and migration/schema-control checks.

---

I can't believe Astra did nearly all of that in just one commit. Sorry, I will be more deliberate in my prompting next time.