Repository navigation
Hosted GitHub source: take GitHub records off the channel streams instead of polling - #539
Merged
Merged
Conversation
sync.github.mode chooses between polling GitHub with the gh CLI and taking the records that the channel gateway polls. It is restart-only and parsed like slack.mode: an unknown value falls back to poll with a warning. sync.github.pr_watch asks the gateway to watch the agent's own open pull requests. cron.delivery_debounce_seconds (0-600, default 60) spaces the job runs that a hosted delivery triggers. A reload publishes the hosted source subscriptions to the gateway again after the source runners are rebuilt.
A source_record event carries one record of a hosted source in a typed record member: id, record type, summary, content, timestamp, and metadata entries that hold a string or a list of strings. Its check requires the id, type and summary and bounds the text by the stream's text limit. The new subscriptions frame goes one way from Nerve to the gateway and names the agent's hosted sources with their filter patterns and the pull request watch. Nerve checks the gateway's bounds before it sends one, and can drop the patterns when a subscription would not fit.
With sync.github.mode set to hosted, the hosted runtime starts also without a hosted chat provider and gives the inbox reader a consumer for the provider's source records. The consumer writes each record to the channel_observations buffer under the provider's name and acknowledges it once the row is stored; it is not a channel, so the router never sees it. A delivery listener hears of every stored record. Each stream publishes the agent's subscriptions once it has negotiated, built from sync.github: the guardrail patterns as filters and pr_watch. A switched-off source subscribes to nothing. A list the gateway would refuse is published wider, never narrower, so the gateway may store a record the inbox filter still drops but never drops one it keeps. The fake gateway builds source record events and keeps the subscriptions frames it receives.
The notification record builder and the GitHub inbox filter move out of GitHubSource.fetch and the registry into github.py, so the polling and hosted paths share them. In hosted mode the registry builds HostedGitHubSource instead of GitHubSource: it reads the buffered records and maps each payload to a SourceRecord without calling GitHub. The runner keeps the name github, so its guardrail, drop hook, condensing, health checks, job id and gates stay as they are. A cursor that is not a timestamp, such as a buffer position left by the hosted source, made every polling fetch fail; it now falls back to the first-run lookback.
In hosted mode the github_pr_activity gate passes without running gh, since the gateway's pull request watch delivers the changes as records. nerve doctor reports the hosted GitHub settings, warns about jobs that still rely on that gate, and no longer asks for gh when nothing needs it. The setup page skips the gh sign-in check, shows that gh is not needed, and reports the gateway's streams also when only GitHub is hosted.
…sted-github-source
CronService.on_source_delivered drains the source at once, after a run of it already in progress, and runs it once more for records delivered meanwhile. Then each enabled job gated on messages for that source, or for any source, gets one run cron.delivery_debounce_seconds after the first delivery of a burst. The run waits for a run of the job in flight, and while it goes on a scheduled or catch-up fire of the job is skipped, or waits when the job has lock. It never takes the reload lock. The gateway server points the hosted runtime's delivery listener at the cron service once it has started, and hands the runtime the database. Source reloads keep sync.github.mode at the value the service started with, since the mode is restart-only.
Contract tests for source records and the subscriptions frame; the settings, the subscriptions builder, the record consumer, the buffer mapping and the registry; end-to-end runs against the fake gateway, from a delivered record to the inbox and to the one gated job it runs; the pull request gate, doctor and setup page in hosted mode; and the delivery semantics of the cron service.
Config parsing keeps every integer setting as written, so the 600 second ceiling of cron.delivery_debounce_seconds now applies when the cron service reads it, and nerve config validate warns about a larger value. A negative value still reads as 0.
A github_pr_activity record is about one of the agent's own open pull requests: its reason is always author, and its actors are the reviewers and commenters. The actor, reason and CI branch rules of sync.github are written for notifications, so an ordinary allow list of actors or reasons dropped every such record. The inbox filter now holds these records to allow_repos and deny_repos only, as the gateway does before it stores a record. The docs state the record's shape and the rule.
Nerve reads the events of every hosted provider from one ordered inbox and stops reading while a hosted channel cannot take events. With slack.mode: hosted and slack.enabled: false, nothing is read from the gateway, so hosted GitHub records wait there too and no delivery starts a job. nerve doctor now warns about that combination, the docs say so where they describe pausing hosted Slack and the hosted GitHub source, and a test pins how records are held back and released. Reading each provider's events on its own is left for later.
The delivery's own drain logs its start before it runs the source runner, so a scheduled fire or a manual sync can take the runner in between. The delivery's run then returned at once, and the jobs gated on the source could run before the records were in the inbox; with no debounce that delivery's run was spent. After its run, the drain now waits until the runner is idle again. Any run that took the runner started after the records arrived, so once it ends they are stored.
A run for delivered source records left the pending runs once it started, and stop() cancelled only those, so a run in progress went on after the cron service stopped. That matters when startup fails part way and the cron service is stopped before the hosted streams. Started runs now have their own set, which stop() cancels and waits for with the pending ones. Once stop() has begun, a delivery starts nothing: the drain does not run again, no run is scheduled, and a run about to start returns. The stop() docstring names everything it stops.
The local check now covers what the gateway also checks on a provider: a lowercase token within its byte limit, named once per frame. A frame that fails the check went out without its filter patterns, even when that could not make it valid. It is now checked again, and if it still fails nothing is sent and a warning is logged, since the gateway would close the stream over it.
In hosted mode the gateway applies the published patterns before it stores a record, so its matcher and the inbox filter must agree, or a record kept here would be dropped there. The same ten pattern and value pairs now run against the inbox filter: a star crosses a slash, a question mark is one character, classes and negated classes work, case is ignored, and a list value matches on any element.
A reload sends the subscriptions again on every ready stream and waits for all of them. Each send waited for the stream's send lock with no deadline, so one stalled stream held up the whole reload. Each stream now gives up after two seconds, as its drain does, and logs it; the other streams still send, and the reload reports how many did. A stream whose publish raised was dropped without a word, and the publish that negotiation starts surfaced a failure only when the stream closed. Both are now logged when they happen.
sync.github.mode is restart-only: the source runner and the hosted streams keep the mode they started with until a restart. The github_pr_activity gate read the live config instead, so after a reload that changed the mode it passed without gh while the box still polled, or ran gh while the gateway already delivered the records. It now reads the configuration the process started with, and the loaded config only in a process that did not start the gateway. The hosted test of the gate made gh raise, which the gate catches and treats as a reason to run, so it could not fail. The tests now record the processes the gate starts and check that none ran.
Two links of hosted GitHub had no test that would notice them missing: the startup step that hands the cron service's delivery hook to the hosted runtime, and the call in reload_all that publishes the subscriptions again. The startup test runs the real lifespan with GitHub hosted and checks what the source consumer calls; the reload test runs reload_all against open streams and checks the frame and the summary.
A consumer cursor expires after sync.consumer_cursor_ttl_days without a read that moves it, and an expired cursor starts again at the source's newest message. A messages gate read the cursors on every check without renewing them, so a pull request monitor whose source stayed quiet over a weekend lost the first new record: the gate restarted the cursor at it and reported nothing, and so would the job's own poll. Each check now renews the TTL of the live cursors it reads, once less than half of it is left, without moving them; the forecast still writes nothing, and a failed renewal leaves the gate's answer as it was. A cursor that has already expired starts again at the newest message, as before.
The cron docs said that a github_pr_activity record starts a job gated on that gate at once. A delivery starts only jobs gated on messages for the source; a job on the PR gate alone is never started by one, and in hosted mode it runs in full on every scheduled tick. Delivery runs are also limited to jobs that are on the schedule. The sources docs now give the safe way back from hosted to poll: switch the source off and reload first, so the gateway is told to stop, then change the mode and restart. They also say that the gateway must know the subscriptions frame, that blank patterns are not sent, and that a subscription above the gateway's frame limit goes without patterns.
A setup that may not store credentials offered the GitHub source only when its token came from the environment. When the channel gateway polls GitHub no credential is used on this machine at all, so the switch is now offered there too, with its own reason beside the closed Gmail and Telegram. In hosted mode the switch writes sync.github.enabled alone: the events source reads with gh on this machine, which hosted mode does not set up. The Sources step also shows what the gateway reports about its streams when GitHub is hosted, as the Channels step does for a hosted Slack, so a setup with only GitHub hosted sees it too.
The route that refuses closed sources now describes the GitHub exception in the same words as the checklist it relies on: the channel gateway polls GitHub, or its token comes from the environment.
The polling source falls back to the last 24 hours when its cursor is not a timestamp, as after a switch back from hosted mode, where the cursor is a buffer position. Eight digits parse as a date, though, so such a position could become a since date years in the past. A cursor of digits alone now always counts as no timestamp.
The enrichments of a listing are gathered with their exceptions returned, and a cancelled one comes back as a CancelledError, which is not an Exception. notification_record is annotated to take any exception there, but checked for Exception only, so such a result failed the whole fetch. It now gives a record built from the notification alone, as any failed enrichment does.
The module docstring of the hosted sources now says that the gateway keeps the records until Nerve reads them, also while Nerve is not running, as the sources docs put it.
The lines added for keeping cursors alive and for the polling cursor test used timezone.utc, which the linter asks to write as UTC; the cron service already imports it that way. The lint count of these changes is back to zero.
The publish that negotiation starts shared the two-second bound of a reload's republish. Nothing waits for that publish, and nothing sends the frame again before the next reload or stream, so a stream that could not send for longer than that, and then recovered, never told the gateway what the agent subscribes to. The gateway then counts the agent as subscribed to nothing. The bound now applies only when the caller asks for it, as the reload's republish does. The publish after negotiation waits for as long as the stream is open, and ends with it. Retrying it instead would not do: the frame is one-way, so a retry cannot tell whether a write that its bound cut short went out after all, and each retry gives up the frame's place in the send queue. Each frame is now built once the stream is free to send it, so a frame that waited carries the settings current when it leaves, and a reload that gave up on the stalled stream still reaches the gateway. The request ID is still taken at the call, so frames leave in call order.
The settings builder turned the gh-based events source on whenever the GitHub switch was on, in either GitHub mode. The installer's full settings write, which the unattended init also makes when gh is on the PATH, therefore switched it on over settings that say hosted, where the source fails on every run without a gh login. The Sources step, which leaves that key alone in hosted mode, could then not switch it off. The setup choices now carry the GitHub mode, and the builder keeps the events source off while it is hosted. The web steps carry the mode of the running config. The installer asks nothing about it and takes the mode that its settings file states, which is the layer of the key that the installer keeps. The Sources step names the events source in either mode again, so with hosted GitHub its switch turns the source off both ways.
The fake gateway's default notification record lacked the description section of the contract's notification sample. It now has it, so the defaults give that sample's record byte for byte. The glob vectors that the gateway shares are also checked through a deny list, which drops a value exactly when its pattern matches, so a mistake on the deny side of the dialect fails them too.
Brings in the forecasts of each job's next run and the next-tick API, the catch-up after a pause of the machine, and the activity reporter, up to fe51a4c. The cron service now stops in one sequence. The hosted source deliveries had a stop flag and a stop step of their own; they now use the service's flag, which stop() sets before anything else, so a delivery that arrives or a drain that ends while the service stops starts nothing, just as the scheduler (paused first), the clock check, the catch-ups and the schedule listeners do nothing more. stop() puts the pending and the started delivery runs into the same set as the clock watcher, the catch-ups and the listener tasks, cancels them together, waits for them within the same bound, and then shuts the scheduler down. Stopping a second time does nothing, deliveries included. A scheduled fire is still skipped while a catch-up or a delivery run of its job goes on, and a job with lock still waits for the delivery run. A delivery run still tells the schedule listeners when it ends. The forecast of a messages gate takes a cursor as it will be at the run, which holds with the renewal on every check: the check reads a cursor before it renews it.
The cron service tracked the runs that hosted source deliveries start apart from the rest: a count of each job's runs in flight with an event for when none is left, and a set of the jobs a delivery run was going for, which skipped a fire of the instance's own. The service already records every run in progress, with how it was started, and decides from that whether a new run may start. Delivery runs now count there too, as runs started for delivered records. So a catch-up is skipped while a delivery run of its job goes on, as it is while any run goes on, also for a job with lock, and a catch-up queued before a delivery run does not run the job again once that run has happened. A run by hand starts while a delivery run goes on, as it does in every other case. The delivery drain counts as a run of the source runner, so the runner's own fire meanwhile is skipped. A scheduled fire of a job is still skipped while a delivery run goes on, and still waits for it when the job has lock, now as one of the rules that decide whether a run may start.
The test of a scheduled fire during a delivery run awaited that fire without a bound. Had the fire not been skipped, it would have waited on the run that the test holds, and the test would hang instead of failing.
alex-clickhouse
added this pull request to stack #540
October 21, 2026 14:46
alex-clickhouse
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hosted GitHub source: take GitHub records off the channel streams instead of polling
Stacked on #538 (activity reporter). In hosted mode the platform behind the gateway polls GitHub for the agent and delivers records over the same hosted stream that hosted Slack uses; Nerve stops polling GitHub itself.
sync.github.mode: poll | hosted(defaultpoll, restart-only),sync.github.pr_watch,cron.delivery_debounce_seconds.Event.record(typed source record with metadata entries) for thesource_recordkind; a one-waysubscriptionsframe that publishes this agent's GitHub filters andpr_watchright after negotiation and on every configuration reload ({"version":"1"}when GitHub is disabled = subscribes to nothing). Filter semantics are Nerve's own (fnmatch, deny wins, allow fails closed), with shared test vectors; PR-activity records are matched against the repository rules only.githubwrites each record to the channel buffer and acknowledges only after the durable write; thegithubsource drains the buffer with the existing runner (guardrails, persistence, condensing, drop hook, health,/api/sources/*and the inbox UI unchanged), sorun_if: [{type: messages, sources: [github]}]keeps working.CronService.on_source_delivered(source): drains the runner at once, then runs the jobs gated on that source once per burst after the debounce; never overlaps a scheduled run; cancelled cleanly bystop().github_pr_activitygate passes withoutgh(the platform watches the PRs);nerve doctorand the setup page stop asking forghwhen nothing needs it; the setup writer andnerve initkeep thegh-basedgithub_eventssource off while GitHub is hosted.subscriptionsframe after negotiation waits out a stalled stream (only the reload republish is time-bounded), and a frame that waited carries the settings from after any reload in that window. Amessagesgate keeps its live cursors renewed while it checks them.docs/sources.md(Hosted GitHub source, switching back topollsafely),docs/cron.md,docs/config.md.Rollout: the gateway must know the
subscriptionsframe before an agent turns hosted mode on (an unknown frame closes the stream). Known limitation (documented,nerve doctorwarns): with hosted Slack configured but switched off, hosted GitHub records wait until it is switched on again.Tests: 122 new (contract, shared glob vectors, fake gateway with record events + captured subscriptions, end-to-end event → buffer → runner → inbox → gated job, setup writer/
nerve init, stalled first frame). Full suite 6,952 passed, 39 skipped; vitest 999; ruff and lint unchanged.🤖 Generated with Claude Code