Skip to content

feat(grabber)!: a run says what its requests were for - #34

Merged
Ruby184 merged 1 commit into
mainfrom
feat/request-labels
Oct 24, 2026
Merged

Ruby184 merged 1 commit into
mainfrom
feat/request-labels

Conversation

@Ruby184

@Ruby184 Ruby184 commented Oct 24, 2026

Copy link
Copy Markdown
Owner

A run says what its requests were for

"4469 requests" did not say which. site:done and grab:done now carry requestsBy. For each label it gives:

  • how many requests went;
  • how long they waited on the source;
  • what failed them: by status, timeout, network, or error for anything else.

A 404 the site goes on to catch is counted too. The text reporter lists the labels on a site's line when it had more than one kind:

[skylink.sk] 144 fetched — 5291 requests (request 49, detail 5242: 3242 × 404 and 3 × timeout) in 11m37s, 585s waiting

Labels

What Label
A site's planned requests request, or the site's requestLabel
A stream's fetches the site's requestLabel, or stream
A parse's paced(task, { label }) its label, or paced
A fetched channel list channels
Schedules Direct lineup calls under channels, then md5, schedules, programs
  • A label is a word with a letter in it, or words joined by ., _, / or -.
  • One that isn't throws where paced is called, and the site is warned once per kind of mistake.
  • A site using more than 64 labels is warned once, and every label is still counted.

Sites in threads

  • Counts in shared memory. Each site gets a SharedArrayBuffer. Its channel-day counts sit in fixed slots and each label's counts in slots named once, so counting sends no messages. Measured with 10,050 one-at-a-time requests: 74 ms of main-thread time, against 958 ms when each count was sent as a message.
  • A thread that dies. Its site is decided once the thread's port has closed, which comes after everything the thread sent, from what the site counted in memory. It gets one site:failed and one site:done, and none is made up for a site that already said its own. A port that never closes is decided after 5 seconds.
  • Late requests. A request a parse didn't wait for is counted until the run hangs up, the same on either thread.
  • Cache and config. A thread's cache answers everything it was asked before it lets go. A worker forgets a config that failed to load, so the next site asks again.

Reporters that throw

A reporter's throw no longer ends a site part way, or crashes the process from a thread's listener.

  • What they reject with. grab, runGrab, build, runMerge and guideStream say everything they did, then reject with a new ReporterError. It's an AggregateError holding each kind of throw once, with how often and at which events (thrown), at most ten kinds.
  • When the run failed too. It rejects with a new RunFailedError: error is the run's own failure, reporter the ReporterError. The CLI exits on the run's own error and prints both.
  • combineReporters tells every reporter whatever another throws, and keeps what a reporter written async rejects with.

Also

  • A site with no name is named a site with no name (n) in its events, and build fetches nothing for it.
  • A fault inside the package fails the grab without saying grab:done.
  • A cancelled run waits for its in-process sites to wind down, as it already did for threaded ones.
  • An error a site throws that can't be read is reported as such, instead of failing the reporter.

Breaking changes

  • requestsBy is required on the site:done and grab:done event types.
  • A reporter that throws makes grab, runGrab, build, runMerge and guideStream reject with a ReporterError (or a RunFailedError) once they're done. Before, the throw failed a site or the run part way. A build whose reporter throws only at its merge writes the guide, then rejects.
  • An undici error carrying a 4xx statusCode now counts as the source's answer, so a channel list that fails with one no longer falls back to the stored list.

New exports: ReporterError, RunFailedError and the ReporterThrow type, from epg-tools and epg-tools/grabber.

Testing

npm run lint, npm run typecheck, npm run build and the suite pass on Node 22.18, 24.15 and 26.12 (2,211 passed, 2 skipped). The new logic was mutation-checked: each change, broken on purpose, fails a test.

"4469 requests" did not say which. `site:done` and `grab:done` now carry
`requestsBy`: for each label, how many requests went, how long they waited
on the source, and what failed them — by status, `timeout`, `network`, or
`error` for anything else, a 404 the site goes on to catch included. The
text reporter lists them on a site's line where it had more than one kind:
`5291 requests (request 49, detail 5242: 3242 × 404 and 3 × timeout)`.

A label names a kind of request. A site's planned requests are `request`
unless its `requestLabel` says otherwise, and a stream's fetches its
`requestLabel` or `stream`; a parse's `paced(task, { label })` calls are
`paced` without one; a fetched channel list is `channels`. Schedules Direct
names its own: its lineup calls under `channels`, then `md5`, `schedules`
and `programs`. A label is a word with a letter in it, or words joined by
`.`, `_`, `/` or `-`: one that is not throws where `paced` is called, and is
warned of once per kind of mistake. A site whose requests come under more
than 64 labels is warned once, and every label is still counted.

A site in a thread counts in memory its thread shares with the run's: a
`SharedArrayBuffer` per site, its channel-days in fixed slots and each
label's counts in slots named once, so the counting crosses no messages. A
thread that dies is decided once its port has closed — after everything it
sent — from that memory: `site:failed` and `site:done`, once, or no
`site:done` made up for a site that said its own; one that never closes is
decided after five seconds. A request a parse did not wait for is counted
until the run hangs up, on either thread. A thread's cache answers what it
was asked before it lets go, and a worker forgets a config that failed to
load, so it is asked again.

A reporter that throws no longer ends a site part way, or the process from a
thread's listener. Every entry point — grab, runGrab, build, runMerge,
guideStream — says all it did and then rejects with a `ReporterError`: each
kind of throw once, how often and at which events, ten kinds kept. A run
that failed of itself as well rejects with a `RunFailedError` of both.
`combineReporters` tells every reporter whatever another throws, and keeps
what one written async rejects with.

Also: a site with no name is named `a site with no name (n)` in its events,
and build fetches nothing for it; a fault in the package fails the grab
without a `grab:done`; a cancelled run waits for its sites on this thread as
it did for those in threads; an error a site throws that cannot be read is
said as such.

BREAKING CHANGE: `requestsBy` is required on the `site:done` and `grab:done`
event types. A reporter that throws makes `grab`, `runGrab`, `build`,
`runMerge` and `guideStream` reject with a `ReporterError` (or a
`RunFailedError`) once they are done, where its throw used to fail a site
or the run part way; a build whose reporter throws only at its merge writes
the guide, then rejects. An undici error carrying a 4xx `statusCode` is the
source's answer: a channel list it fails no longer falls back to the stored
one.
@Ruby184
Ruby184 merged commit cef3ea3 into main Oct 24, 2026
4 checks passed
@Ruby184
Ruby184 deleted the feat/request-labels branch October 24, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant