Skip to content

feat(cache)!: epg cache says what a cache holds - #33

Merged
Ruby184 merged 1 commit into
mainfrom
feat/cache-inspect
Oct 19, 2026
Merged

Ruby184 merged 1 commit into
mainfrom
feat/cache-inspect

Conversation

@Ruby184

@Ruby184 Ruby184 commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Summary

Since 0.7.0 a cache entry is stored as its codec's bytes, so a cache.sqlite row is a blob and ad-hoc SQL shows nothing readable. This PR adds epg cache, which reads a cache through its driver the same way a run does, whichever driver that is. Looking never changes the cache.

epg cache stats                                  # how much is cached, per site
epg cache show example.tv                        # a site's programmes, as XMLTV
epg cache show example.tv one.example.tv 2026-07-17 --format json  # one day, a record per line
epg cache state example.tv                       # the groups a site remembers
epg cache state example.tv validators --keys     # only one group's keys
epg cache prune --before 2026-07-01              # was `epg prune`

Breaking changes

  • CacheDriver.list() is required. A driver of your own has to list what it holds. A driver without it gets an error that points at the "A driver of your own" section in docs/api.md.
  • CacheCodec.decode(body, meta) is now handed the entry's meta. A codec written with one parameter still type-checks.
  • The codecs in CODECS throw CorruptEntryError for bytes they can't read, with the parser's error as cause. Before, json returned [] and the other two threw the parser's own error.
  • epg prune is now epg cache prune. The old command exits 2 with "it is epg cache prune now".

What's added

Listing and inspecting

  • list({ site?, channelId?, kind? }) on every driver: an async iterable of keys, { kind: 'entry', key } or { kind: 'state', site, key }. A driver may return more than it was asked for; CacheManager.list filters to exactly what was asked.

  • inspectMetas, inspect and inspectState on CacheManager. Each judges an entry or state group the way a run does, but removes nothing. The result's status says why something can't be used:

    status meaning
    readable what a run would use
    unreadable the meta can't be read
    incompatible written with another schema (meta.schema says which)
    invalidated the config's invalidate voids it
    corrupt the meta reads but the programmes don't (error says why)
  • read is now inspect plus removal, the same way getState already worked.

Corrupt entries

  • CorruptEntryError, exported from epg-tools/cache and the root, is how a codec says an entry is damaged.
    • A run removes the entry, emits a site:warning for that entry's site, and the next run grabs the day again. The message comes through the new CacheManagerOptions.warn(message, key).
    • Anything else a codec throws is rethrown and the entry is kept. A bug in a codec, or a format changed without a schema bump, must not empty the cache and publish an empty guide.
    • instanceof CorruptEntryError also recognises one thrown from another copy of the package, via static [Symbol.hasInstance]. On a subclass it is the ordinary prototype check.
  • The codecs in CODECS throw it only for what their parser rejects: SyntaxError, XmltvDateError, and a new internal XmltvInputError (a TypeError subclass for the scanner's "not an XMLTV document" errors). They also throw it when the number of programmes read differs from the count the entry was written with. That catches an ndjson entry cut between lines and an xmltv document that ends early.

A look survives what a run would fail on

  • An entry this user may not read (EACCES) is reported on stderr. stats counts it as failed; show and state leave it out and carry on.
  • stats counts unreadable, invalidated and failed separately, and makes no promise about what the next run grabs.
  • state describes a bag as, for example, "12 keys, and 2 expired and 1 unreadable". It makes no claim about which validators a run would drop, since that depends on the run's window.

Other changes

  • sqlite driver
    • Opens nothing until something is asked of it.
    • A read of a missing file, or of tables written by another version, answers empty and creates nothing.
    • The first change switches the file to WAL.
    • A location that could never be created is refused when the driver is made, before a run fetches anything.
    • list reads in pages of 1,000 keys, so no statement stays open across a yield; one that did kept writes uncommitted.
    • It stays closed after close().
  • fs drivers
    • List and prune follow symlinks. A link is never removed, only pruned through.
    • Only a directory beside the sites (a mount's .Trash-0) is skipped when it can't be read. Inside a site, an unreadable channel directory is an error.
    • Names the driver could never have created, such as lost+found, are left alone.
  • stats streams the listing through one worker pool for the whole cache. Memory grows with the number of sites rather than keys, and many small sites are read in parallel. inParallel (now in src/core/parallel.ts) accepts async iterables and closes the source on failure.
  • The grab leaves out a programme without a valid start, with a site warning. No codec can write one in a form it can read back, so it would otherwise make the entry corrupt, and every run would remove it and grab it again.
  • The xmltv codec reads back a programme with channel="" that it wrote.
  • epg cache refuses any flag its subcommand would ignore, from a per-subcommand allow-list.
  • One shared count() helper in src/core/count.ts replaces the hand-written plurals in plan, reporters, run and try.

Testing

  • New suites:
    • test/cache-command.test.ts: the three looks, statuses, failures and aborts, both store shapes, streaming;
    • test/parallel.test.ts.
  • Existing suites, extended:
    • test/cache.test.ts: listing across all four drivers, inspection, corrupt entries and codecs, sqlite laziness and paging, fs links and permissions;
    • test/cli.test.ts: a sqlite run end to end;
    • test/grabber.test.ts;
    • test/build.test.ts.
  • Mutation checks were run on every round of changes. All mutants were killed except a few equivalent ones.
  • Full suite passes on Node 22.18, 24.15 and 26.12: 2,077 passed, 2 skipped.

Not in this PR

  • Where the state-group rules live: channelListIn, bagEntriesIn and isRemembered stay exported from the grabber for the CLI to use, rather than moving behind CacheManager. That is left for a later decision.

Since 0.7.0 an entry is kept as its codec's bytes, so a cache.sqlite row is
a blob and ad-hoc SQL reads nothing worth reading. `epg cache` reads a cache
the way a run does, through its driver: `show <site> [channel] [day]` writes
what is cached as XMLTV, or a record per line with `--format json`; `stats
[site]` counts channels, days, programmes and what a run would not use;
`state <site> [group] [--keys]` shows what a site remembers. Looking changes
nothing. `epg prune` is `epg cache prune` now.

Every driver lists what it holds: `list({ site, channelId, kind })`, an
async iterable of keys, which a driver may over-answer and
`CacheManager.list` narrows exactly. `inspectMetas`, `inspect` and
`inspectState` judge an entry or a group as a run does and leave it where it
is, saying why by `status`: 'readable', 'unreadable', 'incompatible',
'invalidated' or 'corrupt'. `read` is `inspect` with removal, as `getState`
already was.

A codec says an entry is damaged by throwing `CorruptEntryError`; a run
removes it, says so as a warning of its site's, and the next run grabs the
day again. Anything else a codec throws is thrown, so a codec's own fault
cannot empty the cache and publish an empty guide. `decode(body, meta)` is
handed the entry's meta, and the codecs of `CODECS` hold an entry to the
count it was written with, which is how a cut that still parses is told
from a whole one. `instanceof CorruptEntryError` knows one from another copy
of the package by its name.

A look survives what a run would fail on: an entry this user may not read
is said on stderr and counted as failed, and the rest is read. Stats streams
the listing through one pool, so neither a large cache nor many small sites
are read a key or a site at a time. The sqlite driver opens nothing until
asked, and a read of a missing or other-version file makes nothing; its
listing pages, so no statement is held across a yield. The fs driver lists
and prunes through links, and passes over an unreadable directory only
beside the sites. A grab leaves out a programme without a valid start,
which no codec could read back, with a warning.

BREAKING CHANGE: `CacheDriver.list()` is required, so a driver of your own
must list what it holds. `CacheCodec.decode` is handed the entry's meta, and
the codecs of `CODECS` throw `CorruptEntryError` for what they cannot read —
the json one answered `[]` before. `epg prune` is `epg cache prune`.
@Ruby184
Ruby184 merged commit 9a8d370 into main Oct 19, 2026
4 checks passed
@Ruby184
Ruby184 deleted the feat/cache-inspect branch October 19, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant