Repository navigation
feat(cache)!: epg cache says what a cache holds - #33
Merged
Merged
Conversation
Since 0.7.0 an entry is kept as its codec's bytes, so a cache.sqlite row is
a blob and ad-hoc SQL reads nothing worth reading. `epg cache` reads a cache
the way a run does, through its driver: `show <site> [channel] [day]` writes
what is cached as XMLTV, or a record per line with `--format json`; `stats
[site]` counts channels, days, programmes and what a run would not use;
`state <site> [group] [--keys]` shows what a site remembers. Looking changes
nothing. `epg prune` is `epg cache prune` now.
Every driver lists what it holds: `list({ site, channelId, kind })`, an
async iterable of keys, which a driver may over-answer and
`CacheManager.list` narrows exactly. `inspectMetas`, `inspect` and
`inspectState` judge an entry or a group as a run does and leave it where it
is, saying why by `status`: 'readable', 'unreadable', 'incompatible',
'invalidated' or 'corrupt'. `read` is `inspect` with removal, as `getState`
already was.
A codec says an entry is damaged by throwing `CorruptEntryError`; a run
removes it, says so as a warning of its site's, and the next run grabs the
day again. Anything else a codec throws is thrown, so a codec's own fault
cannot empty the cache and publish an empty guide. `decode(body, meta)` is
handed the entry's meta, and the codecs of `CODECS` hold an entry to the
count it was written with, which is how a cut that still parses is told
from a whole one. `instanceof CorruptEntryError` knows one from another copy
of the package by its name.
A look survives what a run would fail on: an entry this user may not read
is said on stderr and counted as failed, and the rest is read. Stats streams
the listing through one pool, so neither a large cache nor many small sites
are read a key or a site at a time. The sqlite driver opens nothing until
asked, and a read of a missing or other-version file makes nothing; its
listing pages, so no statement is held across a yield. The fs driver lists
and prunes through links, and passes over an unreadable directory only
beside the sites. A grab leaves out a programme without a valid start,
which no codec could read back, with a warning.
BREAKING CHANGE: `CacheDriver.list()` is required, so a driver of your own
must list what it holds. `CacheCodec.decode` is handed the entry's meta, and
the codecs of `CODECS` throw `CorruptEntryError` for what they cannot read —
the json one answered `[]` before. `epg prune` is `epg cache prune`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Since 0.7.0 a cache entry is stored as its codec's bytes, so a
cache.sqliterow is a blob and ad-hoc SQL shows nothing readable. This PR addsepg cache, which reads a cache through its driver the same way a run does, whichever driver that is. Looking never changes the cache.Breaking changes
CacheDriver.list()is required. A driver of your own has to list what it holds. A driver without it gets an error that points at the "A driver of your own" section indocs/api.md.CacheCodec.decode(body, meta)is now handed the entry's meta. A codec written with one parameter still type-checks.CODECSthrowCorruptEntryErrorfor bytes they can't read, with the parser's error ascause. Before,jsonreturned[]and the other two threw the parser's own error.epg pruneis nowepg cache prune. The old command exits 2 with "it is epg cache prune now".What's added
Listing and inspecting
list({ site?, channelId?, kind? })on every driver: an async iterable of keys,{ kind: 'entry', key }or{ kind: 'state', site, key }. A driver may return more than it was asked for;CacheManager.listfilters to exactly what was asked.inspectMetas,inspectandinspectStateonCacheManager. Each judges an entry or state group the way a run does, but removes nothing. The result'sstatussays why something can't be used:statusreadableunreadableincompatiblemeta.schemasays which)invalidatedinvalidatevoids itcorrupterrorsays why)readis nowinspectplus removal, the same waygetStatealready worked.Corrupt entries
CorruptEntryError, exported fromepg-tools/cacheand the root, is how a codec says an entry is damaged.site:warningfor that entry's site, and the next run grabs the day again. The message comes through the newCacheManagerOptions.warn(message, key).instanceof CorruptEntryErroralso recognises one thrown from another copy of the package, viastatic [Symbol.hasInstance]. On a subclass it is the ordinary prototype check.CODECSthrow it only for what their parser rejects:SyntaxError,XmltvDateError, and a new internalXmltvInputError(aTypeErrorsubclass for the scanner's "not an XMLTV document" errors). They also throw it when the number of programmes read differs from the count the entry was written with. That catches an ndjson entry cut between lines and an xmltv document that ends early.A look survives what a run would fail on
statscounts it asfailed;showandstateleave it out and carry on.statscountsunreadable,invalidatedandfailedseparately, and makes no promise about what the next run grabs.statedescribes a bag as, for example, "12 keys, and 2 expired and 1 unreadable". It makes no claim about which validators a run would drop, since that depends on the run's window.Other changes
listreads in pages of 1,000 keys, so no statement stays open across ayield; one that did kept writes uncommitted.close()..Trash-0) is skipped when it can't be read. Inside a site, an unreadable channel directory is an error.lost+found, are left alone.statsstreams the listing through one worker pool for the whole cache. Memory grows with the number of sites rather than keys, and many small sites are read in parallel.inParallel(now insrc/core/parallel.ts) accepts async iterables and closes the source on failure.start, with a site warning. No codec can write one in a form it can read back, so it would otherwise make the entry corrupt, and every run would remove it and grab it again.channel=""that it wrote.epg cacherefuses any flag its subcommand would ignore, from a per-subcommand allow-list.count()helper insrc/core/count.tsreplaces the hand-written plurals inplan,reporters,runandtry.Testing
test/cache-command.test.ts: the three looks, statuses, failures and aborts, both store shapes, streaming;test/parallel.test.ts.test/cache.test.ts: listing across all four drivers, inspection, corrupt entries and codecs, sqlite laziness and paging, fs links and permissions;test/cli.test.ts: a sqlite run end to end;test/grabber.test.ts;test/build.test.ts.Not in this PR
channelListIn,bagEntriesInandisRememberedstay exported from the grabber for the CLI to use, rather than moving behindCacheManager. That is left for a later decision.