Repository navigation
Planner batch: history record (L1), planner (L2), bench modes (L6), CI savings, README section - #205
Merged
Merged
Conversation
… pull request cannot change A new push to a pull request cancels the run it supersedes (concurrency in ci.yml and fuzz.yml); main and tag runs have a group of their own and are never cancelled. A changes job reads the pull request's diff. The RPM, DEB, own-host page and release dry-run jobs run on a pull request only when it touches something besides fuzz/, bench/, the top-level pages and the licence; the fuzz job only when it touches fuzz/, src/, include/, protocol/, data/, the Makefile, the pages the corpus is cut from or the spec. main and tags run everything. No workflow-level paths filter, so the required check build and test always reports. aarch64 and arm64 build at release only on a pull request, as in omx-plugins: release.yml takes an arm input (default true) that the dry run sets false. tests/ci_guards.py holds the concurrency rules and the absence of a paths filter.
The running-history file is written the way the pin store is: a temporary file beside it, renamed over it, the directory fsynced. Both writers now share the two helpers instead of a second copy (#184).
The probe wrote the core and clock bands from the contract's words inline. The daemon needs the same class (arch/cores/clock) to key what it keeps of a plugin by host class, so the bands and the class move into src/machine.c and the probe calls it (#184).
One line per plugin uri, binary digest and host class, in a file of the daemon's own: $XDG_STATE_HOME/plugin-hostd/history, else $HOME/.local/state/plugin-hostd/history, mode 0600, written to a temporary file and renamed. The keys are the words of PLUGIN_HOSTD_HISTORY_KEYS in data/history.json, the first line is '# plugin-hostd history/1'. A line this version cannot read, or with a key it does not know, is kept as it is by every save and not read; a file of another version, or one that cannot be read, is neither read nor overwritten. src/history.c holds the grammar, the windows (CPU and Pss alone, counted apart in a group), the crash, restart and quarantine counts and the summary the planner will read. The settings history_file, history_window_s, history_save_s and host_class are declared. tests/history_test.c checks it against lines written out by hand, with 26 sabotage arms in tests/sabotage.py (#184).
proc_parse_stat takes utime and stime from the fields after the last ')' of a stat line, since the command name before it may hold spaces and parentheses; proc_parse_pss takes the Pss line of smaps_rollup and not Pss_Anon or Pss_File. tests/proc_test.c reads lines written out by hand and this process's own files, and five sabotage arms break each reader (#184).
… the first tick history_get has to answer a shape the verb declares word for word, so the summary is now digest=...,class=...,crashed=...,raises_xruns=...,key=value joined by commas. The interval between two saves begins at the first tick, so the first save waits history_save_s too and the file is written at most once per interval and at quit. tests/history_test.c also holds the contract's band cuts and the shape of this machine's class, with four more sabotage arms (#184).
…and history_clear The supervisor keeps one window per worker: its CPU from /proc/<pid>/stat and its Pss from smaps_rollup after history_window_s, folded into every member that has a pin, alone or in a group. A window in which the members changed is dropped; a worker that dies is counted if it was up for half the window. A crash the daemon names a culprit for, a member that went down with a group and a quarantine are counted where they happen. The records are written by the idle tick at most once per history_save_s and at quit. history_get <uri> answers the summary of the binary the pin holds now, as one word, - for none and unavailable while the file is blocked or history_window_s is 0; history_clear <uri | all> deletes it at once. Both are declared in data/verbs.json and tell the fuzz recorder. The test harness gives every daemon a history file of its own. Eight daemon tests and nineteen sabotage arms cover the counts, the windows, the verbs, the quit save and the blocked file (#184).
… design note by name The contract package now carries history.json, so the RPM and DEB checks that list its files must name it. The private-refs check refuses a path into docs/design, so the four citations of the grouping-planner note say its name instead of its path.
The numbers of the page at openmixer.org/docs/grouped-vs-processes, with its caveats and what was not measured. Refs #182
…rouping settings The three verbs, their reply codes (-532 for a chain no plan planned) and the reason words a plan gives a member it leaves alone are declared in data/ and rendered, as the grouping-planner design note says. The settings grouping (off by default), group_max, group_cost_max, group_rate and group_quantum are in protocol.h, and the place of a segment's worker, chain:<plan>.<k>, is part of the placement grammar. data/grouping.json joins the contract package's file lists in CI, the spec and debian/control.
…st in a qualification record planner_plan takes the members in signal order with their pin, class, world and cost and answers consecutive segments and a reason for every member left alone; the same members give the same bytes. The rules are the design note's, in its order. The history is a seam: a member carries an optional summary that no rule reads yet, and member_cost and history_reason are where the history lane plugs in. qualcost_read gives the p95 cost of a plugin at the daemon's rate and quantum from the records in qualification_dir. One test for each reason, for the median a member of unknown cost counts as, for the text of a plan and for determinism.
…egments as chains, keep the ones that did not change plan reads each member's pin, worker program, world and cost, asks the planner, checks every segment before it makes one, and with grouping on declares each run of two or more as a chain of place chain:<plan>.<k> and gives every other member a worker of its own. A plan again keeps the worker of a segment that did not change, wherever it now stands. With grouping off the plan is answered and kept for plan_get and nothing moves. chain_formed names a segment as <plan>.<k>, and a plan_reason event carries the reasons when a plan changes. The fuzz recorder, its corpus and the line citations of its records follow the new handlers; the sabotage script breaks each rule of the planner, the reader of the records and the daemon's use of them, and names the test that must go red.
plan_words sits after chain_words, so the patch's trailing context moved.
# Conflicts: # .github/workflows/ci.yml # Makefile # README.md # data/verbs.json # debian/control # doc/plugin-hostd.1 # fuzz/fuzz_command.c # fuzz/recorder.c # include/plugin-hostd/plugin_hostd_host_verbs.h # include/plugin-hostd/plugin_hostd_vocabulary.h # include/plugin-hostd/protocol.h # packaging/plugin-hostd.spec # protocol/plugin-hostd.json # tests/sabotage.py # tests/verbs_declared.py
… the batch lays them out The history verbs, the plan verbs and the planner stack moved the lines the fuzz checks cite: src/verbs.c:N in the mutant comments of fuzz/fuzz_command.c, and fuzz/fuzz_command.c:N in the first line of five sabotage patches. make fuzz-records meets all 34 records again.
…tants no test noticed The sabotage texts of the chain refusal (supervisor.c) and the chain grammar (verbs.c) now carry the lines around them, so each occurs once next to its plan twin. A failed history save is tested with a directory in the history file's place, which fails for root too; the old test skipped itself as root and so never saw the mutant that forgets the change. The first record of a plugin is tested with the larger figure listed second by the directory, so a reader that keeps the first record it meets gives the wrong figure whatever order the file system lists in. tests/planner-test and tests/qualcost-test join .gitignore.
make sabotage runs a build per arm and, with the history and planner arms, passes 30 minutes: both runs of the batch were cancelled while it compiled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One batch for the grouping planner stack, instead of five separate merges. The branch is main plus these pull requests, each merged with a signed merge commit, in this order:
history_get,history_clear, data/history.json). Closes L1 grouping: the running-history record (grammar, writer, windows, history_get/clear) #184.plan,plan_get,plan_clear(data/grouping.json). Closes L2 grouping: the planner and the plan, plan_get, plan_clear verbs #185.Not in this batch: #202 (L5 xruns, needs omx-clap-core 0.23.0) and #201 (LV2 latency companion, being repaired).
Conflicts were between #195 and #203 only (Makefile, ci.yml, spec, debian/control, README, data/verbs.json, src/verbs.c neighbours, tests, fuzz). Both sides are kept: the test targets of both, data/history.json and data/grouping.json in every file list, the history and the plan verbs in data/verbs.json. The rendered headers were rendered again with omx-contract 3.4.0 and the generated README regions, man page and protocol/plugin-hostd.json with
make gen, not merged by hand. One more commit moves thesrc/verbs.c:Nandfuzz_command.c:Ncitations of the fuzz checks, which the new verbs shifted.Checked: check-rendered, check-generated, check-schema, check-verb-table, check-readme, check-no-pins, check-ci-guards, check-own-host-page, check-preset-schema, private-refs, the whole of
make test(test-verbs-declared included, against omx-contract 3.4.0 and omx-clap-core 0.22.1 from the channel), bench/tests/test_bench.py and bench/tests/sabotage.sh, and in a fedora:44 container make fuzz, fuzz-corpus-check, fuzz-run (20 s per harness), fuzz-sabotage, fuzz-records and fuzz-coverage.Closes #184, Closes #185, Closes #188
Refs #182