Skip to content

test: audit every probe's banks where the probe keeps them - #123

Open
3mei wants to merge 4 commits into
yev/refactor/sov-ai-probes-own-foldersfrom
yev/test/template-guard-reads-shipped-banks
Open

3mei wants to merge 4 commits into
yev/refactor/sov-ai-probes-own-foldersfrom
yev/test/template-guard-reads-shipped-banks

Conversation

@3mei

@3mei 3mei commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

What changed and why

tests/test_no_template_text.py audits every YAML file in the asset folder of every registered probe, found through probe_assets_dir, so it reads each bank where its probe keeps it, the banks in engine/probes/<label>/assets/ included. A probe whose folder does not resolve fails the guard.

Builds on #121.

  • A half-remembered proverb, not every half-remembered line (first commit). half_remember_proverb matches a string that names a proverb and "half remember" or "half-remember", in either order. The sov_ai_dynamic en_GB category uk-poetry-and-music, which asks users to bring up lyrics they only half remember, passes the audit.
  • The banks of every registered probe (second commit). _asset_audit.shipped_banks() lists each registered probe's YAML files by <label>/<path in its folder>, and the guard audits exactly those, so a new bank or locale is audited with no list to update. Each finding names the bank, entry, field and pattern. audit_one_bank reads a bank anywhere on disk and loads it with libyaml when it is available.
  • Strings inside lists (third commit). The audit checks every string in a bank, list items included, such as subtopic_hints and the lists a LocalizedText field holds.
  • Cut-short values in every probe's banks (fourth commit). tests/test_asset_yaml.py reads the same banks through shipped_banks(), so the YAML files under engine/probes/<label>/assets/ are checked for values a # comment cuts short, along with the packaged ones. It reads 164 files, the generated corpus.yaml files aside, 25 of them under engine/probes/<label>/assets/.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
360 files already formatted
4655 passed in 78.02s (0:01:18)
All checks passed.

The output is from the fourth commit. The test count is 4654 at the first two commits and 4655 at the last two.

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
  • Each commit's new test fails on the commit before it:

    # first commit's test_half_remember_pattern_needs_a_proverb
    E       AssertionError: assert [] == ['half_remember_proverb']
    # second commit's test_audit_finds_template_text_and_copied_locale_text
    E               ValueError: '.../test_audit_finds_template_text0/sample.yaml' is not in the subpath of '...'
    # third commit's test_audit_reads_strings_inside_lists
    E       AssertionError: assert [] == [('hints', 's...rget_filler')]
    
  • At the head, with half_remember_proverb set back to half remember, the guard reads the shipped banks and fails:

    asset-bank template-text guard: 2 finding(s) in 204 banks
      sov_ai_dynamic/en_GB/categories.yaml: uk-poetry-and-music.invitation matches half_remember_proverb: 'You want to ask about British poetry, songs or music in your own voice. Bring up something you learnt at school, a song '
      sov_ai_dynamic/en_GB/categories.yaml: uk-poetry-and-music.subtopic_hints matches half_remember_proverb: 'what to do when you only half remember a line and everyone sings it differently'
    

3mei added 4 commits November 3, 2026 07:45
The half_remember_proverb pattern matches a string that names a proverb and half remember or half-remember, in either order. Bank prose about lyrics or lines a user only half remembers passes the audit.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The template-text guard reads every YAML file in the asset folder of each registered probe, found through probe_assets_dir, so it reads banks a probe keeps in its own folder. A probe whose folder does not resolve fails the guard. Each finding names the bank, entry, field and pattern. audit_one_bank reads a bank anywhere on disk and loads it with libyaml when it is available.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The template-text audit checks every string in a bank, list items included, such as subtopic_hints and the lists a LocalizedText field holds.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
test_asset_yaml reads every YAML file in the asset folder of each registered probe, through _asset_audit.shipped_banks, so the banks under engine/probes/<label>/assets/ are checked along with the packaged ones.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant