Feature Request

Problem / Motivation

The current regression testing setup only validates a single configuration per system, meaning most of the available CLI/configuration options are not exercised in end-to-end tests.

While atomic unit tests already provide good coverage of individual functionality, there is currently no systematic way to ensure that:

• different configuration modes work correctly together
• key user-facing options continue to behave as expected
• changes do not unintentionally affect less commonly used features

As the number of configurable options grows, this increases the risk of regressions in real-world usage scenarios.

Proposed Solution

Introduce a scenario-based regression testing framework that expands coverage across key configuration options in a structured and maintainable way.

1. Scenario-based regression tests

• Replace the current “one config per system” model with:

  • multiple named scenarios per system
  • each scenario defined as a separate YAML config

• Each scenario is executed as an individual pytest test case

Example structure:

tests/regression/
  configs/
    dna/
      default.yaml
      frame_window.yaml
      selection_subset.yaml
      axes_off.yaml
      external_force.yaml
  baselines/
    dna/
      default.json
      frame_window.json
      selection_subset.json
      axes_off.json
      external_force.json

• Each *.yaml → one regression test
• Each scenario has a corresponding baseline JSON

2. Automatic test discovery

• Add a small helper (cases.py) to:

  • discover all scenario YAML files
  • generate pytest parameters dynamically

• Each test is identified as:

  system:scenario (e.g. dna:axes_off)

3. Focus on key configuration coverage

Define a curated set of scenarios that cover important configuration groups:

• Frame handling (start, end, step)
• Atom selection (selection_string)
• Algorithm toggles (combined_forcetorque, customised_axes)
• Feature flags (water_entropy, grouping, search_type)

Avoid testing all combinations — instead use representative scenarios.

4. Hybrid testing approach

Combine two complementary strategies:

A. Baseline-backed regression tests

• For meaningful scenarios (YAML-backed)
• Compare full output against stored JSON baselines

B. Generated “smoke” tests (Python)

• Programmatically generate config variations

• Vaulidate:

  • successful execution
  • valid output structure

• No baseline required

5. Optional: invariance (metamorphic) tests

Add lightweight tests for options that should not affect results, e.g.:

• verbose
• outfile

These compare outputs between equivalent configurations rather than against a baseline.

Alternatives Considered

• Single baseline per system (current approach)
  Too limited; does not cover configuration space

• Full combinatorial testing of all options
  Not scalable; leads to excessive number of baselines

• Only unit tests
  Does not capture integration issues between configuration options

Expected Impact

• Improved confidence in end-to-end behaviour across configuration modes
• Better protection against regressions in less frequently used options
• More granular and debuggable test failures (per scenario)
• Scalable structure for adding new configuration options in the future

Potential trade-offs:

• Slight increase in number of regression tests
• Additional baseline files to maintain (mitigated by keeping scenarios curated)

Additional Context

• Existing test infrastructure already supports:

  • config-driven execution
  • baseline comparison
  • slow test handling

• This proposal builds on top of that rather than replacing it

• The approach follows common patterns used in scientific and CLI-heavy Python projects:

  • data-driven tests (YAML)
  • pytest parametrization
  • hybrid baseline + generated test strategies