Incomplete tests: default arguments

A kind of related to gh-301: I asked GPT5.4 to write a wrapper on top of array-api-strict which, for each function call, looks at the arguments the function is called with, and compares the actual argument with the default value for the corresponding formal parameter. The goal is to find functions where the test suite never generates non-default parameter values.

Then, running the test suite produces the output below.

device arguments are Test device support #302 (and we cannot easily fix it in the test suite without sacrificing CuPy coverage, cf ENH: Add device= arguments to array creation functions cupy/cupy#9848)
the rest could use looking at. The table was recorded from a reasonably short run, so it may very well have false negatives (as in, not all parameter combinations were generated)

 +---------------------------+----------------------------------------------------+
   | function                  | arguments                                          |
   +---------------------------+----------------------------------------------------+
   | Array.__array_namespace__ | api_version=ONLY_DEFAULT_USED                      |
   | arange                    | device=ONLY_DEFAULT_USED                           |
   | asarray                   | device=ONLY_DEFAULT_USED                           |
   | astype                    | device=ONLY_DEFAULT_USED                           |
   | empty_like                | device=ONLY_DEFAULT_USED                           |
   | eye                       | device=ONLY_DEFAULT_USED                           |
   | fft.fftfreq               | device=ONLY_DEFAULT_USED                           |
   | fft.rfftfreq              | device=ONLY_DEFAULT_USED                           |
   | full                      | device=ONLY_DEFAULT_USED                           |
   | full_like                 | device=ONLY_DEFAULT_USED                           |
   | linalg.cholesky           | upper=ONLY_DEFAULT_USED                            |
   | linalg.matrix_norm        | keepdims=ONLY_DEFAULT_USED                         |
   | linalg.trace              | dtype=ONLY_DEFAULT_USED                            |
   | linalg.vector_norm        | axis=ONLY_DEFAULT_USED                             |
   | linspace                  | device=ONLY_DEFAULT_USED                           |
   | ones                      | device=ONLY_DEFAULT_USED                           |
   | ones_like                 | device=ONLY_DEFAULT_USED                           |
   | reshape                   | copy=ONLY_DEFAULT_USED                             |
   | stack                     | axis=ONLY_DEFAULT_USED                             |
   | sum                       | axis=ONLY_DEFAULT_USED, keepdims=ONLY_DEFAULT_USED |
   | tril                      | k=ONLY_DEFAULT_USED                                |
   | triu                      | k=ONLY_DEFAULT_USED                                |
   | zeros                     | device=ONLY_DEFAULT_USED                           |
   | zeros_like                | device=ONLY_DEFAULT_USED                           |
   +---------------------------+----------------------------------------------------+

   SQL:

   SELECT
       CASE
           WHEN function LIKE 'recording_backend.%' THEN substr(function, 
   length('recording_backend.') + 1)
           ELSE function
       END AS function_name,
       group_concat(name || '=' || status, ', ') AS arguments
   FROM (
       SELECT function, name, status
       FROM function_arguments
       WHERE status = 'ONLY_DEFAULT_USED'
       ORDER BY function, name
   ) AS filtered
   GROUP BY function
   ORDER BY function_name;

A slightly longer run (with --max-examples=3_000):

Functions with `default only` parameters

Function names are shown without the recording_backend. prefix.
This report includes only functions with at least one parameter whose status is default only.
Status legend: required = no default, non-default used = optional and used with a non-default value, default only = optional and only observed at default, default passed explicitly = optional and explicitly passed at its default.
Functions listed: 21
Parameters covered: 75

Summary

Function	Params	Required	Non-default used	Default only	Explicit default	Parameters
`Array.__array_namespace__`	2	1	0	1	0	`api_version` (default only); `self` (required)
`arange`	5	1	3	1	0	`device` (default only); `dtype` (non-default used); `start` (required); `step` (non-default used); `stop` (non-default used)
`asarray`	4	1	2	1	0	`copy` (non-default used); `device` (default only); `dtype` (non-default used); `obj` (required)
`astype`	4	2	1	1	0	`copy` (non-default used); `device` (default only); `dtype` (required); `x` (required)
`empty_like`	3	1	1	1	0	`device` (default only); `dtype` (non-default used); `x` (required)
`eye`	5	1	3	1	0	`device` (default only); `dtype` (non-default used); `k` (non-default used); `n_cols` (non-default used); `n_rows` (required)
`fft.fftfreq`	4	1	2	1	0	`d` (non-default used); `device` (default only); `dtype` (non-default used); `n` (required)
`fft.rfftfreq`	4	1	2	1	0	`d` (non-default used); `device` (default only); `dtype` (non-default used); `n` (required)
`full`	4	2	1	1	0	`device` (default only); `dtype` (non-default used); `fill_value` (required); `shape` (required)
`full_like`	4	2	1	1	0	`device` (default only); `dtype` (non-default used); `fill_value` (required); `x` (required)
`linalg.trace`	3	1	1	1	0	`dtype` (default only); `offset` (non-default used); `x` (required)
`linspace`	6	3	2	1	0	`device` (default only); `dtype` (non-default used); `endpoint` (non-default used); `num` (required); `start` (required); `stop` (required)
`ones`	3	1	1	1	0	`device` (default only); `dtype` (non-default used); `shape` (required)
`ones_like`	3	1	1	1	0	`device` (default only); `dtype` (non-default used); `x` (required)
`reshape`	3	2	0	1	0	`copy` (default only); `shape` (required); `x` (required)
`std`	4	1	0	2	1	`axis` (default only); `correction` (default passed explicitly); `keepdims` (default only); `x` (required)
`sum`	4	1	1	2	0	`axis` (default only); `dtype` (non-default used); `keepdims` (default only); `x` (required)
`tril`	2	1	0	1	0	`k` (default only); `x` (required)
`triu`	2	1	0	1	0	`k` (default only); `x` (required)
`zeros`	3	1	1	1	0	`device` (default only); `dtype` (non-default used); `shape` (required)
`zeros_like`	3	1	1	1	0	`device` (default only); `dtype` (non-default used); `x` (required)

Details

`Array.__array_namespace__`

Parameter	Status
`api_version`	default only
`self`	required

`arange`

Parameter	Status
`device`	default only
`dtype`	non-default used
`start`	required
`step`	non-default used
`stop`	non-default used

`asarray`

Parameter	Status
`copy`	non-default used
`device`	default only
`dtype`	non-default used
`obj`	required

`astype`

Parameter	Status
`copy`	non-default used
`device`	default only
`dtype`	required
`x`	required

`empty_like`

Parameter	Status
`device`	default only
`dtype`	non-default used
`x`	required

`eye`

Parameter	Status
`device`	default only
`dtype`	non-default used
`k`	non-default used
`n_cols`	non-default used
`n_rows`	required

`fft.fftfreq`

Parameter	Status
`d`	non-default used
`device`	default only
`dtype`	non-default used
`n`	required

`fft.rfftfreq`

Parameter	Status
`d`	non-default used
`device`	default only
`dtype`	non-default used
`n`	required

`full`

Parameter	Status
`device`	default only
`dtype`	non-default used
`fill_value`	required
`shape`	required

`full_like`

Parameter	Status
`device`	default only
`dtype`	non-default used
`fill_value`	required
`x`	required

`linalg.trace`

Parameter	Status
`dtype`	default only
`offset`	non-default used
`x`	required

`linspace`

Parameter	Status
`device`	default only
`dtype`	non-default used
`endpoint`	non-default used
`num`	required
`start`	required
`stop`	required

`ones`

Parameter	Status
`device`	default only
`dtype`	non-default used
`shape`	required

`ones_like`

Parameter	Status
`device`	default only
`dtype`	non-default used
`x`	required

`reshape`

Parameter	Status
`copy`	default only
`shape`	required
`x`	required

`std`

Parameter	Status
`axis`	default only
`correction`	default passed explicitly
`keepdims`	default only
`x`	required

`sum`

Parameter	Status
`axis`	default only
`dtype`	non-default used
`keepdims`	default only
`x`	required

`tril`

Parameter	Status
`k`	default only
`x`	required

`triu`

Parameter	Status
`k`	default only
`x`	required

`zeros`

Parameter	Status
`device`	default only
`dtype`	non-default used
`shape`	required

`zeros_like`

Parameter	Status
`device`	default only
`dtype`	non-default used
`x`	required

So, to summarise, apart from the device parameter, these are the functions (and their parameter) with missing coverage?

Function	Missing coverage
`sum`	`axis`, `keepdims`
`std`	`axis`, `keepdims`, non-default `correction`
`reshape`	`copy`
`tril`	`k`
`triu`	`k`
`linalg.trace`	`dtype`
`Array.__array_namespace__`	`api_version`

I tried to print the generated argument inputs for sum by adding print(f"kw = {kw}") here

array_api_tests/test_statistical_functions.py::test_sum 
kw = {}
kw = {}
kw = {}
kw = {'axis': (), 'dtype': None}
kw = {}
kw = {}
kw = {'axis': 0, 'keepdims': False}
kw = {}
kw = {'axis': None, 'keepdims': False}
kw = {}
kw = {}
kw = {}
kw = {'axis': None, 'keepdims': False}
kw = {'axis': None, 'keepdims': False}
kw = {'axis': None, 'keepdims': False}
kw = {'dtype': None, 'keepdims': False}
kw = {'dtype': None, 'keepdims': False}
kw = {'axis': -2, 'dtype': array_api_strict.float32, 'keepdims': True}
kw = {'axis': -2, 'dtype': array_api_strict.float32, 'keepdims': True}
kw = {'axis': 0, 'dtype': array_api_strict.float32, 'keepdims': True}
kw = {'axis': 0, 'dtype': array_api_strict.float32, 'keepdims': False}
kw = {'axis': 0, 'dtype': array_api_strict.float32, 'keepdims': False}
kw = {'axis': 0, 'dtype': array_api_strict.float32, 'keepdims': False}
kw = {'axis': 0, 'dtype': array_api_strict.float32, 'keepdims': False}
kw = {'keepdims': False}
kw = {'keepdims': False}
kw = {'keepdims': False}
kw = {'keepdims': False}
kw = {'keepdims': False}
kw = {}
kw = {'axis': None, 'keepdims': False}
kw = {'axis': None, 'keepdims': False}
kw = {'dtype': None}
kw = {'dtype': None}
kw = {'axis': None, 'dtype': array_api_strict.complex128, 'keepdims': False}
kw = {'axis': None, 'dtype': array_api_strict.complex128, 'keepdims': False}
kw = {'axis': None, 'dtype': array_api_strict.complex128, 'keepdims': False}
kw = {'axis': None, 'dtype': array_api_strict.complex128, 'keepdims': False}
kw = {'axis': None, 'dtype': array_api_strict.complex128, 'keepdims': False}
kw = {'axis': None, 'dtype': None}
kw = {'axis': None, 'dtype': None}
kw = {'axis': -2, 'dtype': None}
kw = {'keepdims': False}
kw = {'axis': None}
kw = {'axis': None}
kw = {}
kw = {}
kw = {}
kw = {'dtype': None, 'keepdims': False}
kw = {'axis': None, 'keepdims': False}
kw = {'dtype': array_api_strict.float64, 'keepdims': False}
kw = {'dtype': array_api_strict.float64, 'keepdims': False}
kw = {'dtype': array_api_strict.float64, 'keepdims': False}
kw = {'axis': None}
kw = {'dtype': array_api_strict.float64, 'keepdims': False}
kw = {'dtype': array_api_strict.int64}
kw = {'dtype': array_api_strict.int64}
kw = {'dtype': array_api_strict.int8}
kw = {'dtype': array_api_strict.int8}
kw = {}
kw = {'axis': (1, -2), 'dtype': array_api_strict.complex64, 'keepdims': True}
kw = {'axis': (1, -2), 'dtype': array_api_strict.complex64, 'keepdims': True}
kw = {'axis': (-2,), 'keepdims': True}
kw = {'axis': (-2,), 'keepdims': True}
kw = {'axis': (-2,), 'keepdims': True}
kw = {'axis': (-2,), 'keepdims': True}
kw = {}
kw = {'axis': None}
kw = {'axis': None, 'dtype': None}
kw = {'axis': (-1,), 'dtype': array_api_strict.uint8, 'keepdims': False}
kw = {'axis': None}
kw = {'axis': None}
kw = {'keepdims': False}
kw = {}
kw = {}
kw = {'dtype': array_api_strict.uint8}
kw = {'dtype': array_api_strict.uint8}
kw = {}
kw = {'dtype': None, 'keepdims': True}
kw = {'dtype': None, 'keepdims': False}
kw = {'dtype': None, 'keepdims': False}
kw = {'dtype': None, 'keepdims': False}
kw = {'dtype': None, 'keepdims': False}
kw = {'dtype': None}
kw = {'dtype': None}
kw = {'axis': None}
kw = {'axis': None}
kw = {'dtype': None}
kw = {}
kw = {}
kw = {'dtype': None}
kw = {'keepdims': False}
kw = {'axis': (-1,), 'dtype': array_api_strict.int16}
kw = {'keepdims': True}
kw = {}
kw = {'axis': None, 'dtype': None}
kw = {'axis': None, 'dtype': None}
kw = {'dtype': None}
kw = {'dtype': None}
kw = {'dtype': None, 'keepdims': False}
kw = {}
kw = {}
kw = {}
kw = {}
kw = {'axis': None}

IIUC, non-default inputs are generated here.

The checker is entirely vibe-coded, so yeah, let's treat what it emits as a hint and check manually other function, too.

Here's an analysis on the rest of the functions:

Trimming down the output, std does generate non-default inputs:

kw = {'axis': (), 'correction': 5.694276234840636, 'keepdims': True}
kw = {'axis': (), 'correction': 5.694276234840636, 'keepdims': True}
kw = {'correction': 0.0, 'keepdims': True}
kw = {'correction': 0.0, 'keepdims': True}
kw = {'correction': 0.0, 'keepdims': True}
kw = {'axis': -2, 'correction': 0.14222324635672703}
kw = {'axis': None, 'correction': 0.0}
kw = {'axis': None, 'correction': 0.0}
kw = {'correction': 0.0}
kw = {'correction': 0.0, 'keepdims': True}
kw = {'correction': 0.0, 'keepdims': False}

test_reshape does not account for copy, indeed:

array-api-tests/array_api_tests/test_manipulation_functions.py

Line 432 in 55fcc60

def test_reshape(x, shape):

Apart from the signature test, I can't find dedicated tests for tril and triu

array-api-tests/array_api_tests/test_signatures.py

Line 150 in 55fcc60

matrixy_names += ["__matmul__", "triu", "tril"]

In linalg.trace, there is no dtype strategy. See:

array-api-tests/array_api_tests/test_linalg.py

Line 925 in 55fcc60

def test_trace(x, kw):

If I try to print generated inputs:

array_api_tests/test_linalg.py::test_trace kw={}
kw={}
kw={'offset': -272}
kw={'offset': 211}
kw={'offset': -138}
kw={'offset': -292}
kw={}
kw={}
kw={'offset': 563}
kw={}

Which means that functions that are missing coverage are:

reshape: missing copy
tril: missing test
triu: missing test
linalg.trace: missing dtype

Thanks!
reshape is #432
The other three would be great to have PRs for indeed.

Since device is already being tracked separately, I'll be happy to open another tracking issue for triu, tril and linalg.trace and close this one. Or maybe you can rename this PR title to something more accurate.

No need for a separate issue, I'd think. Either a single PR or a sequence, whichever works.

To summarize:

ENH: Adding tests for tril and triu #442 added test_tril and test_triu.
ENH: Adding dtype parameter to the test for linalg.trace #441 adds dtype to test_trace.
reshape's copy= is blocked at ENH: reshape: test copy= kwarg #432
The rest are false positives.

Let's close this.

Navigation Menu

Repository navigation

Incomplete tests: default arguments #439

Functions with `default only` parameters

Summary

Details

`Array.__array_namespace__`

`arange`

`asarray`

`astype`

`empty_like`

`eye`

`fft.fftfreq`

`fft.rfftfreq`

`full`

`full_like`

`linalg.trace`

`linspace`

`ones`

`ones_like`

`reshape`

`std`

`sum`

`tril`

`triu`

`zeros`

`zeros_like`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Incomplete tests: default arguments #439

Description

Activity

ev-br commented on Jul 7, 2026

Functions with default only parameters

Summary

Details

Array.__array_namespace__

arange

asarray

astype

empty_like

eye

fft.fftfreq

fft.rfftfreq

full

full_like

linalg.trace

linspace

ones

ones_like

reshape

std

sum

tril

triu

zeros

zeros_like

prady0t commented on Jul 12, 2026

prady0t commented on Jul 12, 2026

ev-br commented on Jul 12, 2026

prady0t commented on Jul 12, 2026

ev-br commented on Jul 12, 2026

prady0t commented on Jul 12, 2026

ev-br commented on Jul 12, 2026

ev-br commented on Jul 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Functions with `default only` parameters

`Array.__array_namespace__`

`arange`

`asarray`

`astype`

`empty_like`

`eye`

`fft.fftfreq`

`fft.rfftfreq`

`full`

`full_like`

`linalg.trace`

`linspace`

`ones`

`ones_like`

`reshape`

`std`

`sum`

`tril`

`triu`

`zeros`

`zeros_like`