Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 34 additions & 33 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,44 @@
- **`CONTEXT.md` is the shared vocabulary** — getter, query, chunk, plan, fan-out,
page, adapter, facade, leaf, transport, collection, profile, effective
configuration, and the legacy names that are deliberately not renamed. Read it
before writing code, docstrings, or commit messages. Two kinds of term live
there and they bind differently (ADR 0013). A **core** term is ours and takes
one spelling everywhere *including identifiers*, so a name in the code that
conflicts with it is a defect. A **domain** term -- *monitoring location*,
*collection* -- is fixed for prose only: each adapter keeps its own service's
spelling in its parameters, so `nwis.get_record(service="dv")` and WQP's
`Station` are that service's language rather than drift. Each entry states
which kind it is.
before writing code, docstrings, or commit messages. Two kinds of term are
defined there and they bind differently (ADR 0013). A **core** term is ours
and takes one spelling everywhere *including identifiers*, so a name in the
code that conflicts with it is a defect. A **domain** term -- *monitoring
location*, *collection* -- is fixed for prose only: each adapter keeps its
own service's spelling in its parameters, so `nwis.get_record(service="dv")`
and WQP's `Station` are that service's vocabulary rather than drift. Each
entry states which kind it is.
- Architectural decisions and their rationale: `docs/source/architecture/decisions/`
(ADRs, referenced by number throughout the code and by `.importlinter`).
- Contributor workflow, style, and the quality gates in detail: `CONTRIBUTING.md`.

## How the tree is organized
Use `ls`/`grep` for the file list; what follows is the placement logic, so you
can predict where a thing lives.
can predict where a thing is defined.

- `dataretrieval/` — the public surface is one *adapter* module per service,
named for the service (`nldi`, `nwdc`, `ngwmn`, `streamstats`, `wqp`, and
legacy `nwis`); each owns that service's URLs, parameters, and response
quirks. `waterdata/` is the one adapter large enough to be a package, split by
collection family; its `api.py` is a compatibility facade holding no logic.
Everything else in the package is shared machinery the adapters sit on top of
Everything else in the package is shared code the adapters depend on
— configuration, credentials, progress, exceptions, code tables, response
formats. Shared machinery below the adapter layer must not know about any
formats. Shared code below the adapter layer must not reference any
particular service.
- `dataretrieval/ogc/` — the OGC API protocol machinery (chunk planning,
- `dataretrieval/ogc/` — the OGC API protocol code (chunk planning,
filters, request building, response shaping). Shared by the two OGC services
only; `.importlinter` refuses any other importer.
- `dataretrieval/transport/` — service-neutral request machinery (HTTP, retry,
- `dataretrieval/transport/` — service-neutral request code (HTTP, retry,
pagination, fan-out). It names no service and no protocol, and is not public API.
- Leading-underscore top-level modules are private; the dependency-free *leaves*
sit at the floor of the stack so anything may use them without pulling in the
are at the bottom of the stack so anything may use them without pulling in the
rest of the package. Check for an existing leaf before writing a small helper.
- **`.importlinter` is the map.** Its `layers` contract lists every top-level
module in dependency order and is `exhaustive = True`, so it is both the
authoritative statement of where a module sits and the thing that fails when a
new module has no home. Read it before adding a module or an import.
- **`.importlinter` records where every module belongs.** Its `layers` contract
lists every top-level module in dependency order and is `exhaustive = True`,
so it is both the authoritative statement of where a module belongs and the
thing that fails when a new module has not been placed. Read it before adding
a module or an import.
- `tests/` — flat, one `*_test.py` per module or concern, organized into four
dependency-oriented layers (public contract, adapter contract, component,
cross-component) that `tests/contracts/README.md` defines and assigns files to.
Expand Down Expand Up @@ -74,10 +75,10 @@ can predict where a thing lives.
- Tests: `coverage run -m pytest tests/ && coverage report`, or focused like
`pytest tests/waterdata_test.py::test_mock_get_samples`. `coverage report` is
a merge gate: branch coverage with a `fail_under` ratchet in
`[tool.coverage.report]`. Chase the uncovered *branch*, not the number -- a
test written to colour a line green catches nothing and costs a maintenance
slot. If a path is genuinely unreachable, add it to `exclude_also` with a
reason, or leave the ratchet alone.
`[tool.coverage.report]`. Cover the uncovered *branch*, not the number -- a
test written only to mark a line as covered catches nothing and adds
maintenance. If a path is genuinely unreachable, add it to `exclude_also`
with a reason, or leave the ratchet alone.
- Types: `mypy` (`strict = true` in `pyproject.toml`; CI runs it over the
PR-merged-into-main, so bare `dict`/`list` annotations fail there even if they
pass on your branch).
Expand All @@ -88,7 +89,7 @@ can predict where a thing lives.

## Testing gotchas
- The suite is offline by default: `addopts = "-m 'not live'"`. Tests marked
`@pytest.mark.live` hit real USGS services and run on a schedule
`@pytest.mark.live` call real USGS services and run on a schedule
(`.github/workflows/live-api.yml`); run them locally with `pytest tests/ -m live`.
- HTTP is mocked with `pytest-httpx`'s `httpx_mock` fixture plus fixtures under
`tests/data/`; keep new API tests offline.
Expand All @@ -101,23 +102,23 @@ can predict where a thing lives.
## Error messages
Most callers here are programs — a script, a pipeline stage, an agent — so a
message is the only channel through which a caller can correct itself. Every
raise states the problem and then the move that fixes it, in that order.
raise states the problem and then the action that fixes it, in that order.

- Name the remedy, not just the fault. `"Service not recognized"` gives a caller
nothing to try next; listing the services it does accept does. For a transport
failure the remedy is whether to retry, and `transport.pagination.
paginated_failure_message()` is the model: cause, then `To recover: …`.
- Don't invent a phrasing for a check that recurs. `dataretrieval/_validation.py`
owns the wording for the shared shapes — bad value in a closed vocabulary
owns the wording for the shared cases — bad value in a closed vocabulary
(`require_one_of`), missing argument (`require_argument`), incomplete group
(`require_together`), no filter at all (`require_any_of`), and conflicting
arguments (`require_exactly_one`, `reject_together`). Reach for one before
arguments (`require_exactly_one`, `reject_together`). Use one before
hand-writing a message. A service-specific pointer is not a reason to
hand-write: every check takes a `remedy=` for the move it cannot derive. Every
check raises `ValueError` -- one class for a bad argument value, so a caller
catches by shape rather than by which module rejected it.
hand-write: every check takes a `remedy=` for the action it cannot derive.
Every check raises `ValueError` -- one class for a bad argument value, so a
caller catches by error class rather than by which module rejected it.
- `require_argument` returns the narrowed value and `require_exactly_one` the
winning `(name, value)` pair, so use their results rather than re-testing for
selected `(name, value)` pair, so use their results rather than re-testing for
`None` to satisfy mypy — a second, unreachable message beside the first is
how the two drift apart.
- **Paste the remedy back before trusting it.** Whatever a message names must be
Expand All @@ -127,9 +128,9 @@ raise states the problem and then the move that fixes it, in that order.
private local no getter accepts, `configure(Configuration(...))` was a silent
no-op because `configure` is a context manager, `pip install
dataretrieval[nldi]` globs in zsh, and a navigation missing its `data_source`
spelled `None` into the URL and returned an empty frame. Run the corrected
wrote `None` into the URL and returned an empty frame. Run the corrected
call against the real service; wording review does not catch these.
- Shared checks take the caller's spelling. `_validate_data_source`,
- Shared checks take the caller's argument name. `_validate_data_source`,
`_format_api_dates`, and `require_one_of` all accept a `name=` so the subject
of the message is the argument that was actually passed. A helper that hard-codes
one noun reports the wrong parameter the moment a second call site reuses it.
Expand All @@ -151,7 +152,7 @@ raise states the problem and then the move that fixes it, in that order.
- The `API_USGS_PAT` credential is owned by the `credentials` leaf and applied as
the `X-Api-Key` header by `transport.http.default_headers()`, which sends it
only to the host it belongs to. Never hard-code tokens in examples or tests.
- Water Data request builders translate Python kwargs to API spellings
- Water Data request builders translate Python kwargs to API names
(`skip_geometry` -> `skipGeometry`, `filter_lang` -> `filter-lang`); tests
assert exact URLs and query params.
- Multi-value OGC params are comma-joined GETs, except `monitoring-locations`
Expand Down
Loading