Skip to content

[CON-906] Correct the CLI and SDK references against the live product - #12

Open
0xsergen wants to merge 4 commits into
mainfrom
con-906-cli-sdk-audit
Open

[CON-906] Correct the CLI and SDK references against the live product#12
0xsergen wants to merge 4 commits into
mainfrom
con-906-cli-sdk-audit

Conversation

@0xsergen

@0xsergen 0xsergen commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Part of the CON-901 reference audit. Covers CON-906: cli-reference.md and sdk-reference.md.

Every claim in both files was executed against the live account. 16 defects found and fixed.

Method

  • qn 0.6.0 CLI, walked recursively through --help. 126 leaf commands enumerated. All 126 are covered by the reference. 0 missing.
  • @quicknode/sdk 3.8.2, installed from the registry. 12 scripts written and run. Final result 12 pass, 0 fail.
  • All chain reads on ethereum-mainnet. No resource was created and nothing was billed.

What was wrong

# Where Claimed Actual
1 SDK import Quicknode The class is QuicknodeSdk. Quicknode is not an export
2 SDK construction new Quicknode({ apiKey }) QuicknodeSdk.fromEnv(), reading QN_SDK__API_KEY
3 SDK testFilter dataset: "block" The enum member StreamDataset.Block. A raw string is rejected
4 SDK errors one error type A typed tree: QuicknodeError with ConfigError, HttpError, TimeoutError, ConnectionError, ApiError, DecodeError, RpcError, PaymentError
5 qn kv list wording "ordered string collections" Lexicographically sorted. Insert zebra, mango, apple, read back apple, mango, zebra
6 qn stream test-filter output not documented result is a JSON string. It needs a second decode before you can read the filter's return value
7 qn rpc call -o json not documented Prints the JSON-RPC result unwrapped. There is no jsonrpc, id, or result field, so .result is undefined
8 piped output default piped output is always json qn rpc list-networks prints a bare newline-separated list even when piped. With -o json it gives {networks: [...]}, not a data envelope

The full list, with the exact invocation and exit code for each, is in the audit findings file.

The silent one

Item 6 is the one to read twice. It is a response-shape defect (class 1), and it fails silently. qn stream test-filter exits 0 and prints {"result": "...", "logs": []}. A caller that checks the exit code, or that reads .result without decoding it, cannot tell a working filter from a broken one. The reference now says so, and the Test: line asserts it.

Item 8 corrects a claim this PR itself made

Worth calling out, because it is the same mistake the audit exists to catch.

An earlier commit on this branch added a Test: line saying piped output defaults to json, "so a script never has to pass -o json". Then I piped four commands to head -c 1:

Command piped First byte
qn endpoint list {
qn chain list {
qn kv set list {
qn rpc list-networks 0

That last one is the 0 of 0g-galileo. The rule holds for three of four and not the fourth. A script that pipes qn rpc list-networks and parses JSON fails. The Test: line now names all four commands and the exception, and the prose above it says to pass -o json explicitly for that command.

The qn CLI has seven confirmed -o json shapes and no single envelope. Five are in kv-reference.md (see #13) and two are here. Read the shape for the command you ran. Never assume data.

Test: lines

11 added: 8 in cli-reference.md, 3 in sdk-reference.md. Each names the network and the block, then one value a reader can compare against their own run. Where an example touches no chain data it asserts the observable result instead.

Two assertions are deliberately written as shapes, not counts, because both drift: the network-key list names its first and last entry rather than gating on 132, and the block-object assertion names the missing envelope rather than a field count.

Coverage is 11 of 45 runnable blocks, and that is deliberate for 30 of the 34. Full accounting:

Reason a block has no Test: line Count
Creates a billed resource, or moves funds: paid RPC x402 and MPP, endpoint create, stream create, webhook creation, sql query 17
Account-scoped, so no value another reader can compare against: auth whoami, endpoint list/get, endpoint security, team ×3, account overview 9
Environment-specific: installation, auth login/logout 4
Real remaining work 4

The 4 are all SDK blocks: Admin API ×2, Webhooks, Key-Value Store. They are assertable, but running them needs QN_SDK__API_KEY, which was not exported in the session that added these lines. The three free RPC variants that were open earlier are now done, and closing them is what turned up items 7 and 8.

The Test: line assumes a pinned block and a value every reader can reproduce. That fits a filter reference, where streams-reference.md reaches 13 of 19. It mostly does not fit a CLI reference, because most CLI examples are account-scoped. Asserting this account's endpoint count helps no other reader, so those examples get nothing rather than something invented.

Also documented

Every delete command needs --yes when no terminal is attached. Without it the command exits non-zero and deletes nothing. Found while scripting this audit's own resource cleanup.

Checked and found correct, so not changed: qn kv list list is the canonical subcommand with ls as its alias. It is not a typo for ls.

Not covered, and why

  • qn sql query carries a per-query charge and does not auto-retry. Verified for help text and argument parsing only.
  • qn endpoint create and the stream create paths create billed resources. Not executed.

Version bump

None here. The build-web3 patch bump to 1.0.3 lands in the CON-905 PR, which is the last of the three to merge. Three PRs each editing the same four manifests would conflict.

Related: CON-902 and CON-905 PRs cover the other three files from the same audit pass.

Audited cli-reference.md and sdk-reference.md against qn 0.6.0 and
@quicknode/sdk 3.8.2. 14 confirmed defects. Every claim was executed,
not read.

Wrong values and enums:
- Chain vocabularies were conflated. `qn chain list` returns 76 short
  slugs (`eth`), not long names (`ethereum`). Added a vocabulary table
  for short slugs, long names, and network keys.
- `generatePaymentWallet` accepts "evm" | "svm" | "tempo". "tempo" was
  missing.

Examples that could never run:
- `eth_getBalance` used the placeholder `0xabc...`, which fails -32602.
  Replaced with a real EOA.
- The `eth_call` stdin literal was not valid JSON. Replaced with a real
  USDC totalSupply() call.
- The webhook example pointed `--template evm-wallet` at the USDC
  contract, not a wallet. Replaced with an EOA.

Silent failures (a 2xx or an exit 0 that returns nothing usable):
- `stream test-filter` returns `result` as a JSON string. The SDK
  example read fields off it directly, which yields undefined. Added
  JSON.parse and documented the shape.
- `logs` from test-filter can never be observed. The API sends objects,
  both clients decode string[], so any console.log in a filter fails
  the whole call. Documented, and warned against console.log.
- The SQL example read `result.rows`, which is a count. The rows are in
  `result.data`. The example ran a billed query and discarded it.
- "Lists store ordered values" is false. Lists sort lexicographically.
  Added a warning against using a list as a queue or cursor log.

Omissions:
- `supported-payments` can omit `asset` and can return a raw CAIP-2 id.
- `endpoint logs` and `log-details` are Enterprise-only, and report the
  gate as "unauthorized. Check your API key". Documented the -v
  diagnosis.
- Added the default output format (json when piped, table on a TTY) and
  the scope of --wide.
- Added an exit-code table. Every code was verified live.
- Added the nine typed SDK errors. Only the payment errors were listed.

Command coverage was already complete: a recursive --help walk found
126 leaf commands and 0 were missing.

Verification: 22/22 SDK assertions pass, and every corrected CLI
example was re-executed. Findings in the audit workspace.

Version bumps for the three plugin manifests land in the CON-905 PR,
the last of this set to merge.
CON-901 asks for a Test: line on every example. Added them for the
five examples re-verified live in this session, in PR #8's format.

CLI:
- RPC params block. eth_getBalance returns a hex quantity, and the
  piped eth_call returns USDC totalSupply() as one 32-byte hex word.
- stream test-filter on block 17811625. `result` is a JSON string and
  `logs` is [].

SDK, against a freshly installed 3.8.2. 12 of 12 assertions pass:
- RPC block. eth_getBalance returns a hex quantity; the truncated
  placeholder throws RpcError with code -32602.
- testFilter. typeof test.result is "string", test.logs is [], and a
  filter calling console.log throws DecodeError.
- generatePaymentWallet. evm and tempo give a 42-character 0x address,
  svm a base58 address, and any other value throws ConfigError naming
  all three.

The remaining examples in these two files carry no Test: line yet.
Two reasons, both recorded in the report:
- Account-scoped examples (`endpoint list`, `kv set list`, `stream
  list`) can only assert this account's counts. A reader on another
  account cannot compare against them, so the line would mislead.
- Mutating and paid paths were never executed, so there is no measured
  value to assert.

Also confirmed and unchanged: the published package's `types` entry is
sdk.d.ts, the documented class is QuicknodeSdk, the documented env var
is QN_SDK__API_KEY, and `dataset` takes the StreamDataset enum rather
than a raw string. Three drafts of the verification script got those
wrong; the reference had all four right.

Version bumps for the three plugin manifests land in the CON-905 PR,
the last of this set to merge.
CON-901 asks for a Test: line on every runnable example. These three were
assertable with a value any reader can reproduce, and all three are free
reads that create nothing.

- `qn agent context` returns 370 lines of Markdown titled
  "# qn — usage guide for agents", with 9 `##` sections. Verified with
  QN_API_KEY empty, which is how the "needs no authentication" claim in
  that section is actually proved rather than asserted.
- `qn endpoint list | head -c 1` prints `{`. Piped output defaults to
  json, not table, so a script never needs `-o json`. This is the claim
  the Output Formats defect was about.
- `qn kv set list` returns `data` as an array of {key, value};
  `qn kv list list` returns `data` as an object wrapping `keys`. The two
  commands do not share a response shape.

Also documented: every delete command needs `--yes` when no terminal is
attached. Without it the command exits non-zero and deletes nothing. Hit
while scripting this audit's own resource cleanup.

Checked and found correct, so not changed: `qn kv list list` is the
canonical subcommand, with `ls` as its alias. It is not a typo for `ls`.
These were the three remaining assertable and free examples in
cli-reference.md. All three now carry a Test: line. Coverage goes from
5 of 33 runnable blocks to 8.

Two live findings, both response-shape defects that fail silently.

`qn rpc call -o json` prints the JSON-RPC result unwrapped. There is no
jsonrpc, id, or result field. Code that reads `.result` on it gets
undefined.

`qn rpc list-networks -o json` returns `{"networks": [...]}`. That is a
sixth distinct CLI response shape and it is not the `data` envelope the
other list commands use, so `.data` is undefined.

Also corrects a claim this branch added in b4640c0. The piped-output
default is not universal. `qn rpc list-networks` prints a bare
newline-separated list even when piped, with no JSON, so a script that
pipes it and parses JSON fails. Verified: endpoint list, chain list and
kv set list all print `{` when piped; rpc list-networks prints `0`. The
Test: line now names all four commands and the exception.

The asserted network count is dated in the line and marked as growing,
because it drifts. The block-object assertion names the key range and
the missing envelope rather than a field count, for the same reason.

Verified live against account 464482. All calls read-only and free. No
resource created.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant