[CON-906] Correct the CLI and SDK references against the live product - #12
Open
0xsergen wants to merge 4 commits into
Open
[CON-906] Correct the CLI and SDK references against the live product#120xsergen wants to merge 4 commits into
0xsergen wants to merge 4 commits into
Conversation
Audited cli-reference.md and sdk-reference.md against qn 0.6.0 and @quicknode/sdk 3.8.2. 14 confirmed defects. Every claim was executed, not read. Wrong values and enums: - Chain vocabularies were conflated. `qn chain list` returns 76 short slugs (`eth`), not long names (`ethereum`). Added a vocabulary table for short slugs, long names, and network keys. - `generatePaymentWallet` accepts "evm" | "svm" | "tempo". "tempo" was missing. Examples that could never run: - `eth_getBalance` used the placeholder `0xabc...`, which fails -32602. Replaced with a real EOA. - The `eth_call` stdin literal was not valid JSON. Replaced with a real USDC totalSupply() call. - The webhook example pointed `--template evm-wallet` at the USDC contract, not a wallet. Replaced with an EOA. Silent failures (a 2xx or an exit 0 that returns nothing usable): - `stream test-filter` returns `result` as a JSON string. The SDK example read fields off it directly, which yields undefined. Added JSON.parse and documented the shape. - `logs` from test-filter can never be observed. The API sends objects, both clients decode string[], so any console.log in a filter fails the whole call. Documented, and warned against console.log. - The SQL example read `result.rows`, which is a count. The rows are in `result.data`. The example ran a billed query and discarded it. - "Lists store ordered values" is false. Lists sort lexicographically. Added a warning against using a list as a queue or cursor log. Omissions: - `supported-payments` can omit `asset` and can return a raw CAIP-2 id. - `endpoint logs` and `log-details` are Enterprise-only, and report the gate as "unauthorized. Check your API key". Documented the -v diagnosis. - Added the default output format (json when piped, table on a TTY) and the scope of --wide. - Added an exit-code table. Every code was verified live. - Added the nine typed SDK errors. Only the payment errors were listed. Command coverage was already complete: a recursive --help walk found 126 leaf commands and 0 were missing. Verification: 22/22 SDK assertions pass, and every corrected CLI example was re-executed. Findings in the audit workspace. Version bumps for the three plugin manifests land in the CON-905 PR, the last of this set to merge.
CON-901 asks for a Test: line on every example. Added them for the five examples re-verified live in this session, in PR #8's format. CLI: - RPC params block. eth_getBalance returns a hex quantity, and the piped eth_call returns USDC totalSupply() as one 32-byte hex word. - stream test-filter on block 17811625. `result` is a JSON string and `logs` is []. SDK, against a freshly installed 3.8.2. 12 of 12 assertions pass: - RPC block. eth_getBalance returns a hex quantity; the truncated placeholder throws RpcError with code -32602. - testFilter. typeof test.result is "string", test.logs is [], and a filter calling console.log throws DecodeError. - generatePaymentWallet. evm and tempo give a 42-character 0x address, svm a base58 address, and any other value throws ConfigError naming all three. The remaining examples in these two files carry no Test: line yet. Two reasons, both recorded in the report: - Account-scoped examples (`endpoint list`, `kv set list`, `stream list`) can only assert this account's counts. A reader on another account cannot compare against them, so the line would mislead. - Mutating and paid paths were never executed, so there is no measured value to assert. Also confirmed and unchanged: the published package's `types` entry is sdk.d.ts, the documented class is QuicknodeSdk, the documented env var is QN_SDK__API_KEY, and `dataset` takes the StreamDataset enum rather than a raw string. Three drafts of the verification script got those wrong; the reference had all four right. Version bumps for the three plugin manifests land in the CON-905 PR, the last of this set to merge.
CON-901 asks for a Test: line on every runnable example. These three were
assertable with a value any reader can reproduce, and all three are free
reads that create nothing.
- `qn agent context` returns 370 lines of Markdown titled
"# qn — usage guide for agents", with 9 `##` sections. Verified with
QN_API_KEY empty, which is how the "needs no authentication" claim in
that section is actually proved rather than asserted.
- `qn endpoint list | head -c 1` prints `{`. Piped output defaults to
json, not table, so a script never needs `-o json`. This is the claim
the Output Formats defect was about.
- `qn kv set list` returns `data` as an array of {key, value};
`qn kv list list` returns `data` as an object wrapping `keys`. The two
commands do not share a response shape.
Also documented: every delete command needs `--yes` when no terminal is
attached. Without it the command exits non-zero and deletes nothing. Hit
while scripting this audit's own resource cleanup.
Checked and found correct, so not changed: `qn kv list list` is the
canonical subcommand, with `ls` as its alias. It is not a typo for `ls`.
These were the three remaining assertable and free examples in
cli-reference.md. All three now carry a Test: line. Coverage goes from
5 of 33 runnable blocks to 8.
Two live findings, both response-shape defects that fail silently.
`qn rpc call -o json` prints the JSON-RPC result unwrapped. There is no
jsonrpc, id, or result field. Code that reads `.result` on it gets
undefined.
`qn rpc list-networks -o json` returns `{"networks": [...]}`. That is a
sixth distinct CLI response shape and it is not the `data` envelope the
other list commands use, so `.data` is undefined.
Also corrects a claim this branch added in b4640c0. The piped-output
default is not universal. `qn rpc list-networks` prints a bare
newline-separated list even when piped, with no JSON, so a script that
pipes it and parses JSON fails. Verified: endpoint list, chain list and
kv set list all print `{` when piped; rpc list-networks prints `0`. The
Test: line now names all four commands and the exception.
The asserted network count is dated in the line and marked as growing,
because it drifts. The block-object assertion names the key range and
the missing envelope rather than a field count, for the same reason.
Verified live against account 464482. All calls read-only and free. No
resource created.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the CON-901 reference audit. Covers CON-906:
cli-reference.mdandsdk-reference.md.Every claim in both files was executed against the live account. 16 defects found and fixed.
Method
qn0.6.0 CLI, walked recursively through--help. 126 leaf commands enumerated. All 126 are covered by the reference. 0 missing.@quicknode/sdk3.8.2, installed from the registry. 12 scripts written and run. Final result 12 pass, 0 fail.ethereum-mainnet. No resource was created and nothing was billed.What was wrong
QuicknodeQuicknodeSdk.Quicknodeis not an exportnew Quicknode({ apiKey })QuicknodeSdk.fromEnv(), readingQN_SDK__API_KEYtestFilterdataset: "block"StreamDataset.Block. A raw string is rejectedQuicknodeErrorwithConfigError,HttpError,TimeoutError,ConnectionError,ApiError,DecodeError,RpcError,PaymentErrorqn kvlist wordingzebra, mango, apple, read backapple, mango, zebraqn stream test-filteroutputresultis a JSON string. It needs a second decode before you can read the filter's return valueqn rpc call -o jsonresultunwrapped. There is nojsonrpc,id, orresultfield, so.resultisundefinedjsonqn rpc list-networksprints a bare newline-separated list even when piped. With-o jsonit gives{networks: [...]}, not adataenvelopeThe full list, with the exact invocation and exit code for each, is in the audit findings file.
The silent one
Item 6 is the one to read twice. It is a response-shape defect (class 1), and it fails silently.
qn stream test-filterexits 0 and prints{"result": "...", "logs": []}. A caller that checks the exit code, or that reads.resultwithout decoding it, cannot tell a working filter from a broken one. The reference now says so, and theTest:line asserts it.Item 8 corrects a claim this PR itself made
Worth calling out, because it is the same mistake the audit exists to catch.
An earlier commit on this branch added a
Test:line saying piped output defaults tojson, "so a script never has to pass-o json". Then I piped four commands tohead -c 1:qn endpoint list{qn chain list{qn kv set list{qn rpc list-networks0That last one is the
0of0g-galileo. The rule holds for three of four and not the fourth. A script that pipesqn rpc list-networksand parses JSON fails. TheTest:line now names all four commands and the exception, and the prose above it says to pass-o jsonexplicitly for that command.The
qnCLI has seven confirmed-o jsonshapes and no single envelope. Five are inkv-reference.md(see #13) and two are here. Read the shape for the command you ran. Never assumedata.Test: lines
11 added: 8 in
cli-reference.md, 3 insdk-reference.md. Each names the network and the block, then one value a reader can compare against their own run. Where an example touches no chain data it asserts the observable result instead.Two assertions are deliberately written as shapes, not counts, because both drift: the network-key list names its first and last entry rather than gating on 132, and the block-object assertion names the missing envelope rather than a field count.
Coverage is 11 of 45 runnable blocks, and that is deliberate for 30 of the 34. Full accounting:
endpoint create,stream create, webhook creation,sql queryauth whoami,endpoint list/get, endpoint security, team ×3, account overviewauth login/logoutThe 4 are all SDK blocks: Admin API ×2, Webhooks, Key-Value Store. They are assertable, but running them needs
QN_SDK__API_KEY, which was not exported in the session that added these lines. The three free RPC variants that were open earlier are now done, and closing them is what turned up items 7 and 8.The
Test:line assumes a pinned block and a value every reader can reproduce. That fits a filter reference, wherestreams-reference.mdreaches 13 of 19. It mostly does not fit a CLI reference, because most CLI examples are account-scoped. Asserting this account's endpoint count helps no other reader, so those examples get nothing rather than something invented.Also documented
Every
deletecommand needs--yeswhen no terminal is attached. Without it the command exits non-zero and deletes nothing. Found while scripting this audit's own resource cleanup.Checked and found correct, so not changed:
qn kv list listis the canonical subcommand withlsas its alias. It is not a typo forls.Not covered, and why
qn sql querycarries a per-query charge and does not auto-retry. Verified for help text and argument parsing only.qn endpoint createand thestream createpaths create billed resources. Not executed.Version bump
None here. The build-web3 patch bump to 1.0.3 lands in the CON-905 PR, which is the last of the three to merge. Three PRs each editing the same four manifests would conflict.
Related: CON-902 and CON-905 PRs cover the other three files from the same audit pass.