diff --git a/.claude/skills/build-plugin/SKILL.md b/.claude/skills/build-plugin/SKILL.md index e842a27e..bf98e0c4 100644 --- a/.claude/skills/build-plugin/SKILL.md +++ b/.claude/skills/build-plugin/SKILL.md @@ -62,7 +62,7 @@ Before writing a single file, understand and explore the API. **Use `AskUserQues 1. **Find the docs** — Gather URLs or spec files from the user, then fetch and read them. 2. **Identify the object model** — What are the core entities? (e.g. installations, devices, sites). These become the **indexed objects** in SquaredUp — available for drilldown, search, scoping dashboards, and use as variables. -3. **Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout but supports as many paged requests as needed. +3. **Find the list endpoints** — Used to import objects. Note the page-size parameter's maximum, and estimate how many objects the largest environment holds (ask the user if unsure). An import run is capped at ~220 pages and ~6MB per page, with a per-page timeout. Start from **50–250 records per page** when that keeps the largest environment comfortably under ~200 pages; tens of thousands of objects need a larger `pageSize` with lean rows — see [index-defs.md](references/index-defs.md#large-imports--choosing-pagesize). 4. **Find the data endpoints** — These power data streams. For each, record three things; Phase 2 turns the last two into the stream's `timeframes`: - **Scoping** — scoped to a single object, multiple objects, or global (no object context). - **Time-range control** — does the endpoint accept a queryable time range at all (a `from`/`to`, `start`/`end`, or `period` parameter), or only return a fixed snapshot / current values? diff --git a/.claude/skills/build-plugin/references/checkpoints.md b/.claude/skills/build-plugin/references/checkpoints.md index 2fb123f8..fdb8c8e5 100644 --- a/.claude/skills/build-plugin/references/checkpoints.md +++ b/.claude/skills/build-plugin/references/checkpoints.md @@ -56,7 +56,12 @@ squaredup objects --matches '{"sourceType":{"type":"equals","value":"` (not `--plugin-id`), since the re-index endpoint is keyed purely on the datasource. -- **Branch on the JSON, not the exit code.** `index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger. +- **Branch on the JSON, not the exit code.** `index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger. Three platform errors read as noise but have plugin-side causes: + - `Payload failed schema validation` with `/vertices/N/` — a [reserved property name](index-defs.md) (`id`, `state`, `__…`) in `objectMapping.properties`. + - `Function.ResponseSizeTooLarge` — one page is over ~6MB; lower `pageSize` or trim the row. + - `'call-chain-length' reached threshold of 220` — too many pages in the run; raise `pageSize`. + + For the last two, see [Large imports](index-defs.md#large-imports--choosing-pagesize). - **`status` is the run's lifecycle, not the outcome.** While running it's `ready`/`inProgress`; once `done` it's one of `succeeded`, `failed`, `warning`, or `cancelled`. `succeeded` and `warning` both report `succeeded: true` (a `warning` run finished but a step emitted warnings — check the `steps[]`); `failed` and `cancelled` report `succeeded: false`. A datasource that has never imported reports `status: "notRun"`, `done: false`. - `--since` is **exclusive** (`scheduledStart > since`): always pass the `since` from `index` so `done` can't latch on a stale previous run. - If `index` reports `alreadyRunning: true`, it adopted the in-flight run — poll with the `since` it returned. Imports can take several minutes (object import allows up to ~10 min); use a generous overall timeout. diff --git a/.claude/skills/build-plugin/references/index-defs.md b/.claude/skills/build-plugin/references/index-defs.md index cc7bc710..fab10aa2 100644 --- a/.claude/skills/build-plugin/references/index-defs.md +++ b/.claude/skills/build-plugin/references/index-defs.md @@ -60,6 +60,12 @@ Defines what gets imported into the SquaredUp graph. - `objectMapping.properties` are extra fields stored on the graph node, accessible in scripts as `object.propName`. - Use `{ "targetProp": "sourceProp" }` syntax when the column name differs from the desired property name. - **Never map a column into `properties` that is already mapped as `id`, `name`, or `type`.** The id's raw value is always available on every object as `rawId` (`{{object.rawId}}` in templates, `context.objects[N].rawId` in scripts — and as a scalar, unlike user-defined properties, which arrive as arrays), and the name as `name`. Adding e.g. `{ "projectId": "id" }` to `properties` when `"id": "id"` already exists creates a duplicate that has to be re-indexed to take effect and otherwise sits as dead config — use `rawId`/`name` instead. +- **Never use a reserved name as a property name.** The property name — a string entry, or the *key* in `{ "targetProp": "sourceProp" }` — becomes a property on the graph object, and the graph owns some names. Using one deploys cleanly, then fails the import with a raw schema error that doesn't mention your mapping: + - `id`, and any `__`-prefixed name — always rejected: `Payload failed schema validation` … `/vertices/0/id` … `boolean schema is false`. You don't need it — the object's id is already `rawId` (above). + - `state` — reserved by the platform; an API status (`"active"`) or a postal state (`"OH"`) fails with `must be equal to one of the allowed values`. Rename it: `{ "status": "state" }`, or `{ "region": "state" }` for an address. + - `rawId`, `sourceId`, `sourceType`, `name`, `label` — silently overwritten by the platform, so the mapping does nothing. + + A reserved name is fine as the *source column* (`{ "status": "state" }` reads a column called `state`). `squaredup validate` rejects `id`, `state` and `__` names; it can't catch the silent overwrites. - The `objectMapping.sourceType` column value **must** match an entry in `objectTypes` in `metadata.json`. For dynamic ones add these based on API response data later. - `frequencyMinutes` — controls re-import interval. Defaults to `720` (12 hours). @@ -79,6 +85,30 @@ Building and testing a dependent step needs objects from its dependency to alrea --- +## Large imports — choosing `pageSize` + +An import step fetches one page at a time and writes each page's objects to the graph before asking for the next, so a step has no object-count limit of its own. Two limits bound the stream's [`paging.pageSize`](data-streams.md#pagination) from opposite sides instead: + +| Limit | What it counts | `errorReason` when hit | Pushes `pageSize` | +| --- | --- | --- | --- | +| **~6MB per page** | One page, measured after SquaredUp formats it — larger than the raw API body. That page's objects are then written to the graph in a single request. | `Function.ResponseSizeTooLarge` | down | +| **~220 calls per import run** | Each page is one more call in a chain. A dependent step continues the chain of the step it `dependsOn`. | `'call-chain-length' reached threshold of 220` | up | + +Each page must also return within the ~28s per-page timeout, another reason not to max out `pageSize`. + +Size it for the **largest environment the plugin will meet**, not your test tenant — a 500-object test account passes at any page size: + +1. **Pages** — `objects ÷ pageSize`, plus the pages of every step this one transitively `dependsOn`, must stay comfortably under ~200. +2. **Bytes** — `pageSize × bytes per row` should stay under ~2MB of raw API response, leaving headroom for formatting. Measure a real page during exploration, e.g. `curl … | wc -c`. + +A worked example: a CrowdStrike import of ~92,700 hosts overflowed 6MB at 1,000 hosts per page. 250 per page would have needed ~371 calls, so 500 per page with a lean row (~186 calls) was the size that fit both limits. + +**Keep import rows lean.** An import stream exists to feed `objectMapping` — return only the id, name, type and mapped properties. If the API has a field-selection parameter (`fields`, `select`, `$select`), use it, and leave nested blobs and anything only a dashboard needs to a separate data stream. Smaller rows let a larger `pageSize` fit under 6MB, which is what lets a big inventory fit under the call cap. + +**If no `pageSize` satisfies both**, import less rather than fighting the limits: filter at the API (e.g. only active or managed hosts), or import a subset and query the rest live through unscoped data streams. Raise it with the user — the call cap is a platform setting, not something a plugin can change. + +--- + ## Stream-level requirements Import streams are ordinary data stream files in `dataStreams/` — they need `name`, `displayName`, `description`, and `tags` like any other stream. Mark import-only streams as hidden so they don't clutter the tile editor: diff --git a/REVIEW.md b/REVIEW.md index f21d7c37..95d6cf37 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -104,6 +104,8 @@ When suggesting changes: ### Source types (indexDefinitions/) - Name source types after how they are referred to in the upstream product or API (e.g. `agent`, `device`). Do not prefix them with the plugin name (e.g. avoid `NinjaOne Device`). A separate friendly display name can be configured if needed (via custom_types.json). +- objectMapping.properties - Never use `id`, `state`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing. +- pageSize - For APIs where a tenant can hold tens of thousands of objects, check the import stream's `pageSize` against both import limits: ~6MB per page and ~220 pages per import run. See the build-plugin skill's index-defs.md, "Large imports". ### Documentation - (docs/README.md)