From 39e2a37df648b75b2a64820cbbea1ab92ebd354d Mon Sep 17 00:00:00 2001 From: Dave Clarke Date: Wed, 14 Oct 2026 17:24:21 +0100 Subject: [PATCH 1/3] Warn LCP authors off reserved property names A customer's Entra ID plugin deployed cleanly, then every import failed with "boolean schema is false" at /vertices/0/id. The graph owns some property names, and nothing in the skill or the review rules said so, so authors had no way to know `id` or `state` in objectMapping.properties would fail. The import-definition reference now lists the names that fail (`id`, `__` internals, `state`), the ones the platform silently overwrites, and the raw errors each produces, with renames to use instead. `state` is documented only as reserved: it can hold object health, but that's not something we want plugins to start relying on. REVIEW.md gets the matching reviewer rule. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/build-plugin/references/index-defs.md | 6 ++++++ REVIEW.md | 1 + 2 files changed, 7 insertions(+) diff --git a/.claude/skills/build-plugin/references/index-defs.md b/.claude/skills/build-plugin/references/index-defs.md index cc7bc710..062eb0a6 100644 --- a/.claude/skills/build-plugin/references/index-defs.md +++ b/.claude/skills/build-plugin/references/index-defs.md @@ -60,6 +60,12 @@ Defines what gets imported into the SquaredUp graph. - `objectMapping.properties` are extra fields stored on the graph node, accessible in scripts as `object.propName`. - Use `{ "targetProp": "sourceProp" }` syntax when the column name differs from the desired property name. - **Never map a column into `properties` that is already mapped as `id`, `name`, or `type`.** The id's raw value is always available on every object as `rawId` (`{{object.rawId}}` in templates, `context.objects[N].rawId` in scripts — and as a scalar, unlike user-defined properties, which arrive as arrays), and the name as `name`. Adding e.g. `{ "projectId": "id" }` to `properties` when `"id": "id"` already exists creates a duplicate that has to be re-indexed to take effect and otherwise sits as dead config — use `rawId`/`name` instead. +- **Never use a reserved name as a property name.** The property name — a string entry, or the *key* in `{ "targetProp": "sourceProp" }` — becomes a property on the graph object, and the graph owns some names. Using one deploys cleanly, then fails the import with a raw schema error that doesn't mention your mapping: + - `id`, and any `__`-prefixed name — always rejected: `Payload failed schema validation` … `/vertices/0/id` … `boolean schema is false`. You don't need it — the object's id is already `rawId` (above). + - `state` — reserved by the platform; an API status (`"active"`) or a postal state (`"OH"`) fails with `must be equal to one of the allowed values`. Rename it: `{ "status": "state" }`, or `{ "region": "state" }` for an address. + - `rawId`, `sourceId`, `sourceType`, `name`, `label` — silently overwritten by the platform, so the mapping does nothing. + + A reserved name is fine as the *source column* (`{ "status": "state" }` reads a column called `state`). `squaredup validate` rejects `id`, `state` and `__` names; it can't catch the silent overwrites. - The `objectMapping.sourceType` column value **must** match an entry in `objectTypes` in `metadata.json`. For dynamic ones add these based on API response data later. - `frequencyMinutes` — controls re-import interval. Defaults to `720` (12 hours). diff --git a/REVIEW.md b/REVIEW.md index f21d7c37..d61ea5c4 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -104,6 +104,7 @@ When suggesting changes: ### Source types (indexDefinitions/) - Name source types after how they are referred to in the upstream product or API (e.g. `agent`, `device`). Do not prefix them with the plugin name (e.g. avoid `NinjaOne Device`). A separate friendly display name can be configured if needed (via custom_types.json). +- objectMapping.properties - Never use `id`, `state`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing. ### Documentation - (docs/README.md) From f2e00a262bf31b56736f177ea2ca2d0f1b1a6b3c Mon Sep 17 00:00:00 2001 From: Dave Clarke Date: Wed, 14 Oct 2026 17:24:21 +0100 Subject: [PATCH 2/3] Size import pageSize against both import limits CoPA's CrowdStrike import (~92,700 hosts) hit two limits the skill never mentioned. At 1,000 hosts per page one page overflowed the ~6MB Lambda response cap, and dropping to 250 per page would have needed ~371 calls against a ~220 call-chain cap per import run. The skill also claimed SquaredUp "supports as many paged requests as needed", which steered authors towards small pages. Import steps run page by page, so the 6MB cap applies to each page (after formatting) while every page adds a hop to a chain shared with any step it depends on. The two limits push pageSize in opposite directions, so the new section explains how to size it for the largest expected environment, keep import rows lean so a bigger page fits, and fall back to importing less when nothing fits. Phase 1 now asks for an object-count estimate up front, and checkpoints decode the raw import errors these limits produce. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/build-plugin/SKILL.md | 2 +- .../build-plugin/references/checkpoints.md | 7 +++++- .../build-plugin/references/index-defs.md | 24 +++++++++++++++++++ REVIEW.md | 1 + 4 files changed, 32 insertions(+), 2 deletions(-) diff --git a/.claude/skills/build-plugin/SKILL.md b/.claude/skills/build-plugin/SKILL.md index e842a27e..61c6ffc0 100644 --- a/.claude/skills/build-plugin/SKILL.md +++ b/.claude/skills/build-plugin/SKILL.md @@ -62,7 +62,7 @@ Before writing a single file, understand and explore the API. **Use `AskUserQues 1. **Find the docs** — Gather URLs or spec files from the user, then fetch and read them. 2. **Identify the object model** — What are the core entities? (e.g. installations, devices, sites). These become the **indexed objects** in SquaredUp — available for drilldown, search, scoping dashboards, and use as variables. -3. **Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout but supports as many paged requests as needed. +3. **Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout. Note the page-size parameter's maximum, and estimate how many objects the largest environment holds (ask the user if unsure): an import run is capped at ~220 pages and ~6MB per page, so tens of thousands of objects constrain `pageSize` — see [index-defs.md](references/index-defs.md#large-imports--choosing-pagesize). 4. **Find the data endpoints** — These power data streams. For each, record three things; Phase 2 turns the last two into the stream's `timeframes`: - **Scoping** — scoped to a single object, multiple objects, or global (no object context). - **Time-range control** — does the endpoint accept a queryable time range at all (a `from`/`to`, `start`/`end`, or `period` parameter), or only return a fixed snapshot / current values? diff --git a/.claude/skills/build-plugin/references/checkpoints.md b/.claude/skills/build-plugin/references/checkpoints.md index 2fb123f8..fdb8c8e5 100644 --- a/.claude/skills/build-plugin/references/checkpoints.md +++ b/.claude/skills/build-plugin/references/checkpoints.md @@ -56,7 +56,12 @@ squaredup objects --matches '{"sourceType":{"type":"equals","value":"` (not `--plugin-id`), since the re-index endpoint is keyed purely on the datasource. -- **Branch on the JSON, not the exit code.** `index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger. +- **Branch on the JSON, not the exit code.** `index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger. Three platform errors read as noise but have plugin-side causes: + - `Payload failed schema validation` with `/vertices/N/` — a [reserved property name](index-defs.md) (`id`, `state`, `__…`) in `objectMapping.properties`. + - `Function.ResponseSizeTooLarge` — one page is over ~6MB; lower `pageSize` or trim the row. + - `'call-chain-length' reached threshold of 220` — too many pages in the run; raise `pageSize`. + + For the last two, see [Large imports](index-defs.md#large-imports--choosing-pagesize). - **`status` is the run's lifecycle, not the outcome.** While running it's `ready`/`inProgress`; once `done` it's one of `succeeded`, `failed`, `warning`, or `cancelled`. `succeeded` and `warning` both report `succeeded: true` (a `warning` run finished but a step emitted warnings — check the `steps[]`); `failed` and `cancelled` report `succeeded: false`. A datasource that has never imported reports `status: "notRun"`, `done: false`. - `--since` is **exclusive** (`scheduledStart > since`): always pass the `since` from `index` so `done` can't latch on a stale previous run. - If `index` reports `alreadyRunning: true`, it adopted the in-flight run — poll with the `since` it returned. Imports can take several minutes (object import allows up to ~10 min); use a generous overall timeout. diff --git a/.claude/skills/build-plugin/references/index-defs.md b/.claude/skills/build-plugin/references/index-defs.md index 062eb0a6..fab10aa2 100644 --- a/.claude/skills/build-plugin/references/index-defs.md +++ b/.claude/skills/build-plugin/references/index-defs.md @@ -85,6 +85,30 @@ Building and testing a dependent step needs objects from its dependency to alrea --- +## Large imports — choosing `pageSize` + +An import step fetches one page at a time and writes each page's objects to the graph before asking for the next, so a step has no object-count limit of its own. Two limits bound the stream's [`paging.pageSize`](data-streams.md#pagination) from opposite sides instead: + +| Limit | What it counts | `errorReason` when hit | Pushes `pageSize` | +| --- | --- | --- | --- | +| **~6MB per page** | One page, measured after SquaredUp formats it — larger than the raw API body. That page's objects are then written to the graph in a single request. | `Function.ResponseSizeTooLarge` | down | +| **~220 calls per import run** | Each page is one more call in a chain. A dependent step continues the chain of the step it `dependsOn`. | `'call-chain-length' reached threshold of 220` | up | + +Each page must also return within the ~28s per-page timeout, another reason not to max out `pageSize`. + +Size it for the **largest environment the plugin will meet**, not your test tenant — a 500-object test account passes at any page size: + +1. **Pages** — `objects ÷ pageSize`, plus the pages of every step this one transitively `dependsOn`, must stay comfortably under ~200. +2. **Bytes** — `pageSize × bytes per row` should stay under ~2MB of raw API response, leaving headroom for formatting. Measure a real page during exploration, e.g. `curl … | wc -c`. + +A worked example: a CrowdStrike import of ~92,700 hosts overflowed 6MB at 1,000 hosts per page. 250 per page would have needed ~371 calls, so 500 per page with a lean row (~186 calls) was the size that fit both limits. + +**Keep import rows lean.** An import stream exists to feed `objectMapping` — return only the id, name, type and mapped properties. If the API has a field-selection parameter (`fields`, `select`, `$select`), use it, and leave nested blobs and anything only a dashboard needs to a separate data stream. Smaller rows let a larger `pageSize` fit under 6MB, which is what lets a big inventory fit under the call cap. + +**If no `pageSize` satisfies both**, import less rather than fighting the limits: filter at the API (e.g. only active or managed hosts), or import a subset and query the rest live through unscoped data streams. Raise it with the user — the call cap is a platform setting, not something a plugin can change. + +--- + ## Stream-level requirements Import streams are ordinary data stream files in `dataStreams/` — they need `name`, `displayName`, `description`, and `tags` like any other stream. Mark import-only streams as hidden so they don't clutter the tile editor: diff --git a/REVIEW.md b/REVIEW.md index d61ea5c4..95d6cf37 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -105,6 +105,7 @@ When suggesting changes: - Name source types after how they are referred to in the upstream product or API (e.g. `agent`, `device`). Do not prefix them with the plugin name (e.g. avoid `NinjaOne Device`). A separate friendly display name can be configured if needed (via custom_types.json). - objectMapping.properties - Never use `id`, `state`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing. +- pageSize - For APIs where a tenant can hold tens of thousands of objects, check the import stream's `pageSize` against both import limits: ~6MB per page and ~220 pages per import run. See the build-plugin skill's index-defs.md, "Large imports". ### Documentation - (docs/README.md) From 581f5dc51014aec747083016112302498f9a156e Mon Sep 17 00:00:00 2001 From: Dave Clarke Date: Wed, 14 Oct 2026 19:57:45 +0100 Subject: [PATCH 3/3] Size list-endpoint pages to fit the import cap Phase 1 still recommended 50-250 records per page unconditionally, which contradicts the large-imports guidance: a ~92,700-object tenant at 250 per page needs ~371 calls, well over the ~220-call cap. The range is now a starting point only when it keeps the largest environment under ~200 pages, with larger inventories pointed at a bigger pageSize and lean rows. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/build-plugin/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/build-plugin/SKILL.md b/.claude/skills/build-plugin/SKILL.md index 61c6ffc0..bf98e0c4 100644 --- a/.claude/skills/build-plugin/SKILL.md +++ b/.claude/skills/build-plugin/SKILL.md @@ -62,7 +62,7 @@ Before writing a single file, understand and explore the API. **Use `AskUserQues 1. **Find the docs** — Gather URLs or spec files from the user, then fetch and read them. 2. **Identify the object model** — What are the core entities? (e.g. installations, devices, sites). These become the **indexed objects** in SquaredUp — available for drilldown, search, scoping dashboards, and use as variables. -3. **Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout. Note the page-size parameter's maximum, and estimate how many objects the largest environment holds (ask the user if unsure): an import run is capped at ~220 pages and ~6MB per page, so tens of thousands of objects constrain `pageSize` — see [index-defs.md](references/index-defs.md#large-imports--choosing-pagesize). +3. **Find the list endpoints** — Used to import objects. Note the page-size parameter's maximum, and estimate how many objects the largest environment holds (ask the user if unsure). An import run is capped at ~220 pages and ~6MB per page, with a per-page timeout. Start from **50–250 records per page** when that keeps the largest environment comfortably under ~200 pages; tens of thousands of objects need a larger `pageSize` with lean rows — see [index-defs.md](references/index-defs.md#large-imports--choosing-pagesize). 4. **Find the data endpoints** — These power data streams. For each, record three things; Phase 2 turns the last two into the stream's `timeframes`: - **Scoping** — scoped to a single object, multiple objects, or global (no object context). - **Time-range control** — does the endpoint accept a queryable time range at all (a `from`/`to`, `start`/`end`, or `period` parameter), or only return a fixed snapshot / current values?