Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 43 additions & 2 deletions administration/managing_custom_attestation_types/overview.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Managing Custom Attestation Types
description: Learn how to manage Kosli custom attestation types via Terraform, including creating and importing types with JSON Schema and jq evaluation rules.
description: Learn how to manage Kosli custom attestation types via Terraform, including creating and importing types with JSON Schema, jq evaluation rules, and summaries.
---

The preferred way to manage custom attestation types is via the <Tooltip tip="An official HashiCorp-registered Terraform provider that lets you manage Kosli resources (environments, flows, policies, etc.) as infrastructure as code." cta="View on Terraform Registry" href="https://registry.terraform.io/providers/kosli-dev/kosli/latest/docs/">Kosli Terraform provider</Tooltip>, so your Kosli configuration is version-controlled alongside your infrastructure. You can also manage custom attestation types through the Kosli CLI.
Expand All @@ -13,8 +13,12 @@ Custom attestation types define how Kosli validates evidence from tools that don

- A **JSON Schema** (optional) that defines the expected structure of attestation data
- **jq rules** (optional) that evaluate the data to determine compliance
- A **summary** (optional) — ordered, labeled jq expressions that Kosli renders as rows on the attestation detail page

At least one of the two must be provided.
At least one of the schema or the jq rules must be provided. The summary is independent of both:
it only affects how attestations of the type are displayed, never whether they are compliant. See
[Summarizing custom attestations](/getting_started/attestations#summarizing-custom-attestations) for
how summaries render.

## Create a custom attestation type

Expand Down Expand Up @@ -72,6 +76,43 @@ resource "kosli_custom_attestation_type" "deployment_record" {
}
```

### With a summary

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion — this heading reads as a fourth mutually-exclusive option.

The three preceding ### headings under Create a custom attestation type enumerate the alternatives of one choice: With schema and jq rules / With jq rules only / With schema only. With a summary is orthogonal — it combines with any of the three, and the example below in fact pairs it with jq rules only.

The body text at line 18-21 makes the independence clear, but a reader scanning the right-hand ToC sees four peers and infers a fourth alternative. Consider ### Adding a summary (verb phrase breaks the "With …" series), or promoting it to its own ## after the three shape variants.


`summary` takes a JSON array of `{name, expression}` objects. Each expression is a jq expression
evaluated against the attestation data when the attestation is displayed, and the entries render in
the order given. A string value beginning with `http://` or `https://` renders as a clickable link.

```hcl
resource "kosli_custom_attestation_type" "vulnerability_scan" {
name = "vulnerability-scan"
description = "Validates vulnerability scan results"

jq_rules = [".critical_vulnerabilities == 0"]

summary = jsonencode([
{ name = "Critical", expression = ".critical_vulnerabilities" },
{ name = "High", expression = ".high_vulnerabilities" },
{ name = "Scanner", expression = ".scanner_version" },
{ name = "Report", expression = ".report_url" },
])
}
```

Use `file()` instead of `jsonencode()` to keep the summary in a standalone JSON file, so the same
definition can be shared with other tooling:

```hcl
summary = file("${path.module}/summaries/vulnerability-scan.json")
```

<Note>
The summary is part of the versioned type definition, so changing it creates a new version of the
attestation type — exactly like changing the schema or the jq rules. Removing `summary` from a type
that had one clears the summary on the new version, so attestations reported against it fall back to
showing the jq evaluation results as a pass/fail checklist. Attestations reported against an earlier
version keep the summary that version defined.
</Note>
Comment on lines +108 to +114

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical — this contradicts the versioning statement added in getting_started/attestations.md.

Here: "changing it creates a new version of the attestation type ... Removing summary from a type that had one clears the summary, and its attestations fall back to showing the jq evaluation results as a pass/fail checklist."

In getting_started/attestations.md:388-390: "Changing it creates a new version of the type ... Each attestation keeps the version of the type it was reported against."

Both can't be true for pre-existing attestations. If attestations are pinned to the version they were reported against, then removing the summary only affects attestations reported after the change — existing ones keep rendering their summary.

Suggested change
<Note>
The summary is part of the versioned type definition, so changing it creates a new version of the
attestation type — exactly like changing the schema or the jq rules. Removing `summary` from a type
that had one clears the summary, and its attestations fall back to showing the jq evaluation results
as a pass/fail checklist.
</Note>
<Note>
The summary is part of the versioned type definition, so changing it creates a new version of the
attestation type — exactly like changing the schema or the jq rules. Removing `summary` from a type
that had one clears the summary on the new version, so attestations reported against it fall back to
showing the jq evaluation results as a pass/fail checklist. Attestations reported against an earlier
version keep the summary that version defined.
</Note>

Worth confirming the actual behavior with engineering before applying — if the detail page renders against the current type version rather than the pinned one, then it's the sentence in attestations.md that needs fixing instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Settled from the implementation rather than by guessing: src/model/attestations_model/attestation.py resolves find_by_id(self.type_id).get_versioned_type(self.type_version).evaluate_summary(...), so the detail page renders against the pinned version, not the current one. The handover doc for kosli-dev/server#6145 lists "render/pinning tests" too.

So the sentence in attestations.md was the correct one, and your suggested rewording of this <Note> is right — applied as written, plus I made the same point explicit in attestations.md: "Each attestation is displayed using the version of the type it was reported against, so editing a summary does not change how existing attestations render."

Same source turned up one thing neither of us had right: link rendering is narrower than "a valid URL". The hardening commit on #6428 requires a string value with an http/https scheme, specifically so non-string values don't 500 and javascript:/data: values can't become clickable — which supersedes the is_valid_url annotation convention the issue mentioned. Both pages now say http:// or https:// instead.


## Import an existing custom attestation type

If you have custom attestation types created via the CLI, you can bring them under Terraform management by importing them into your <Tooltip tip="The Terraform state file tracks the mapping between your configuration and real-world resources. Importing adds an existing resource to this state without recreating it.">Terraform state</Tooltip>.
Expand Down
65 changes: 61 additions & 4 deletions getting_started/attestations.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,15 +329,15 @@ Currently, we support the following types of evidence:
You could create a custom attestation type called `coverage-metrics` using a [jq expression](https://jqlang.org/manual/) rule defining a minimum line coverage of 95%:

```bash
kosli create attestation-type coverage-metrics
kosli create attestation-type coverage-metrics \
--jq=".code.lines.missed / .code.lines.total * 100 <= 5"
```

You could then make your custom attestation with the json file:
```bash
kosli attest custom
--type=coverage-metrics
--attestation-data=unit-test-coverage.json
kosli attest custom \
--type=coverage-metrics \
--attestation-data=unit-test-coverage.json \
...
```

Expand All @@ -349,6 +349,10 @@ Currently, we support the following types of evidence:
- So `32 / 1209 * 100 <= 5` evaluates to `2.64 <= 5` which is `true`


A custom attestation type can also define a **summary**, so that attestations of the type show
named values from their data instead of only raw JSON. See
[Summarizing custom attestations](#summarizing-custom-attestations) below.

See:
* [create custom attestation type](/client_reference/kosli_create_attestation-type) and
* [report custom attestation to an artifact or a trail](/client_reference/kosli_attest_custom/) for usage details and examples.
Expand All @@ -364,3 +368,56 @@ Currently, we support the following types of evidence:
</Accordion>
</AccordionGroup>

## Summarizing custom attestations

By default, the attestation detail page in Kosli shows a custom attestation type's evaluation rules
as a pass/fail checklist, plus the raw attestation data. Add a **summary** to the type to pull named
values out of the attestation data and display them as labeled rows instead — the way the built-in
Sonar, Snyk and JUnit attestations do.

A summary is an ordered list of `name`/`expression` pairs, where each expression is a
[jq expression](https://jqlang.org/manual/) evaluated against the attestation data. Add one entry
per `--summary` flag:

```bash
kosli create attestation-type coverage-metrics \
--jq=".code.lines.missed / .code.lines.total * 100 <= 5" \
--summary="Lines missed=.code.lines.missed" \
--summary="Lines total=.code.lines.total"
```

Attestations of the `coverage-metrics` type then show `Lines missed` and `Lines total`, in that
order, with the values taken from each attestation's own data. Re-running `kosli create
attestation-type` for a name that already exists updates that type rather than creating a second
one — see the note on versioning below.

Each `--summary` value is split on its first `=` only, so `==` inside a jq expression is safe. Use
[`--summary-json`](/client_reference/kosli_create_attestation-type) instead if the list is easier to
express as JSON, or the
[`summary` attribute](/terraform-reference/resources/custom_attestation_type) if you manage your
types with Terraform:

```json
[
{ "name": "Lines missed", "expression": ".code.lines.missed" },
{ "name": "Lines total", "expression": ".code.lines.total" }
]
```

Worth knowing:

- The summary is part of the **versioned** type definition. Changing it creates a new version of the
type, just like changing the schema or the evaluation rules. Each attestation is displayed using
the version of the type it was reported against, so editing a summary does not change how existing
attestations render.
- Expressions are evaluated when the attestation is **displayed**, against the stored attestation
data. A summary never affects the compliance status of an attestation.
- Invalid jq is rejected when the type is created, so authoring mistakes surface immediately rather
than as a broken detail page later.
- An expression that returns `null`, or that fails against a particular attestation's data, renders
as `N/A`. The rest of the summary still renders.
- A value that is a string beginning with `http://` or `https://` renders as a clickable link, as

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement — three different descriptions of the same linkification rule.

This says a value linkifies when it's "a string beginning with http:// or https://", and asserts it works "as annotation values do" — but line 246 of this same file says annotation values linkify when they are "valid URLs", and terraform-reference/resources/custom_attestation_type.mdx:189 says "values that are valid URLs render as links". overview.md:83 repeats the narrow phrasing.

The two rules differ for real inputs (ftp://…, mailto:…, a bare example.com), so one of them is wrong. Pick the accurate one and use it in all three places:

  • if the scheme check is the real behavior, line 246 and the Terraform provider description need updating (the latter upstream in the provider repo, since that page is generated);
  • if "valid URL" is the real behavior, this line and overview.md:83 should say that.

Fix this →

[annotation values](#annotating-attestations) do. Other values render as text.
- If the attestation data is a top-level array, Kosli renders one summary group per element
(`Summary 1:`, `Summary 2:`, and so on). Write the expressions against a single element
(`.code.lines.missed`) — each element is evaluated separately.
10 changes: 8 additions & 2 deletions tutorials/attest_custom.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,20 @@ Before you can report a custom attestation, the type referenced by `--type` must
* **CLI** — [`kosli create attestation-type`](/client_reference/kosli_create_attestation-type) (good for quick experiments).
* **Terraform** — the [`kosli_custom_attestation_type` resource](/terraform-reference/resources/custom_attestation_type) (recommended so the type is version-controlled).

For this tutorial we'll create a minimal `coverage-report` type that requires a `coverage` field of at least 80:
For this tutorial we'll create a small `coverage-report` type that requires a `coverage` field of at least 80:

```shell
kosli create attestation-type coverage-report \
--description "Code coverage report" \
--jq '.coverage >= 80'
--jq '.coverage >= 80' \
--summary "Coverage=.coverage" \
--summary "Tool=.tool"
```

The `--summary` entries are optional. Each is a `'NAME=EXPRESSION'` pair, so the attestation you
report below opens on a readable `Coverage: 92` / `Tool: pytest-cov` summary instead of raw JSON —
see [Summarizing custom attestations](/getting_started/attestations#summarizing-custom-attestations).

Prepare a JSON file with the data you want to attest. Save it as `coverage.json`:

```json
Expand Down