Sync ess- items verbatim and add ess-scan hook

.cursor/rules/cli-write-safety.mdc → .cursor/rules/ess-cli-write-safety.mdc

@@ -1,6 +1,6 @@
 ---
 description: Every CLI in this repo that writes or destroys must support --dry-run, confirm by default, and accept --yes
-alwaysApply: true
+globs: **/tools/**, **/cli.py, **/commands/**, **/*.sh, **/__main__.py
 ---
 
 # CLI Write Safety
@@ -46,10 +46,7 @@ Must not prompt, and must not offer `--dry-run` — there is nothing to preview.
 ## Where the flags live
 
 Put `--dry-run` on the group when it governs every subcommand, and `--yes` on each write
-command. `keys delete` in `packages/python/ess-langsmith-client/src/ess_langsmith_client/tools/keys.py`
-shows the confirmation half: it lists the keys it will delete, prompts by default, and
-accepts `--yes`. It does not yet offer `--dry-run`, and its prompt goes to stdout; do not
-copy those two gaps. The complete shape:
+command. The complete shape:
 
 ```python
 @click.option("--dry-run", is_flag=True, default=False,

.gitignore

@@ -272,6 +272,9 @@ Pulumi.*.yaml
 # Output files
 output/
 
+# Node.js
+node_modules/
+
 # Claude
 .claude/settings.json
 

.pre-commit-config.yaml

@@ -1,9 +1,9 @@
 repos:
   # Ruff for Python linting and formatting
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.8.6
+    rev: v0.15.6
     hooks:
-      - id: ruff
+      - id: ruff-check
         args: [--fix, --exit-non-zero-on-fix]
         exclude: ^.*/migrations/
         stages: [pre-push]

.pre-commit-hooks.yaml

@@ -0,0 +1,13 @@
+- id: ess-scan
+  name: ess-scan (secrets, jargon, PII in ess-* items)
+  description: >-
+    Runs the essentials-sync deterministic scanners on each changed ess-* item
+    and fails on critical findings. Needs Node.js 22 or later; trufflehog is
+    used when installed.
+  entry: ess-scan
+  language: node
+  # Use the Node on PATH. Otherwise pre-commit skips a Node under $HOME (nvm)
+  # and downloads one through nodeenv, which fails on prerelease versions.
+  language_version: system
+  files: '(^|/)ess-[^/]+/|(^|/)ess/ess-[^/]+$'
+  require_serial: true

examples/python/ess-hello-jwt-auth-code/pyproject.toml

@@ -3,8 +3,8 @@ name = "ess-hello-jwt-auth-code"
 version = "0.1.0"
 description = "Obtain a JWT access token from any OpenID Connect provider via the OAuth 2.0 Authorization Code Flow with PKCE"
 readme = "README.md"
+license = "Apache-2.0"
 requires-python = ">=3.12,<3.13"
-license = { text = "Apache-2.0" }
 keywords = ["oauth2", "oidc", "jwt", "pkce", "authorization-code"]
 dependencies = [
     "httpx>=0.28.0",
@@ -18,7 +18,7 @@ ess-hello-jwt-auth-code = "ess_hello_jwt_auth_code.main:main"
 dev = ["ruff>=0.14.2", "pytest>=8.0.0"]
 
 [build-system]
-requires = ["hatchling"]
+requires = ["hatchling>=1.27"]
 build-backend = "hatchling.build"
 
 [tool.hatch.build.targets.wheel]

examples/python/ess-hello-jwt-auth-code/src/ess_hello_jwt_auth_code/test_main.py

@@ -1,4 +1,4 @@
-# ruff: noqa: S106, PLR2004 -- test credentials are not real secrets; magic numbers expected
+# ruff: noqa: S106, PLR2004, B110 -- test credentials are not real secrets; magic numbers expected; server may close before response is read
 """Tests for the OAuth 2.0 Authorization Code Flow."""
 
 from __future__ import annotations

examples/python/ess-messages/.env.example

@@ -1,8 +1,15 @@
-# Outlook (Microsoft Graph)
+# Outlook (Microsoft Graph). CLIENT_ID and TENANT_ID are required for the
+# email source unless GRAPH_ACCESS_TOKEN is set; there is no default client ID.
 # CLIENT_ID: Microsoft Entra app registration (e.g., the "Office Desktop Apps" public client)
 CLIENT_ID=
 # TENANT_ID: your organization's Microsoft Entra (Azure AD) tenant ID
 TENANT_ID=
+# Optional: a token from https://developer.microsoft.com/graph/graph-explorer,
+# for tenants where Conditional Access blocks the device-code flow (~1h TTL)
+# GRAPH_ACCESS_TOKEN=
+# Optional: browser sign-in instead of device code. Requires CLIENT_ID to be an
+# app registration that lists "http://localhost" as a redirect URI.
+# GRAPH_INTERACTIVE_AUTH=1
 
 # Webex OAuth integration (from https://developer.webex.com/my-apps)
 WEBEX_CLIENT_ID=

examples/python/ess-messages/README.md

@@ -33,13 +33,28 @@ uv sync --all-packages
 ## Auth
 
 Requires credentials for one or both sources. See:
-- [Outlook setup](../../../packages/python/ess-outlook/.env.example) -- `CLIENT_ID`, `TENANT_ID`
+- [Outlook setup](../../../packages/python/ess-outlook/README.md#configuration) --
+  `CLIENT_ID` and `TENANT_ID`, both required unless `GRAPH_ACCESS_TOKEN` is set
 - [Webex setup](../../../packages/python/ess-webex/.env.example) -- `WEBEX_CLIENT_ID`/`WEBEX_CLIENT_SECRET` or `WEBEX_ACCESS_TOKEN`
 
 Copy this example's `.env.example` to `.env` and fill in values, or set the
 variables in your shell environment. Never commit `.env` -- it is meant to
 hold secrets.
 
+If Conditional Access blocks the Outlook device-code flow, set
+`GRAPH_ACCESS_TOKEN` to a token from
+[Graph Explorer](https://developer.microsoft.com/graph/graph-explorer) --
+see [Bring your own token](../../../packages/python/ess-outlook/README.md#bring-your-own-token).
+If Outlook authentication fails, the command exits unless you pass
+`--webex-only`, which skips Outlook and fetches Webex only. Within
+ess-outlook, multiple auth methods are tried before failure (supplied token,
+cached refresh, optional browser sign-in, then device code).
+
+For a permanent fix, ask your identity team for a dedicated app registration: a
+single-tenant public client with delegated `Mail.ReadWrite` on Microsoft Graph and
+`http://localhost` as a redirect URI. Set its Application (client) ID and Directory
+(tenant) ID as `CLIENT_ID` and `TENANT_ID`.
+
 ## Usage
 
 ```bash

examples/python/ess-messages/pyproject.toml

@@ -3,6 +3,7 @@ name = "ess-messages"
 version = "0.1.0"
 description = "Unified inbox -- aggregates email and Webex messages into Activity Streams 2.0"
 readme = "README.md"
+license = "Apache-2.0"
 requires-python = ">=3.12,<3.13"
 dependencies = [
     "activitystreams2>=0.5.0",
@@ -16,7 +17,7 @@ dependencies = [
 ess-messages = "ess_messages.__main__:main"
 
 [build-system]
-requires = ["hatchling"]
+requires = ["hatchling>=1.27"]
 build-backend = "hatchling.build"
 
 [tool.uv.sources]

instructions/ess/ess-cli-write-safety.instructions.md

@@ -0,0 +1,96 @@
+---
+description: Every CLI in this repo that writes or destroys must support --dry-run, confirm by default, and accept --yes
+applyTo: "**/tools/**,**/cli.py,**/commands/**,**/*.sh,**/__main__.py"
+---
+
+# CLI Write Safety
+
+Applies to **every command-line tool produced in this repo**, whatever the language or
+argument parser: Click groups, `argparse` scripts under `tools/`, Django management
+commands, TypeScript `bin` entries, and shell scripts.
+
+## The three requirements
+
+Any command that writes remote or local state, deletes, rotates a credential, or is
+otherwise not trivially reversible must have all three:
+
+1. **`--dry-run`** — runs the same code path, prints exactly what would change, exits 0,
+   and touches nothing. It is not a separate "plan" branch that can drift from the real
+   one; resolve the same targets and build the same payloads, then stop before the write.
+2. **Confirmation by default** — print a summary of what is about to happen (counts,
+   target names, account/stack/environment), then prompt. Abort on anything but yes.
+3. **`--yes`** (alias `-y`) — bypasses the prompt, and nothing else. It must never imply
+   or cancel `--dry-run`.
+
+## Use these exact flag names
+
+`--dry-run` and `--yes`. Do not invent `--apply`, `--submit`, `--commit`, `--force`,
+`--no-confirm`, or `--for-real`.
+
+Opt-in-to-write flags look safe but are not: they make the *safe* path the one the
+operator has to remember, and every tool picks a different word for it, so muscle memory
+from one tool does nothing for the next. Prompting by default gives the same protection
+without depending on recall.
+
+## Precedence and non-interactive runs
+
+- `--dry-run` wins if both flags are passed.
+- When a **real write** would occur, never silently proceed without explicit consent.
+  Without a TTY, or in `--json` mode, fail and tell the operator to pass `--yes`.
+  `--dry-run` skips this check because nothing is written.
+
+## Read-only commands
+
+Must not prompt, and must not offer `--dry-run` — there is nothing to preview.
+
+## Where the flags live
+
+Put `--dry-run` on the group when it governs every subcommand, and `--yes` on each write
+command. The complete shape:
+
+```python
+@click.option("--dry-run", is_flag=True, default=False,
+              help="Show planned changes without submitting them.")
+...
+@click.option("--yes", is_flag=True, default=False, help="Skip the confirmation prompt.")
+```
+
+with a single guard called by every write path:
+
+```python
+def _confirm_or_abort(options, account, locations, *, skip: bool) -> None:
+    """Require explicit consent before writing credentials."""
+    if skip or options.dry_run:
+        return
+
+    # Real write only; --dry-run returned above. --json has no interactive
+    # prompt to fall back on, so make the caller opt in rather than silently
+    # skipping the guard.
+    if options.as_json:
+        raise click.UsageError("--json requires --yes, which confirms the write.")
+
+    click.confirm(f"\nSet a new password for {account} in {len(locations)} location(s)?",
+                  abort=True, err=True)
+```
+
+For interactive **real writes**, prompt on **stderr** (`err=True`) so piping stdout to
+a file or `jq` still shows the question. Dry-run and `--yes` paths skip the prompt, so
+`--dry-run --json` without `--yes` is valid.
+
+`@click.confirmation_option(prompt=...)` gives you `--yes` for free and satisfies
+requirements 2 and 3, but you still owe `--dry-run`.
+
+For `argparse`, the equivalent is `--dry-run` / `--yes` store-true flags plus an explicit
+`sys.stdin.isatty()` check before `input()` for real writes (not `--dry-run`); fail with
+a message to pass `--yes` if non-interactive.
+
+## Tests (Click / Python)
+
+For Click-based Python CLIs, co-locate `test_*.py` per the repo's testing convention.
+Two cases at minimum, using Click's `CliRunner`:
+
+- `--dry-run` performs no writes (assert the client/session mock was never called).
+- `--yes` skips the prompt (assert the command succeeds with no stdin).
+
+TypeScript, shell, and Django commands must still prove the same three requirements,
+using that language's test tooling and conventions.

instructions/ess/ess-dockerfile-runtime-copy.instructions.md

@@ -0,0 +1,29 @@
+---
+description: Copy runtime image files as root-owned and not writable, so the process cannot modify its own code (SonarQube docker:S6504)
+applyTo: "**/Dockerfile,**/Dockerfile.*,**/*.dockerfile"
+---
+
+# Runtime image copies stay root-owned
+
+A container should not be able to rewrite its own application code. If the runtime user owns the files it runs, a compromised process can change them, which breaks least privilege and container immutability. SonarQube reports this as **docker:S6504**, on any `COPY` or `ADD` whose `--chown` user is not `root`.
+
+In the final stage, copy application files as root and drop the write bit. The runtime user can read and traverse them and cannot modify them.
+
+```dockerfile
+# Bad — node owns the app and can rewrite it
+COPY --from=build --chown=node:node /app/dist ./dist
+USER node
+
+# Good
+COPY --from=build --chown=root:root --chmod=555 /app/dist ./dist
+USER node
+```
+
+`--chmod=555` is read and execute for everyone, so directories stay traversable and files stay readable. The same flags apply whether the runtime user is `node`, `nextjs`, or any other non-root account.
+
+## Rules
+
+- On every runtime-stage `COPY` or `ADD` of application files, set `--chown=root:root --chmod=555`.
+- Do not `--chown` to a non-root user or a uid other than `0`.
+- A path the process must write (cache, uploads, temp) is a separate empty directory. Do not loosen the application copy to make that path writable.
+- Build-stage copies that the builder then writes, such as `package.json` before `npm ci`, stay default root and omit `--chmod=555`. Still do not `--chown` them to a non-root user.

instructions/ess/ess-docs-example-urls.instructions.md

@@ -0,0 +1,47 @@
+---
+description: >-
+  Use example.com, example.net, example.org, or example.edu, a subdomain of
+  one, or a name under the reserved .example, .invalid, or .test TLDs for every
+  illustrative host, URL, and email in code, tests, comments, and docs.
+applyTo: "**"
+---
+
+# Example URLs
+
+Placeholder hosts, URLs, and email addresses use a reserved name, which no one can
+register:
+
+| Reserved | Names |
+| -------- | ----- |
+| Documentation domains | `example.com`, `example.net`, `example.org`, `example.edu`, and their subdomains |
+| Top-level domains | Any name under `.example`, `.invalid`, or `.test` |
+
+RFC 2606 reserves `example.com`, `example.net`, `example.org`, and the three top-level
+domains. IANA holds all four documentation domains, and each serves a page that reads:
+"This domain is for use in documentation examples without needing permission." The
+top-level domains are never delegated, so names under them never resolve.
+
+| Placeholder for | Use | Not |
+| --------------- | --- | --- |
+| A remote host | `host.example.com` | `host.com`, `myhost` |
+| An identity provider | `idp.example.com` | `login.vendor.com` |
+| A deployed app | `app.example.com` | `my-app.awsapprunner.com` |
+| A person | `ada@example.com` | an address at `x.com` or a real company |
+| A second organization | `corp.example.net` | `corp.example.io` |
+| A host that must never resolve | `unreachable.invalid` | `nonexistent.com` |
+| A throwaway test identity | `test@example.invalid` | an address at `test.com` |
+
+- **Default to `example.com`.** Use `example.net`, `example.org`, or `example.edu` when an
+  example needs organizations that are visibly distinct.
+- **Use a reserved top-level domain when its meaning helps.** `.invalid` marks a value
+  that must never resolve or be delivered, such as an unreachable host in a failure test
+  or a git identity in a throwaway repo. `.test` suits hosts on a local test network, and
+  `.example` reads as an illustration.
+- **Only these.** Other `example.<tld>` names, such as `example.io` or `example.dev`, and
+  look-alikes such as `test.com`, are ordinary domains someone can register.
+- **Real endpoints stay real.** A host the code actually calls (`github.com`,
+  `graph.microsoft.com`, `api.smith.langchain.com`) is not a placeholder; leave it.
+- **Exception: a domain the code enforces.** When the code under test accepts only a
+  provider's domain, keep the provider suffix and make `example` the leftmost label. A
+  client that rejects any host outside `*.service-now.com` is tested with
+  `example.service-now.com`.

instructions/ess/ess-npmrc-env.instructions.md

@@ -0,0 +1,24 @@
+---
+description: In .npmrc, reference environment variables as ${VAR}. A trailing ? is not portable.
+applyTo: "**/.npmrc,**/.npmrc.*,**/npmrc"
+---
+
+# npmrc environment variables
+
+npm replaces `${VARIABLE_NAME}` in `.npmrc` values. The text inside the braces is the environment variable name, exactly.
+
+```ini
+//registry.example.com/:_authToken=${NPM_TOKEN}
+```
+
+Do not append `?`.
+
+npm 10, bundled with Node 20 and Node 22, has no optional form. It looks up a variable named `NPM_TOKEN?`, so exporting `NPM_TOKEN` leaves `${NPM_TOKEN?}` unexpanded and authenticated installs fail.
+
+```ini
+//registry.example.com/:_authToken=${NPM_TOKEN?}
+```
+
+npm 11 and later document `?` as "substitute an empty string when unset." That is still the wrong form for a required value such as an auth token: an unset variable becomes an empty token instead of a visible unreplaced placeholder. Write `${VARIABLE_NAME}` so the same file works on npm 10 and later.
+
+Keep the `${...}` reference. Never paste a token into a committed `.npmrc`. Export the variable before `npm ci`.

instructions/ess/ess-okf-docs.instructions.md

@@ -0,0 +1,19 @@
+---
+description: >-
+  When editing or creating markdown with OKF YAML frontmatter (type, generated,
+  tags, etc.), consult Context7 for the OKF v0.2 spec before guessing.
+applyTo: "**/*.md"
+---
+
+Use the Context7 MCP for the Open Knowledge Format spec. Context7 is a
+configured docs server in this repo. Do not call an `okf-spec` server; none
+is defined.
+
+1. Call `resolve-library-id` with library name `Open Knowledge Format` unless
+   you already have the id `/googlecloudplatform/open-knowledge-format`
+   (OKF v0.2, GoogleCloudPlatform/open-knowledge-format).
+2. Call `query-docs` with that library id. Keep each query to one frontmatter
+   concept (`type`, `title`, `description`, `tags`, `generated`).
+
+`type` is the only required key. Consumers must tolerate unknown types and
+preserve unknown keys.