From d09da6ded6b640bddc9bce056f1903413fd38a17 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 21 May 2026 09:09:34 +0200 Subject: [PATCH 1/5] tmp --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) create mode 120000 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 120000 index 0000000000..ac55cbdc9c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1 @@ +.claude/CLAUDE.md \ No newline at end of file From 38cc810b5eeaa2f129d6e1e77bd0c468ea48e2fd Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 3 Sep 2026 09:07:28 +0200 Subject: [PATCH 2/5] [HWORKS-3005] Document the API key scopes https://hopsworks.atlassian.net/browse/HWORKS-3005 The API key guide told the reader to "select the required scopes" but nothing in the docs said what any scope grants, so the only way to find out was to read the @ApiKeyRequired annotations in hopsworks-ee or hit a 403 320004 and read the scope name out of the error. Add a reference page listing every scope with the endpoint groups it unlocks, which scopes are privileged, which scopes each account role may select, the scopes hops setup grants, and what the 320004 error looks like. Link it from the create-key guide and the projects index, and add it to the nav under Api Keys. Signed-off-by: Jim Dowling Co-Authored-By: Claude Fable 5.1 --- .../projects/api_key/api_key_scopes.md | 76 +++++++++++++++++++ .../projects/api_key/create_api_key.md | 1 + docs/user_guides/projects/index.md | 1 + mkdocs.yml | 1 + 4 files changed, 79 insertions(+) create mode 100644 docs/user_guides/projects/api_key/api_key_scopes.md diff --git a/docs/user_guides/projects/api_key/api_key_scopes.md b/docs/user_guides/projects/api_key/api_key_scopes.md new file mode 100644 index 0000000000..ac6ccd4fe1 --- /dev/null +++ b/docs/user_guides/projects/api_key/api_key_scopes.md @@ -0,0 +1,76 @@ +# API Key Scopes + +Every API key carries a set of scopes. +A scope unlocks a group of REST endpoints; a request made with a key that lacks the scope an endpoint requires is rejected before it reaches the endpoint. +Scopes are chosen when a key is created and can be changed later from the key's edit page, without regenerating the secret. +See [How To Create An API Key][how-to-create-an-api-key] for the UI walkthrough. + +A scope never grants more than the account itself may do. +Endpoints still check the caller's role in the project, so a Data Scientist's key with the `FEATURESTORE` scope cannot do what a Data Owner's key with the same scope can. + +## Scope reference + +| Scope | Grants access to | +| --- | --- | +| `FEATURESTORE` | Feature stores and everything inside them: feature groups, feature views, training datasets, data sources and storage connectors, transformation functions, statistics, data validation, feature monitoring, tags, keywords, provenance and feature store search. Also Hopsworks actions and the tag schema catalogue; creating or deleting a tag schema additionally requires the `HOPS_ADMIN` role. | +| `PROJECT` | Project management: list, create, update and delete projects; read project information and client credentials; manage members; project alerts, receivers, routes and silences; cloud role mappings; the operation log; tutorials and product news. | +| `JOB` | Jobs and executions: create, update, schedule, start, stop and delete jobs; read execution logs; default job configurations; job alerts and tags; Python apps; expectation suites and validation reports. | +| `DATASET_VIEW` | Read access to project datasets: list datasets, browse and download files, and use global, project and dataset search. | +| `DATASET_CREATE` | Create datasets and directories, upload files, and copy, move, zip or unzip them. | +| `DATASET_DELETE` | Delete datasets, directories and files. | +| `MODELREGISTRY` | Model registries and models: register, update and delete models; model tags and provenance; Hugging Face imports; generated deployment configurations. | +| `SERVING` | Model deployments: create, start, stop and delete deployments; read deployment logs; send inference requests; deployment tags; OpenTelemetry traces and metrics. | +| `KAFKA` | The project's Kafka topics and schema registry: topics, subjects, schema versions and compatibility settings. Also accepted, as an alternative to `FEATURESTORE` or `PROJECT`, by the few read endpoints a Kafka or OnlineFS client needs, such as listing projects and feature stores. | +| `PYTHON_LIBRARIES` | Python environments: list, create and delete environments; install and uninstall pip, conda and npm packages; search package indexes; environment build commands, history and conflicts. | +| `GIT` | Git repositories in the project: clone, branches, commits, remotes, repository actions and their executions, and the account's Git provider credentials. | +| `TRINO` | The Trino query engine: submit and cancel SQL statements, read query, worker and cluster status, and manage Trino catalogs. | +| `SUPERSET` | Superset dashboards: log in to Superset, list dashboards, create permalinks, make a dashboard public or share it with another project, and delete dashboards. | +| `TERMINAL` | The web terminal: start, extend, stop and inspect terminal sessions, and mint the proxy tokens used to attach to them. | +| `MOUNTABLE_SECRET` | The project's mountable secrets: named bundles of credential files (Oracle wallets, JKS keystores, service account JSON) that a service mounts read-only. Create, list and delete bundles. Contents are never returned. Requires the Data Owner role. | +| `USER` | The account itself: profile, secrets, account environment variables, AI provider settings, and API keys. A key with this scope can create, edit and delete API keys, including keys carrying any other scope the account is allowed to hold, so treat it as equivalent to all of them. | +| `ADMIN` | Cluster administration: the admin API (configuration variables, backups, projects, users, Trino, TTL purge, coding agent configuration, cloud role mappings, search reindexing, the operation log), compute resources and the UI theme. Privileged. | +| `ADMINISTER_USERS` | User administration in the admin API: list, accept, reject, block, modify and delete users, change roles, reset passwords and sync remote groups. Privileged. | +| `ADMINISTER_USERS_REGISTER` | Only the user registration endpoint of the admin API. Privileged. | +| `AUTH` | The JWT service: issue, renew and invalidate tokens and remove signing keys. Privileged, and also available to accounts in the `AGENT` group. | +| `KUBE` | Reserved. No REST endpoint currently accepts it. | +| `SINK` | Reserved for feature groups that ingest from a data source through a DLT sink. No REST endpoint currently accepts it. | + +## Privileged scopes + +`ADMIN`, `ADMINISTER_USERS`, `ADMINISTER_USERS_REGISTER` and `AUTH` are privileged. +Only accounts with the `HOPS_ADMIN` role can create keys carrying them, because the endpoints they unlock act on the whole cluster rather than on a project the caller is a member of. + +## Scopes an account can select + +The set of scopes offered when creating or editing a key depends on the account's role. + +| Account role | Selectable scopes | +| --- | --- | +| `HOPS_ADMIN` | All scopes. | +| `HOPS_USER` | All unprivileged scopes. | +| `AGENT` | All unprivileged scopes plus `AUTH`. | +| `HOPS_SERVICE_USER` | All unprivileged scopes except `GIT` and `KUBE`. | + +The API key form preselects `FEATURESTORE`, `PROJECT`, `JOB`, `DATASET_VIEW`, `DATASET_CREATE`, `DATASET_DELETE`, `KAFKA`, `SERVING`, `MODELREGISTRY`, `USER` and `PYTHON_LIBRARIES`. +Deselect what the key's consumer does not need. + +## Scopes of a key created by hops setup + +`hops setup` creates its key through the browser token flow rather than the API key form, so the scopes are not chosen interactively. +The key carries every scope a `hops` subcommand needs: `FEATURESTORE`, `PROJECT`, `JOB`, `DATASET_VIEW`, `DATASET_CREATE`, `DATASET_DELETE`, `MODELREGISTRY`, `SERVING`, `USER`, `KAFKA`, `TERMINAL`, `PYTHON_LIBRARIES`, `GIT`, `TRINO` and `SUPERSET`. +A key created by an older release lacks the last six; edit it in the UI to add them, or run `hops setup --force` to mint a new one. + +## Scope errors + +A request made with a key that lacks the required scope fails with HTTP 403 and error code 320004. +The message names the scope the endpoint accepts. + +```json +{ + "errorCode": 320004, + "usrMsg": "No valid scope found for this invocation. Valid scope for this invocation is: [PYTHON_LIBRARIES]", + "errorMsg": "No valid scope found for this invocation" +} +``` + +Add the named scope to the key from the _API_ section of _Account Settings_, or create a new key that has it. diff --git a/docs/user_guides/projects/api_key/create_api_key.md b/docs/user_guides/projects/api_key/create_api_key.md index bcf750e52e..b80b598522 100644 --- a/docs/user_guides/projects/api_key/create_api_key.md +++ b/docs/user_guides/projects/api_key/create_api_key.md @@ -33,6 +33,7 @@ Keys with no expiration show _Never_ in the expiration column. ### Step 2: Create an API Key Click `New API key`, enter a name, optionally set an expiration, select the required scopes, and click `Create API key`. +Each scope unlocks a group of REST endpoints; see [API Key Scopes][api-key-scopes] for what every scope grants. **Expiration options:** diff --git a/docs/user_guides/projects/index.md b/docs/user_guides/projects/index.md index d8ac87980f..1c51444b9b 100644 --- a/docs/user_guides/projects/index.md +++ b/docs/user_guides/projects/index.md @@ -5,6 +5,7 @@ This section serves to provide guides and examples for the common usage of servi - [Projects](project/create_project.md) - [Authentication](auth/login.md) - [API Keys](api_key/create_api_key.md) +- [API Key Scopes](api_key/api_key_scopes.md) - [Jupyter](jupyter/python_notebook.md) - [Apps](apps/index.md) - [Jobs](jobs/pyspark_job.md) diff --git a/mkdocs.yml b/mkdocs.yml index b53bc4becc..f351cf7ff8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -202,6 +202,7 @@ nav: - Account-level Environment Variables: user_guides/projects/env_vars/create.md - Api Keys: - Create API Key: user_guides/projects/api_key/create_api_key.md + - API Key Scopes: user_guides/projects/api_key/api_key_scopes.md - AWS IAM Roles: user_guides/projects/iam_role/iam_role_chaining.md - Query Engine (Trino): user_guides/projects/trino/query_engine.md - Superset: user_guides/projects/superset/superset.md From 13e3b958c522b57770702c29420d55d673ef7245 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 3 Sep 2026 09:17:00 +0200 Subject: [PATCH 3/5] [HWORKS-3005] Agent guides: replace the stale docs setup commands https://hopsworks.atlassian.net/browse/HWORKS-3005 The command blocks in .claude/CLAUDE.md (AGENTS.md is a symlink to it) and .claude/docs/README.md still installed from a requirements-docs.txt that no longer exists and built with a hand-made docs/javadoc stub, which fails strict mode with a nav warning. CI uses the repo's own uv project and hopsworks-docs CLI, so list those: uv sync --extra cli, the sibling hopsworks-api install, and hopsworks-docs check / serve / markdownlint / snakeoil / linkchecker. Signed-off-by: Jim Dowling Co-Authored-By: Claude Fable 5.1 --- .claude/CLAUDE.md | 13 +++++++------ .claude/docs/README.md | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 8bbf2dbc3a..a9a181e635 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -3,12 +3,13 @@ ## Commands ```bash -uv venv && uv pip install -r requirements-docs.txt # setup -uv pip install "git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python" # install Python API (needed for API docs section) -touch docs/javadoc; uv run mkdocs build -s; rm docs/javadoc # build (strict) -uv run mkdocs serve # preview with live reload -npx markdownlint-cli2 "**/*.md" # lint Markdown (requires Node.js) -uv tool install md-snakeoil && snakeoil --line-length 88 --rules "E,F,B,C4,ISC,PIE,PYI,Q,RSE,RET,SIM,TC,I,W,D2,D3,D4,INP,UP,FA" docs # lint Python code blocks +uv sync --extra cli # setup: mkdocs plus the hopsworks-docs CLI in .venv +uv pip install ../hopsworks-api/python # Python API for the API docs section; CI clones hopsworks-api next to this repo +uv run hopsworks-docs check # build in strict mode (creates the docs/javadoc stub itself) +uv run hopsworks-docs serve # preview with live reload +uv run hopsworks-docs markdownlint # lint Markdown +uv run hopsworks-docs snakeoil # lint Python code blocks (ruff at line length 88) +uv run hopsworks-docs linkchecker # check for broken links ``` ## Rules diff --git a/.claude/docs/README.md b/.claude/docs/README.md index 64e4a9f278..0378612aa4 100644 --- a/.claude/docs/README.md +++ b/.claude/docs/README.md @@ -6,12 +6,13 @@ There is no application code — all work is writing Markdown under `docs/` and ## Commands ```bash -uv venv && uv pip install -r requirements-docs.txt # setup -uv pip install "git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python" # needed for Python API section -touch docs/javadoc; uv run mkdocs serve; rm docs/javadoc # preview with live reload -touch docs/javadoc; uv run mkdocs build -s; rm docs/javadoc # build in strict mode -npx markdownlint-cli2 "**/*.md" # lint Markdown (requires Node.js) -uv tool install md-snakeoil && snakeoil --line-length 88 --rules "E,F,B,C4,ISC,PIE,PYI,Q,RSE,RET,SIM,TC,I,W,D2,D3,D4,INP,UP,FA" docs # lint Python code blocks +uv sync --extra cli # setup: mkdocs plus the hopsworks-docs CLI in .venv +uv pip install ../hopsworks-api/python # Python API for the API docs section; CI clones hopsworks-api next to this repo +uv run hopsworks-docs check # build in strict mode (creates the docs/javadoc stub itself) +uv run hopsworks-docs serve # preview with live reload +uv run hopsworks-docs markdownlint # lint Markdown +uv run hopsworks-docs snakeoil # lint Python code blocks (ruff at line length 88) +uv run hopsworks-docs linkchecker # check for broken links ``` `docs/javadoc` is a directory generated by CI from the `hopsworks-api` Java source. From 7dd5442330ef54279bb4803cbf434d0477172471 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 3 Sep 2026 10:05:35 +0200 Subject: [PATCH 4/5] [HWORKS-3005] Scope reference: drop the removed KUBE and SINK scopes https://hopsworks.atlassian.net/browse/HWORKS-3005 hopsworks-ee removes both scopes, since no endpoint ever accepted them, so the reference no longer lists them as reserved and the service-user row no longer excludes KUBE. Signed-off-by: Jim Dowling Co-Authored-By: Claude Fable 5.1 --- docs/user_guides/projects/api_key/api_key_scopes.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/user_guides/projects/api_key/api_key_scopes.md b/docs/user_guides/projects/api_key/api_key_scopes.md index ac6ccd4fe1..297545f037 100644 --- a/docs/user_guides/projects/api_key/api_key_scopes.md +++ b/docs/user_guides/projects/api_key/api_key_scopes.md @@ -32,8 +32,6 @@ Endpoints still check the caller's role in the project, so a Data Scientist's ke | `ADMINISTER_USERS` | User administration in the admin API: list, accept, reject, block, modify and delete users, change roles, reset passwords and sync remote groups. Privileged. | | `ADMINISTER_USERS_REGISTER` | Only the user registration endpoint of the admin API. Privileged. | | `AUTH` | The JWT service: issue, renew and invalidate tokens and remove signing keys. Privileged, and also available to accounts in the `AGENT` group. | -| `KUBE` | Reserved. No REST endpoint currently accepts it. | -| `SINK` | Reserved for feature groups that ingest from a data source through a DLT sink. No REST endpoint currently accepts it. | ## Privileged scopes @@ -49,7 +47,7 @@ The set of scopes offered when creating or editing a key depends on the account' | `HOPS_ADMIN` | All scopes. | | `HOPS_USER` | All unprivileged scopes. | | `AGENT` | All unprivileged scopes plus `AUTH`. | -| `HOPS_SERVICE_USER` | All unprivileged scopes except `GIT` and `KUBE`. | +| `HOPS_SERVICE_USER` | All unprivileged scopes except `GIT`. | The API key form preselects `FEATURESTORE`, `PROJECT`, `JOB`, `DATASET_VIEW`, `DATASET_CREATE`, `DATASET_DELETE`, `KAFKA`, `SERVING`, `MODELREGISTRY`, `USER` and `PYTHON_LIBRARIES`. Deselect what the key's consumer does not need. From c40bf745bbc1d423bb8a31c52ca6e9ce57571cec Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 12 Sep 2026 09:50:13 +0200 Subject: [PATCH 5/5] [HWORKS-3005] Give the search page's scopes heading its own anchor Adding the API Key Scopes page gave two headings the same id, so mkdocs_autorefs could not pick a primary URL for `api-key-scopes` and the strict build aborted. The new page is the one `create_api_key.md` links to, so it keeps the canonical anchor and the search subsection takes an explicit qualified id, the way `search.md` in concepts already does with `#search-concept`. Nothing links to the search anchor. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/projects/search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guides/projects/search.md b/docs/user_guides/projects/search.md index 7d76fc2f9e..caef863042 100644 --- a/docs/user_guides/projects/search.md +++ b/docs/user_guides/projects/search.md @@ -87,7 +87,7 @@ Without any of them it is rejected with a `422`, because there is no "match ever The response carries one bucket per class, each with its own total, for example `featuregroups` with `featuregroupsTotal` and `apps` with `appsTotal`. -### API key scopes +### API key scopes { #search-api-key-scopes } Search results are filtered to the scopes of the API key you use, so a key cannot discover a class it was not minted for. A `FEATURESTORE` key sees feature groups, feature views, training datasets and features, `JOB` sees jobs and apps, `MODELREGISTRY` sees models, and `SERVING` sees deployments and agents.