feat(mcp): add HTTP JWT delegation

[nmartorell]

Overview

This PR adds a production-oriented Streamable HTTP deployment mode while preserving the existing local stdio workflow.

In stdio mode, the MCP server remains a local process configured with DSS API keys. In HTTP mode, it can be deployed centrally behind HTTPS, expose an administrator-defined catalog of reachable DSS instances, and authenticate multiple users through an enterprise identity provider.

Authentication and request flow

HTTP callers authenticate with an access token issued for the MCP server. For each tool call, the server identifies the caller, snapshots that user’s selected DSS instance, and exchanges the incoming token for a delegated token accepted by that DSS instance. The delegated token and request identity exist only for the lifetime of the call.

The HTTP configuration supports Microsoft Entra ID v2 on-behalf-of delegation and a generic OIDC flow using RFC 8693 token exchange. Interactive OAuth login can be enabled for agent harnesses, while custom clients can supply bearer tokens directly.

Tools that only manage MCP-local state do not require DSS token exchange. Retained Cobuild conversations are isolated by authenticated user and instance, and their DSS client is refreshed with the delegated token from each subsequent request.

Configuration and startup

The configuration layer is split by transport but uses shared validated models and file handling:

• stdio uses locally managed DSS profiles containing API keys;
• HTTP uses centrally managed DSS instances, identity-provider settings, delegation settings, and per-user instance selections;
• both transports use the same explicit launcher interface and load their configuration once during startup;
• configuration writes are atomic, credential files are protected from accidental commits, and runtime representations avoid exposing secrets.

The HTTP deployment guide documents the architecture, Entra setup, generic RFC 8693 setup, and the DSS JWT trust configuration required for delegated access.

Validation

• full automated test suite
• pre-commit and formatting checks
• license-header validation
• locked dependency validation
• source distribution and wheel build

Review

To facilitate the review, I setup a remote MCP server connected to the ai-dev-kit instance. If reviewing with codex, first register and login to the mcp server:

codex mcp add dataiku-http --url https://mcp.nedmartorell.fe-aws.dkucloud-dev.com/mcp
codex mcp login dataiku-http

Then start codex, optionally stop the local mcp server version (by disabling the plugin), and then run some functional tests (e.g. list instances, switch instance, count projects, etc.).

Note: due to the issue with jwt<>dss user mapping (which will be fixed in DSS 15.0.2), I will need to create a user for you manually on the instance with the EntraID sub present in your jwt tokens. It is not possible for me to find these out prior to them being issued by Entra, so we will probably have to do this together (as I'm printing the jwt sub in the mcp logs temporarily).

It would also be good to manually test the stdio server, by e.g. checking out the repo on this branch, and installing headless as a local plugin. Just to make sure nothing is broken in the stdio setup (I've checked, but would be good for someone else to check too).

[yjouini]

Nice implementation @nmartorell ! Two questions I had :

• How are local file operations handled in HTTP mode? It looks like an authenticated caller could pass a server-local path to an upload tool and copy that file into a DSS project they control. Similarly, could export_dataset(overwrite=True) overwrite files on the MCP host? if yes, should we have an isolated upload/download mechanism
• What happens to existing stdio configurations on upgrade? Do we need migration guidance for the config.json → stdio-config.json rename and removal of DKU_CONFIG_FILE?

[nmartorell]

@yjouini Addressed in 886b0fa.

• Streamable HTTP now rejects caller-controlled host-path operations: dataset export, managed-folder and project-library uploads, and plugin updates. HTTP dataset creation accepts bounded structured rows only (10,000 max); stdio keeps local-file support.
• Stdio now rejects DKU_CONFIG_FILE with --settings-path guidance. Canonical stdio-config.json takes precedence; valid legacy config.json files migrate atomically, while invalid or unrelated legacy files are ignored so other software can continue using them.
• Added upgrade and transport documentation plus focused coverage for guards, row validation and cleanup, and migration behavior.

Validation: pre-commit, 430 tests, and package build all pass.

[pmasiphelps]

I split my comments and suggested changes into two stacked PRs because some are more exploratory than others:

• #155 is the more straightforward one. It updates user-facing “DSS” terminology to “Dataiku,” introduces personal stdio vs. customer-managed HTTP in the README, and routes the dataiku-headless-setup skill to separate setup references because the two flows are quite different.
• #156 explores an easier distribution path for customer-managed HTTP. Command-line setup may be too much for some users, so it adds a generator that creates a customer-specific Codex/Claude plugin containing the skills and MCP endpoint, with interactive OAuth. It deliberately excludes the customer’s server configuration and secrets. Very open to feedback—or to leaving this kind of packaging to customers if that’s preferable.

[pmasiphelps]

@nmartorell i made a slight change to the dataiku-headless-setup skill to add instructions for adding the mcp server based on cli or desktop app (was prev just cli)

[pmasiphelps]

@nmartorell codex wants these two changes too....i think 1 seems straightforward. 2 ill leave up to you whether to only support interactive oauth in the plugin or change the generator direct bearer tokens?

1. The HTTP middleware stringifies missing token claims, so a missing/null sub can become "None" and potentially group unrelated users under the same identity. We should reject missing or non-string iss/sub claims.
2. The generated plugin is described as supporting direct bearer tokens, but it only packages the MCP URL—no token provider or header configuration. I’d scope the generator to interactive OAuth and document direct-bearer clients as a separate managed-configuration path.

[pmasiphelps]

looks great!