Add opt-in OAuth endpoint scopes and bump to 2.15.0

CHANGELOG

@@ -1,5 +1,20 @@
 # Changelog
 
+## v2.15.0
+
+### Improvements
+
+- Add optional OAuth 2.0 / JWT Bearer token authentication on HTTP and
+  Arrow Flight. Configure it with `TABPY_OAUTH_ENABLED` plus issuer, JWKS
+  URI, and audience. Basic Auth is unchanged and can run alongside OAuth.
+- Add optional global JWT scope checks (`TABPY_OAUTH_REQUIRED_SCOPES`) and
+  opt-in per-endpoint scopes (`TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES`) for
+  `tabpy:query` on `/query`, `tabpy:evaluate` on `/evaluate`, and
+  `tabpy:deploy` on mutating `/endpoints` operations. Endpoint scope names
+  are configurable for identity providers with different naming conventions.
+- Add optional per-user logging of the JWT `sub` claim when
+  `TABPY_OAUTH_LOG_USER` is enabled (requires `TABPY_LOG_DETAILS`).
+
 ## v2.14.0
 
 ### Improvements

docs/server-config.md

@@ -14,6 +14,9 @@
   * [Adding an Account](#adding-an-account)
   * [Updating an Account](#updating-an-account)
   * [Deleting an Account](#deleting-an-account)
+  * [OAuth / JWT Bearer Token Authentication](#oauth--jwt-bearer-token-authentication)
+  * [Endpoint Security](#endpoint-security)
+- [Arrow Flight](#arrow-flight)
 - [Logging](#logging)
   * [Request Context Logging](#request-context-logging)
 
@@ -73,6 +76,8 @@ at [`logging.config` documentation page](https://docs.python.org/3.6/library/log
   section. Default value - not set.
 - `TABPY_OAUTH_ENABLED`, `TABPY_OAUTH_ISSUER`, `TABPY_OAUTH_JWKS_URI`,
   `TABPY_OAUTH_AUDIENCE`, `TABPY_OAUTH_REQUIRED_SCOPES`,
+  `TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES`, `TABPY_OAUTH_QUERY_SCOPE`,
+  `TABPY_OAUTH_EVALUATE_SCOPE`, `TABPY_OAUTH_DEPLOY_SCOPE`,
   `TABPY_OAUTH_LOG_USER` - configure OAuth/JWT Bearer token authentication.
   See [OAuth / JWT Bearer Token Authentication](#oauth--jwt-bearer-token-authentication).
 - `TABPY_TRANSFER_PROTOCOL` - transfer protocol. Default value - `http`. If
@@ -307,16 +312,70 @@ service (e.g. a cloud metadata endpoint).
   the signing keys used to verify JWT signatures.
 - `TABPY_OAUTH_AUDIENCE` is the expected `aud` claim on incoming JWTs.
 
-Two additional parameters are optional:
+Six additional parameters are optional:
 
 ```sh
-TABPY_OAUTH_REQUIRED_SCOPES = tabpy:query,tabpy:evaluate
+TABPY_OAUTH_REQUIRED_SCOPES = tabpy
+TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES = true
+TABPY_OAUTH_QUERY_SCOPE = tabpy:query
+TABPY_OAUTH_EVALUATE_SCOPE = tabpy:evaluate
+TABPY_OAUTH_DEPLOY_SCOPE = tabpy:deploy
 TABPY_OAUTH_LOG_USER = true
 ```
 
 - `TABPY_OAUTH_REQUIRED_SCOPES` is a comma-separated list of scopes that
-  must all be present in the JWT's `scope` claim for the request to be
-  accepted. If unset, no scope check is performed.
+  must all be present in the JWT's `scope` claim on **every** request,
+  including `/info`. If unset, no global scope check is performed. A
+  missing global scope is rejected with HTTP 401 (Flight:
+  `UNAUTHENTICATED`). Do not put the configured endpoint scope names here
+  if you want them bound only to their specific paths; use
+  `TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES` for that.
+
+  Scope names are IdP-defined and compared exactly. For example, Amazon
+  Cognito custom scopes use
+  `<resource-server-identifier>/<scope-name>`. A Cognito resource server
+  named `tabpy` with `access` and `finance` scopes could restrict a finance
+  team's TabPy deployment with:
+
+  ```sh
+  TABPY_OAUTH_REQUIRED_SCOPES = tabpy/access,tabpy/finance
+  ```
+
+  A token whose `scope` claim is `openid tabpy/access tabpy/finance` would
+  pass the global scope check, while one containing
+  `openid tabpy/access tabpy/marketing` would be rejected. When multiple
+  scopes are configured, the token must contain **all** of them.
+- `TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES` (default `false`) requires
+  scopes on specific HTTP paths after the JWT itself is valid: by default,
+  `/query` needs `tabpy:query`, `/evaluate` needs `tabpy:evaluate`, and
+  mutating management operations need `tabpy:deploy` (`POST /endpoints`,
+  `PUT`/`DELETE /endpoints/{name}`, and
+  `GET /configurations/endpoint_upload_destination`). Insufficient
+  endpoint scope is rejected with HTTP 403 and
+  `WWW-Authenticate: Bearer error="insufficient_scope"`. `/info`,
+  `/status`, and `GET /endpoints` are not gated by those scopes. A
+  `SCRIPT_*` that calls `tabpy.query()` from `/evaluate` needs **both**
+  `tabpy:evaluate` and `tabpy:query`, because the nested `/query` call
+  forwards the original token. Arrow Flight is not per-endpoint scoped;
+  it still uses only `TABPY_OAUTH_REQUIRED_SCOPES`. Basic Auth is
+  unaffected.
+- `TABPY_OAUTH_QUERY_SCOPE`, `TABPY_OAUTH_EVALUATE_SCOPE`, and
+  `TABPY_OAUTH_DEPLOY_SCOPE` configure the exact scope names used by
+  endpoint enforcement and advertised by `/info`. Their defaults are
+  `tabpy:query`, `tabpy:evaluate`, and `tabpy:deploy`. For an Amazon Cognito
+  resource server with identifier `tabpy`, configure its slash-form custom
+  scopes with:
+
+  ```sh
+  TABPY_OAUTH_QUERY_SCOPE = tabpy/query
+  TABPY_OAUTH_EVALUATE_SCOPE = tabpy/evaluate
+  TABPY_OAUTH_DEPLOY_SCOPE = tabpy/deploy
+  ```
+
+  Each configured value must be a valid, non-empty OAuth scope token.
+  Assigning the same value to multiple endpoint groups gives a token with
+  that scope access to all of those groups; TabPy logs a warning when it
+  detects this configuration.
 - `TABPY_OAUTH_LOG_USER` (default `false`) sets the JWT's `sub` claim as the
   authenticated user for logging purposes. The `sub` claim is often a
   user's email or SSO ID, so leave this disabled unless that's an
@@ -325,6 +384,12 @@ TABPY_OAUTH_LOG_USER = true
   enabled -- that's what actually logs the authenticated user, for both
   basic auth and OAuth.
 
+When OAuth is enabled, `/info` advertises the configured endpoint scopes
+(by default, `tabpy:query`, `tabpy:evaluate`, and `tabpy:deploy`) under
+`versions.v1.features.authentication.methods.oauth-jwt`
+so an IdP or Tableau connection can request those scopes even when
+endpoint enforcement is off.
+
 To authenticate a request, send the JWT as a Bearer token:
 
 ```sh

tabpy/VERSION

@@ -1 +1 @@
-2.14.0
+2.15.0

tabpy/tabpy_server/app/app.py

@@ -4,6 +4,7 @@
 import logging
 import multiprocessing
 import os
+import re
 import shutil
 import signal
 import socket
@@ -23,6 +24,12 @@
 from tabpy.tabpy_server.handlers.basic_auth_server_middleware_factory import (
     BasicAuthServerMiddlewareFactory,
 )
+from tabpy.tabpy_server.handlers.jwt_auth import (
+    SCOPE_DEPLOY,
+    SCOPE_EVALUATE,
+    SCOPE_QUERY,
+    endpoint_scope_names,
+)
 from tabpy.tabpy_server.handlers.jwt_server_middleware_factory import (
     JwtAuthServerMiddlewareFactory,
 )
@@ -397,12 +404,26 @@ def _parse_config(self, config_file):
             (SettingsParameters.OAuthAudience, ConfigParameters.TABPY_OAUTH_AUDIENCE, None, None),
             (SettingsParameters.OAuthRequiredScopes, ConfigParameters.TABPY_OAUTH_REQUIRED_SCOPES,
              None, None),
+            (SettingsParameters.OAuthEnforceEndpointScopes,
+             ConfigParameters.TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES, False, parser.getboolean),
+            (SettingsParameters.OAuthQueryScope,
+             ConfigParameters.TABPY_OAUTH_QUERY_SCOPE, SCOPE_QUERY, None),
+            (SettingsParameters.OAuthEvaluateScope,
+             ConfigParameters.TABPY_OAUTH_EVALUATE_SCOPE, SCOPE_EVALUATE, None),
+            (SettingsParameters.OAuthDeployScope,
+             ConfigParameters.TABPY_OAUTH_DEPLOY_SCOPE, SCOPE_DEPLOY, None),
             (SettingsParameters.OAuthLogUser, ConfigParameters.TABPY_OAUTH_LOG_USER, False, parser.getboolean),
         ]
 
         for setting, parameter, default_val, parse_function in settings_parameters:
             self._set_parameter(parser, setting, parameter, default_val, parse_function)
 
+        self.settings[SettingsParameters.OAuthEndpointScopes] = {
+            SCOPE_QUERY: self.settings[SettingsParameters.OAuthQueryScope],
+            SCOPE_EVALUATE: self.settings[SettingsParameters.OAuthEvaluateScope],
+            SCOPE_DEPLOY: self.settings[SettingsParameters.OAuthDeployScope],
+        }
+
         if not os.path.exists(self.settings[SettingsParameters.UploadDir]):
             os.makedirs(self.settings[SettingsParameters.UploadDir])
 
@@ -574,6 +595,44 @@ def _validate_oauth_settings(self):
             logger.critical(msg)
             raise RuntimeError(msg)
 
+        endpoint_scopes = [
+            (SettingsParameters.OAuthQueryScope,
+             ConfigParameters.TABPY_OAUTH_QUERY_SCOPE),
+            (SettingsParameters.OAuthEvaluateScope,
+             ConfigParameters.TABPY_OAUTH_EVALUATE_SCOPE),
+            (SettingsParameters.OAuthDeployScope,
+             ConfigParameters.TABPY_OAUTH_DEPLOY_SCOPE),
+        ]
+        invalid_scopes = [
+            config_key for setting, config_key in endpoint_scopes
+            if not re.fullmatch(
+                r"[\x21\x23-\x5B\x5D-\x7E]+", self.settings[setting]
+            )
+        ]
+        if invalid_scopes:
+            msg = (
+                f"{', '.join(invalid_scopes)} must be valid, non-empty OAuth "
+                "scope tokens"
+            )
+            logger.critical(msg)
+            raise RuntimeError(msg)
+
+        scope_counts = {}
+        for setting, _ in endpoint_scopes:
+            value = self.settings[setting]
+            scope_counts[value] = scope_counts.get(value, 0) + 1
+        duplicated = [
+            f"{config_key}={self.settings[setting]}"
+            for setting, config_key in endpoint_scopes
+            if scope_counts[self.settings[setting]] > 1
+        ]
+        if duplicated:
+            logger.warning(
+                "OAuth endpoint scope settings contain duplicate values "
+                f"({', '.join(duplicated)}); a token with a shared scope "
+                "can access multiple endpoint groups"
+            )
+
         # JWKS/issuer are the trust anchor for JWT verification, so both must
         # be fetched over https to prevent an on-path attacker from substituting
         # their own keys/issuer.
@@ -638,7 +697,18 @@ def _get_features(self):
             if ConfigParameters.TABPY_PWD_FILE in self.settings:
                 methods["basic-auth"] = {}
             if self.settings[SettingsParameters.OAuthEnabled]:
-                methods["oauth-jwt"] = {}
+                methods["oauth-jwt"] = {
+                    "scopes": list(
+                        endpoint_scope_names(
+                            self.settings[SettingsParameters.OAuthEndpointScopes]
+                        )
+                    ),
+                    "endpoint_scopes_enforced": bool(
+                        self.settings.get(
+                            SettingsParameters.OAuthEnforceEndpointScopes, False
+                        )
+                    ),
+                }
             features["authentication"] = {
                 "required": True,
                 "methods": methods,

tabpy/tabpy_server/app/app_parameters.py

@@ -28,6 +28,10 @@ class ConfigParameters:
     TABPY_OAUTH_JWKS_URI = "TABPY_OAUTH_JWKS_URI"
     TABPY_OAUTH_AUDIENCE = "TABPY_OAUTH_AUDIENCE"
     TABPY_OAUTH_REQUIRED_SCOPES = "TABPY_OAUTH_REQUIRED_SCOPES"
+    TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES = "TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES"
+    TABPY_OAUTH_QUERY_SCOPE = "TABPY_OAUTH_QUERY_SCOPE"
+    TABPY_OAUTH_EVALUATE_SCOPE = "TABPY_OAUTH_EVALUATE_SCOPE"
+    TABPY_OAUTH_DEPLOY_SCOPE = "TABPY_OAUTH_DEPLOY_SCOPE"
     TABPY_OAUTH_LOG_USER = "TABPY_OAUTH_LOG_USER"
 
 
@@ -62,4 +66,9 @@ class SettingsParameters:
     OAuthJwksUri = "oauth_jwks_uri"
     OAuthAudience = "oauth_audience"
     OAuthRequiredScopes = "oauth_required_scopes"
+    OAuthEnforceEndpointScopes = "oauth_enforce_endpoint_scopes"
+    OAuthQueryScope = "oauth_query_scope"
+    OAuthEvaluateScope = "oauth_evaluate_scope"
+    OAuthDeployScope = "oauth_deploy_scope"
+    OAuthEndpointScopes = "oauth_endpoint_scopes"
     OAuthLogUser = "oauth_log_user"

tabpy/tabpy_server/common/default.conf

@@ -53,8 +53,26 @@
 # TABPY_OAUTH_AUDIENCE = tabpy
 
 # Comma-separated list of scopes that must all be present in the JWT's
-# `scope` claim. Leave unset to skip scope enforcement.
-# TABPY_OAUTH_REQUIRED_SCOPES = tabpy:query,tabpy:evaluate
+# `scope` claim on every request (including /info). Leave unset to skip
+# this global check. Do not put the configured endpoint scope names here
+# if you want them bound only to their specific paths; use
+# TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES for that.
+# TABPY_OAUTH_REQUIRED_SCOPES = tabpy
+
+# When true, /query requires tabpy:query, /evaluate requires
+# tabpy:evaluate, and mutating /endpoints plus the upload-destination
+# path require tabpy:deploy. Default false. Independent of
+# TABPY_OAUTH_REQUIRED_SCOPES. GET /endpoints stays readable. A SCRIPT_*
+# that calls tabpy.query() needs both query and evaluate. Arrow Flight is
+# not per-endpoint scoped; it still uses TABPY_OAUTH_REQUIRED_SCOPES only.
+# TABPY_OAUTH_ENFORCE_ENDPOINT_SCOPES = true
+
+# Scope names advertised by /info and required when endpoint enforcement is
+# enabled. Override these when the IdP uses a different naming convention.
+# Amazon Cognito custom scopes commonly use names such as tabpy/query.
+# TABPY_OAUTH_QUERY_SCOPE = tabpy:query
+# TABPY_OAUTH_EVALUATE_SCOPE = tabpy:evaluate
+# TABPY_OAUTH_DEPLOY_SCOPE = tabpy:deploy
 
 # Log the JWT subject as the authenticated user for OAuth requests.
 # TABPY_OAUTH_LOG_USER = true

tabpy/tabpy_server/handlers/base_handler.py

@@ -5,7 +5,12 @@
 import logging
 import tornado.web
 from tabpy.tabpy_server.app.app_parameters import SettingsParameters
-from tabpy.tabpy_server.handlers.jwt_auth import JwtValidationError, validate_jwt
+from tabpy.tabpy_server.handlers.jwt_auth import (
+    JwtValidationError,
+    endpoint_scope_for_path,
+    token_has_scope,
+    validate_jwt,
+)
 from tabpy.tabpy_server.handlers.util import hash_password
 from tabpy.tabpy_server.handlers.util import AuthErrorStates
 import uuid
@@ -139,9 +144,11 @@ def initialize(self, app):
         self.username = None
         self.password = None
         self.jwt_token = None
+        self.jwt_claims = None
         self.auth_method = None
         self.eval_timeout = self.settings[SettingsParameters.EvaluateTimeout]
         self.max_request_size = app.max_request_size
+        self.subdirectory = getattr(app, "subdirectory", "") or ""
 
         self.logger = ContextLoggerWrapper(self.request)
         self.logger.enable_context_logging(
@@ -434,6 +441,8 @@ def _validate_jwt_credentials(self) -> bool:
             self.logger.log(logging.ERROR, str(ex))
             return False
 
+        self.jwt_claims = claims
+
         if self.settings.get(SettingsParameters.OAuthLogUser, False):
             subject = claims.get("sub")
             if subject:
@@ -511,8 +520,34 @@ def handle_authentication(self, api_version):
         if not self._validate_credentials(method):
             return AuthErrorStates.NotAuthorized
 
+        if method == "oauth-jwt":
+            scope_error = self._endpoint_scope_error()
+            if scope_error is not None:
+                return scope_error
+
         return AuthErrorStates.NONE
 
+    def _endpoint_scope_error(self):
+        """
+        After a valid JWT, optionally require well-known endpoint scopes
+        on the matching HTTP path and method. Returns InsufficientScope or None.
+        """
+        if not self.settings.get(SettingsParameters.OAuthEnforceEndpointScopes, False):
+            return None
+        if self.request.method == "OPTIONS":
+            return None
+        required = endpoint_scope_for_path(
+            self.request.path,
+            self.subdirectory,
+            self.request.method,
+            self.settings.get(SettingsParameters.OAuthEndpointScopes),
+        )
+        if not required:
+            return None
+        if token_has_scope(self.jwt_claims or {}, required):
+            return None
+        return AuthErrorStates.InsufficientScope
+
     def should_fail_with_auth_error(self):
         """
         Checks if authentication is required:
@@ -558,6 +593,19 @@ def fail_with_auth_error(self):
                 info="Unauthorized request.",
                 log_message="Invalid credentials provided.",
             )
+        elif self.auth_error == AuthErrorStates.InsufficientScope:
+            self.logger.log(logging.ERROR, "Failing with 403 for insufficient scope")
+            self.set_status(403)
+            self.set_header(
+                "WWW-Authenticate",
+                f'{scheme} realm="{self.tabpy_state.name}", '
+                'error="insufficient_scope"',
+            )
+            self.error_out(
+                403,
+                info="Forbidden request.",
+                log_message="Token is missing a required endpoint scope.",
+            )
         else:
             self.logger.log(logging.ERROR, "Failing with 406 for Not Acceptable")
             self.set_status(406)