Domain APIs and Doobie related fix for Postgres 16

CLAUDE.md

@@ -269,6 +269,8 @@ Symptoms in tests: a v4-specific assertion fails (e.g. an entitlement should-be-
 
 **The build stamp comes from a script, not a Maven plugin**: `git.properties` (what `/status` and the root endpoint's `git_commit` report) is written by `scripts/write_git_properties.sh`, invoked from `obp-api/pom.xml`'s `maven-antrun-plugin` execution `generate-git-properties` at `generate-resources`, straight into `target/classes`. It used to be `git-commit-id-maven-plugin`, which was wrong in two ways: its bundled JGit 6.7 has no `commondir` support, so `GitDirLocator.resolveWorktree()` redirects a linked worktree's gitdir to the *main* checkout's `.git` — every build run from `.claude/worktrees/*` stamped the main checkout's branch and commit — and its `PropertiesFileGenerator` skips rewriting when only `git.build.time` differs, freezing the timestamp. Add stamp fields by editing the script (keep the `git.*` key names; `StatusPage.scala` and `APIUtil.gitCommit` read them by name), and don't reintroduce a per-module generator: exactly one `git.properties` may be on the runtime classpath, otherwise which one is reported is incidental. `.github/workflows/test_worktree_build.yml` guards both failure modes.
 
+**SQL must run on PostgreSQL 16+, not just the local Postgres**: PostgreSQL 16 refuses a bind parameter followed directly by a letter (`ERROR: trailing junk after parameter at or near "$1AND"`), which 14 and earlier accept. In doobie, `fr"…"` appends a space and `fr0"…"` doesn't, so `fr0"$value" ++ fr"AND"` renders `$1AND`: it passes every local test on Postgres 14 and fails on a server running 16. Use `fr0` only where the next fragment starts with a space, a comma or a closing parenthesis, or nothing follows; otherwise use `fr`. The same goes for raw SQL strings: always put a space after `?` / `$n`. This shipped once: every Dynamic Entity projection build failed on dcr (Postgres 16) with that error, logged as `DE projection provisioning failed`, while working locally — `ProjectionStore.scope` built `entityname = $1AND bankid = $2AND …`. Test new SQL against Postgres 16 or later.
+
 ## CI (shard map + run tips)
 
 Perf note: integration tests are DB/HTTP-bound (~0.4 s/test) on both frameworks; the http4s win is the **pure-unit tier** (no running server, ~0.008 s/test). `ResourceDocsTest`/`SwaggerDocsTest` are the slowest per-test cost — they serialize the whole API surface, so cost grows with endpoint count. `Http4sResourceDocs` already caches the serialized output (`Caching.{getDynamic,getStatic,getAll}ResourceDocCache` + `getStaticSwaggerDocCache`, keyed via `APIUtil.createResourceDocCacheKey`), so repeat requests for the same version/params skip re-serialization.

obp-api/src/main/scala/bootstrap/liftweb/Boot.scala

@@ -1144,6 +1144,7 @@ object ToSchemify extends MdcLoggable {
     code.glossaryitem.DynamicGlossaryItem,
     code.platformapp.PlatformApp,
     code.platformapp.PlatformAppRequiredScope,
+    code.domainapi.DomainApi,
     PayeeLookup,
     UtilityPaymentCallback,
     BulkPayment,

obp-api/src/main/scala/code/api/dynamic/domainapi/DomainApiPaths.scala

@@ -0,0 +1,181 @@
+/**
+Open Bank Project - API
+Copyright (C) 2011-2026, TESOBE GmbH.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Email: contact@tesobe.com
+TESOBE GmbH.
+Osloer Strasse 16/17
+Berlin 13359, Germany
+
+This product includes software developed at
+TESOBE (http://www.tesobe.com/)
+
+  */
+
+package code.api.dynamic.domainapi
+
+import cats.effect.IO
+import cats.effect.unsafe.implicits.global
+import code.api.Constant.ApiPathZero
+import code.api.berlin.group.ConstantsBG
+import code.api.util.APIUtil.ResourceDoc
+import code.domainapi.DomainApiRoute
+import com.openbankproject.commons.util.{ApiShortVersions, ApiStandards, ApiVersion}
+import org.json4s.JsonAST.{JObject, JValue}
+
+/**
+ * This object holds the rules that relate a Domain API's URLs to OBP's own, in one place.
+ *
+ * A Domain API publishes the Dynamic Entities and Dynamic Resource Docs of one space under a base path,
+ * so that `/carbon-registry/v1/activity` reaches what OBP serves at
+ * `/obp/v7.0.0/banks/SYS/dynamic-entities/activity`. Two things need the same mapping: the front door
+ * ([[Http4sDomainApi]]), which rewrites an incoming call to the OBP URL, and the Domain API's OpenAPI
+ * file, which rewrites each documented OBP URL to the published one. Both read the functions here, so
+ * the published documentation cannot describe a path the front door does not serve, or the reverse.
+ *
+ * The same holds for the one thing a Domain API changes in a response: a Dynamic Entity record response
+ * leaves out `bank_id`, because the base path already fixes the space. [[responseUnderDomainApi]] is
+ * applied both to the real response and to the documented example.
+ */
+object DomainApiPaths {
+
+  /** What the front door records on a request it rewrote: which Domain API, and the path that was called. */
+  case class DomainApiCall(domainApiId: String, basePath: String, calledPath: String)
+
+  val domainApiCallKey: org.typelevel.vault.Key[DomainApiCall] =
+    org.typelevel.vault.Key.newKey[IO, DomainApiCall].unsafeRunSync()
+
+  private val obp = ApiStandards.obp.toString
+  private val v700 = ApiVersion.v7_0_0.toString
+  private val dynamicEndpoint = ApiShortVersions.`dynamic-endpoint`.toString
+  private val dynamicResourceDocSegment = "dynamic-resource-doc"
+  private val dynamicEntitiesSegment = "dynamic-entities"
+
+  /**
+   * The first path segments OBP serves itself. A base path may not start with one of them, so a Domain
+   * API can never be confused with, or hide, one of OBP's own URLs. The front door also runs last in the
+   * request chain, after every OBP route, as a second line of defence.
+   */
+  def reservedFirstSegments: Set[String] = Set(
+    obp, ApiPathZero, "open-banking", ConstantsBG.berlinGroupVersion1.urlPrefix, ConstantsBG.berlinGroupVersion2.urlPrefix,
+    "my", "apps", "status", "health", "alive", "banks", "oauth", "dauth", "siwe", ".well-known", "static",
+    "openapi.json", "openapi.yaml"
+  )
+
+  /**
+   * Path segments a Dynamic Entity URL already gives a meaning to right after the space, and the names of
+   * the Domain API's own documentation. A Dynamic Resource Doc whose path starts with one of them would be
+   * hidden under a Domain API, so it counts as a clash.
+   */
+  val reservedUnderBasePath: Set[String] = Set("my", "public", "community", "openapi.json", "openapi.yaml")
+
+  private val Segment = "[a-z0-9]([a-z0-9.-]*[a-z0-9])?"
+  private val MajorVersionSegment = "v(0|[1-9][0-9]*)".r
+  private val SemanticVersion = "(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)".r
+
+  /** None when the base path is acceptable, otherwise why not. */
+  def basePathProblem(basePath: String): Option[String] = {
+    val segments = basePath.split("/", -1).toList
+    if (segments.length < 2 || segments.length > 5) Some("it must have two to five segments")
+    else if (!segments.forall(_.matches(Segment))) Some("each segment must be lowercase letters, digits, hyphens or dots")
+    else if (reservedFirstSegments.contains(segments.head)) Some(s"its first segment, ${segments.head}, is one OBP serves")
+    else if (MajorVersionSegment.unapplySeq(segments.last).isEmpty) Some("its last segment must be the major version, vN")
+    else None
+  }
+
+  /** The N of the `vN` that ends a base path. */
+  def majorOf(basePath: String): Option[Int] =
+    basePath.split("/").lastOption.collect { case MajorVersionSegment(n) => n.toInt }
+
+  /** A version is MAJOR.MINOR.PATCH, and its MAJOR is the base path's. */
+  def versionFits(version: String, basePath: String): Boolean = version match {
+    case SemanticVersion(major, _, _) => majorOf(basePath).contains(major.toInt)
+    case _ => false
+  }
+
+  /**
+   * Two base paths overlap when one is the other or starts with it, segment by segment. Overlapping base
+   * paths are refused, so a call never has more than one Domain API it could belong to.
+   */
+  def overlap(a: String, b: String): Boolean = {
+    val (as, bs) = (a.split("/").toList, b.split("/").toList)
+    as.startsWith(bs) || bs.startsWith(as)
+  }
+
+  /** The Domain API a request path is under, and the segments after its base path. */
+  def find(routes: List[DomainApiRoute], pathSegments: List[String]): Option[(DomainApiRoute, List[String])] =
+    routes.collectFirst {
+      case route if pathSegments.startsWith(route.basePathSegments) => (route, pathSegments.drop(route.basePathSegments.length))
+    }
+
+  /** The OBP path a call to a Dynamic Entity under a Domain API is served at. */
+  def dynamicEntityPath(space: String, rest: List[String]): List[String] =
+    obp :: v700 :: "banks" :: space :: dynamicEntitiesSegment :: rest
+
+  /** The OBP path a call to a Dynamic Resource Doc under a Domain API is served at. */
+  def dynamicResourceDocPath(space: String, rest: List[String]): List[String] =
+    obp :: dynamicEndpoint :: "banks" :: space :: dynamicResourceDocSegment :: rest
+
+  /**
+   * The path under the base path at which a documented endpoint is published, from its ResourceDoc's
+   * request URL: the v7.0.0 Dynamic Entity docs (`/banks/SPACE/dynamic-entities/...`) and the Dynamic
+   * Resource Docs (`/banks/SPACE/dynamic-resource-doc/...`). Anything else is not published (None).
+   */
+  def publishedPath(space: String, docRequestUrl: String): Option[String] = {
+    // A doc's request URL may carry the prefix of the version it is served in (/obp/v7.0.0/...); the
+    // space starts at `banks`.
+    val segments = docRequestUrl.split("/").filter(_.nonEmpty).toList.dropWhile(_ != "banks")
+    segments match {
+      case "banks" :: `space` :: kind :: rest if rest.nonEmpty && (kind == dynamicEntitiesSegment || kind == dynamicResourceDocSegment) =>
+        Some(rest.mkString("/", "/", ""))
+      case _ => None
+    }
+  }
+
+  /** A path template with each placeholder (an all-capitals segment) reduced to one form, for comparison. */
+  private def templateKey(path: String): String =
+    path.split("/").filter(_.nonEmpty).map(s => if (s.matches("[A-Z][A-Z0-9_]*")) "{}" else s).mkString("/", "/", "")
+
+  /**
+   * The verb and path pairs that more than one endpoint of the space would publish, and the Dynamic
+   * Resource Docs whose path starts with a segment a Dynamic Entity URL or the documentation already uses.
+   * Each is described for the person who has to resolve it.
+   */
+  def clashes(space: String, docs: List[ResourceDoc]): List[String] = {
+    val published = docs.flatMap(doc => publishedPath(space, doc.requestUrl).map(path => (doc, path)))
+    val duplicates = published
+      .groupBy { case (doc, path) => (doc.requestVerb.toUpperCase, templateKey(path)) }
+      .collect { case ((verb, key), entries) if entries.length > 1 =>
+        s"$verb $key (${entries.map(_._1.partialFunctionName).sorted.mkString(", ")})"
+      }.toList
+    val hidden = published.collect {
+      case (doc, path) if doc.requestUrl.contains(s"/$dynamicResourceDocSegment/") &&
+        reservedUnderBasePath.contains(path.split("/").filter(_.nonEmpty).headOption.getOrElse("")) =>
+        s"${doc.requestVerb.toUpperCase} $path (${doc.partialFunctionName}) starts with a reserved segment"
+    }
+    (duplicates ++ hidden).sorted
+  }
+
+  /** A Dynamic Entity record response as a Domain API returns it: without `bank_id`. */
+  def responseUnderDomainApi(response: JObject): JObject =
+    JObject(response.obj.filterNot(_._1 == "bank_id"))
+
+  /** The same rule applied to a documented example, which may be any JSON. */
+  def exampleUnderDomainApi(example: Any): Any = example match {
+    case o: JObject => responseUnderDomainApi(o)
+    case other => other
+  }
+}

obp-api/src/main/scala/code/api/dynamic/domainapi/Http4sDomainApi.scala

@@ -0,0 +1,138 @@
+/**
+Open Bank Project - API
+Copyright (C) 2011-2026, TESOBE GmbH.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Email: contact@tesobe.com
+TESOBE GmbH.
+Osloer Strasse 16/17
+Berlin 13359, Germany
+
+This product includes software developed at
+TESOBE (http://www.tesobe.com/)
+
+  */
+
+package code.api.dynamic.domainapi
+
+import cats.data.{Kleisli, OptionT}
+import cats.effect.IO
+import code.api.Constant.HostName
+import code.api.ResourceDocs1_4_0.OpenAPI31JSONFactory
+import code.api.ResourceDocs1_4_0.OpenAPI31JSONFactory.{InfoJson, ServerJson}
+import code.api.cache.Caching
+import code.api.dynamic.domainapi.DomainApiPaths.{DomainApiCall, domainApiCallKey}
+import code.api.dynamic.endpoint.Http4sDynamicEndpoint
+import code.api.dynamic.entity.Http4sDynamicEntity
+import code.api.util.APIUtil.ResourceDoc
+import code.api.util.{APIUtil, YAMLUtils}
+import code.api.v1_4_0.JSONFactory1_4_0
+import code.domainapi.{DomainApiRoute, DomainApis}
+import code.util.Helper.MdcLoggable
+import com.openbankproject.commons.util.ApiVersion
+import org.http4s._
+import org.http4s.headers.`Content-Type`
+import org.json4s.JsonAST.JValue
+import org.json4s.native.JsonMethods.compact
+
+/**
+ * This object is the front door of the Domain APIs: it serves the URLs under each registered base path.
+ *
+ * A call under a base path is rewritten to the OBP URL of the same endpoint (see [[DomainApiPaths]]) and
+ * handed to the handler that serves that URL, so authentication, Roles, Consents, rate limiting,
+ * row-level access, field restrictions, Dynamic Query checks and metrics all run exactly as they do for
+ * the OBP URL. A Domain API grants nothing. A Dynamic Entity of the space is tried first, then a Dynamic
+ * Resource Doc of the space; registering or updating a Domain API is refused while two of the space's
+ * endpoints would answer the same verb and path, so the order only matters for a clash created later.
+ *
+ * `BASE_PATH/openapi.json` and `BASE_PATH/openapi.yaml` serve the Domain API's own OpenAPI document.
+ *
+ * It is wired into Http4sApp.baseServices last, just before the JSON 404, so no OBP route can be hidden
+ * by a base path; base paths are also refused when their first segment is one OBP serves.
+ */
+object Http4sDomainApi extends MdcLoggable {
+
+  private type HttpF[A] = OptionT[IO, A]
+
+  private val jsonContentType = `Content-Type`(MediaType.application.json, Charset.`UTF-8`)
+  private val yamlContentType = `Content-Type`(new MediaType("application", "yaml"), Charset.`UTF-8`)
+
+  private def withPath(req: Request[IO], segments: List[String]): Request[IO] =
+    req.withUri(req.uri.withPath(Uri.Path.unsafeFromString(segments.mkString("/", "/", ""))))
+
+  lazy val routes: HttpRoutes[IO] =
+    Kleisli[HttpF, Request[IO], Response[IO]] { (req: Request[IO]) =>
+      val segments = req.uri.path.segments.map(_.encoded).toList
+      DomainApiPaths.find(DomainApis.domainApiProvider.vend.routes(), segments) match {
+        case None => OptionT.none[IO, Response[IO]]
+        case Some((route, rest)) =>
+          rest match {
+            case "openapi.json" :: Nil if req.method == Method.GET =>
+              OptionT.liftF(IO(openApiJson(route)).map(body =>
+                Response[IO](Status.Ok).withEntity(body).withContentType(jsonContentType)))
+            case "openapi.yaml" :: Nil if req.method == Method.GET =>
+              OptionT.liftF(IO(openApiYaml(route)).map(body =>
+                Response[IO](Status.Ok).withEntity(body).withContentType(yamlContentType)))
+            case Nil => OptionT.none[IO, Response[IO]]
+            case _ =>
+              val marked = req.withAttribute(domainApiCallKey,
+                DomainApiCall(route.domainApiId, route.basePath, req.uri.path.renderString))
+              Http4sDynamicEntity.wrappedRoutesDynamicEntityV700.run(withPath(marked, DomainApiPaths.dynamicEntityPath(route.bankId, rest)))
+                .orElse(Http4sDynamicEndpoint.wrappedRoutesDynamicEndpoint.run(withPath(marked, DomainApiPaths.dynamicResourceDocPath(route.bankId, rest))))
+          }
+      }
+    }
+
+  /**
+   * The ResourceDocs of a space's endpoints that a Domain API publishes: the v7.0.0 Dynamic Entity docs
+   * and the Dynamic Resource Docs of that space.
+   */
+  def spaceDocs(space: String): List[ResourceDoc] =
+    APIUtil.allDynamicResourceDocsIn(ApiVersion.v7_0_0).filter(doc => APIUtil.dynamicResourceDocBelongsToSpace(doc, space))
+
+  /** The space's docs as the Domain API publishes them: at their published path, examples without bank_id. */
+  def publishedDocs(route: DomainApiRoute): List[ResourceDoc] =
+    spaceDocs(route.bankId).flatMap { doc =>
+      DomainApiPaths.publishedPath(route.bankId, doc.requestUrl).map { path =>
+        val published = doc.copy(requestUrl = path, successResponseBody = DomainApiPaths.exampleUnderDomainApi(doc.successResponseBody))
+        published.connectorMethods = doc.connectorMethods
+        published
+      }
+    }
+
+  /** The Domain API's OpenAPI 3.1 document: the published docs, with the Domain API's own title, version and server. */
+  def openApi(route: DomainApiRoute): JValue = {
+    val docsJson = JSONFactory1_4_0.createResourceDocsJson(publishedDocs(route), isVersion4OrHigher = true, locale = None).resource_docs
+    val document = OpenAPI31JSONFactory.createOpenAPI31Json(docsJson, route.version, HostName).copy(
+      info = InfoJson(title = route.title, version = route.version, description = Some(route.description).filter(_.nonEmpty)),
+      servers = List(ServerJson(url = s"$HostName/${route.basePath}", description = Some(route.title))))
+    OpenAPI31JSONFactory.OpenAPI31JsonFormats.toJValue(document)
+  }
+
+  // Cached with the dynamic resource docs, which are cleared whenever a Dynamic Entity or Dynamic Resource
+  // Doc changes. The key carries everything of the registration the document shows.
+  private def cached(route: DomainApiRoute, format: String)(build: => String): String = {
+    val key = s"domain-api-openapi:$format:${route.domainApiId}:${route.basePath}:${route.version}:${(route.title + route.description).hashCode}"
+    Caching.getDynamicResourceDocCache(key).getOrElse {
+      val rendered = build
+      Caching.setDynamicResourceDocCache(key, rendered)
+      rendered
+    }
+  }
+
+  def openApiJson(route: DomainApiRoute): String = cached(route, "json")(compact(org.json4s.native.JsonMethods.render(openApi(route))))
+
+  def openApiYaml(route: DomainApiRoute): String = cached(route, "yaml")(YAMLUtils.jValueToYAMLSafe(openApi(route), "# Error converting to YAML"))
+}