fix: signing baskets follow NextGenPSD2 1.3.16, belong to their TPP, and book what they authorise

docs/signing_basket_operations.md

@@ -0,0 +1,137 @@
+# Signing baskets: operating notes
+
+For whoever runs an OBP-API instance that offers the Berlin Group signing basket. The behaviour of the API
+itself is in the ResourceDocs; this covers what an operator does and sees.
+
+## Turning authorisation on
+
+`signing_basket_authorisation_enabled` is `false` by default. While it is, creating, reading, deleting a
+basket and starting an authorisation work, and answering the authorisation (the PUT) answers 403
+`SERVICE_BLOCKED`. Roll it out in stages: first the ownership guard (this change with the property off),
+then the property, with the recovery rehearsal below in between.
+
+Related properties:
+
+| Property | Default | Meaning |
+|---|---|---|
+| `signing_basket_member_max_attempts` | 3 | Times a payment that failed to book is claimed again. Mapped connector only. |
+| `signing_basket_resume_interval_in_seconds` | 593 | How often stopped executions are resumed. 0 switches it off. |
+| `signing_basket_execution_lease_in_seconds` | 300 | How long a basket or member may sit without moving before it counts as stopped. |
+
+## What the stored status means
+
+A basket reports `RCVD`, `PATC`, `ACTC`, `CANC` or `RJCT`. Three more are stored and reported as `RCVD`:
+
+* `AUTHORISING`: the authorisation was answered correctly and the basket was claimed; its members are being
+  executed.
+* `EXECUTION_INCOMPLETE`: execution stopped with a member that is not `DONE`.
+* `EXECUTION_FAILED`: the end of an incomplete basket. Every member that is not `DONE` has failed as often as it
+  is allowed to (`signing_basket_member_max_attempts`), so running it again would change nothing. The basket is no
+  longer picked up and what it held is free; it cannot be authorised again, cancelled or restarted. The creating
+  TPP still reads what happened from the results endpoint.
+
+`ACTC` means every member is `DONE`. Each member has its own state in `SigningBasketMemberExecution`, and the
+creating TPP reads it from `GET /signing-baskets/{basketId}/execution`.
+
+| Member state | Meaning |
+|---|---|
+| `PENDING` | Not started. |
+| `EXECUTING` | Claimed by an executor. Past the lease it becomes `UNKNOWN`. |
+| `DONE` | Payment booked, or consent activated. |
+| `FAILED` | Refused before it took effect. Claimed again automatically, on the mapped connector, up to the attempts allowed. |
+| `UNKNOWN` | The executor stopped without recording an outcome, or a connector other than the mapped one failed after it may have booked. On the mapped connector the resumption turns it into `DONE` (the payment has a transaction id) or `FAILED` (it has not). |
+
+Several payments in one basket are not one transaction. A failure leaves the earlier payments booked.
+
+### What is committed when
+
+The ledger (the claim `RCVD -> AUTHORISING`, the member rows and their states, the release of members) is written on
+a database connection of its own and committed at once. Everything else a request writes (the finalised challenge,
+the booking and transaction id on the mapped connector, the payment status) is committed when the response is sent.
+So if a node dies in the middle of answering an authorisation, the ledger survives and says what was under way, and
+the basket is not answered a second time: it is `AUTHORISING`, not `RCVD`. A request that executes a basket holds two
+connections from the pool for that time.
+
+On the mapped connector a booking that did not commit leaves no transaction id, so the resumption knows that member
+was not booked (it becomes `FAILED` and is claimed again). On any other connector there is no way to know, and the
+member stays `UNKNOWN` for you. The lease (`signing_basket_execution_lease_in_seconds`) must be longer than the
+longest an answer to an authorisation can take, or a request still working can have its members taken over.
+
+## Looking at baskets that did not complete
+
+```sql
+-- Baskets that did not reach ACTC after their authorisation was answered
+SELECT basketid, status, consumerid, psuuserid, updatedat
+FROM signingbasket
+WHERE status IN ('AUTHORISING', 'EXECUTION_INCOMPLETE')
+ORDER BY updatedat;
+
+-- What happened to each member of one basket
+SELECT membertype, memberid, position, state, detail, attempts, updatedat
+FROM SigningBasketMemberExecution
+WHERE basketid = '<basket id>'
+ORDER BY position;
+```
+
+## Members left UNKNOWN
+
+The resumption reconciles an `UNKNOWN` payment by its transaction id: if the payment carries one it was
+booked, and the member becomes `DONE`. Without one, the mapped connector did not book it (it records the
+transaction id in the same transaction as the booking), so the member becomes `FAILED` and is claimed again. On any
+other connector nothing proves whether it was booked, so it is left for you.
+
+1. Find the payment's debit in the ledger (the debtor account, the amount, the time of the execution).
+2. If it was booked, set the payment's transaction id and mark the member `DONE`; the next resumption completes
+   the basket. If it was not, mark the member `FAILED`; it is then claimed again, up to the attempts allowed.
+
+On a connector other than the mapped one, a failure is always recorded as `UNKNOWN`, because that connector
+may have booked before it failed, and automatic retry is off.
+
+## Baskets created before ownership was recorded
+
+Baskets created before this change have no creating TPP and no PSU. They cannot be attributed safely, so every
+operation answers them as unknown (403 `RESOURCE_UNKNOWN`), they are never authorised, and their rows are kept.
+Nothing assigns one to whoever asks first, and no property re-opens them.
+
+To find them:
+
+```sql
+SELECT basketid, status, createdat
+FROM signingbasket
+WHERE consumerid IS NULL OR consumerid = '';
+```
+
+If a particular one has to be revived, assign it explicitly, after establishing who created it:
+
+```sql
+UPDATE signingbasket
+SET consumerid = '<consumer id of the TPP that created it>'
+WHERE basketid = '<basket id>' AND (consumerid IS NULL OR consumerid = '');
+```
+
+Its payments and consents are not held by any claim. If the basket is to be used, add the claims:
+
+```sql
+INSERT INTO SigningBasketMemberClaim (memberkey, basketid, createdat, updatedat)
+VALUES ('payment:<payment id>', '<basket id>', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);
+```
+
+(one row per payment, `consent:<consent id>` for consents). Without them another basket could take the same
+payment. A basket that is only to be closed needs none of this; set its status to `CANC`.
+
+## Things that can still surprise
+
+* A payment that an active basket holds cannot be authorised on its own (the payment authorisation answers 409
+  `STATUS_INVALID`), and a payment that is no longer waiting for SCA (rejected, cancelled, failed) stops the answer to
+  the basket's authorisation with 409 before anything is booked.
+* Wrong answers are counted for the basket, over all its authorisations, against
+  `answer_transactionRequest_challenge_allowed_attempts`. The answer that uses the allowance up rejects the basket
+  (`RJCT`) and its payments, so starting new authorisations does not give new guesses.
+* A payment waiting in a basket is still a payment waiting for SCA: if
+  `berlin_group_outdated_transactions_interval_in_seconds` is set, the outdated-payment task rejects it after
+  `berlin_group_outdated_transactions_time_in_seconds`, and the basket's member then fails as not waiting for
+  authorisation.
+* A consent in a basket is `received` until activated, and the consent scheduler rejects a Berlin Group consent
+  that stays `received` for `berlin_group_outdated_consents_time_in_seconds`.
+* The recurrence of a periodic payment is not stored, so a periodic payment cannot be recognised and refused when
+  it is put in a basket; it is treated as a single payment.

obp-api/src/main/resources/props/sample.props.template

@@ -1672,9 +1672,27 @@ default_auth_context_update_request_key=CUSTOMER_NUMBER
 ## Berlin Group Create Consent Frequency per Day Upper Limit
 #berlin_group_frequency_per_day_upper_limit = 4
 
-## Berlin Group Create Consent ASPSP-SCA-Approach response header value
+## Berlin Group ASPSP-SCA-Approach response header value (create consent, create signing basket and its authorisations)
 #berlin_group_aspsp_sca_approach = redirect
 
+# Whether a Berlin Group signing basket may be authorised (PUT /signing-baskets/{basketId}/authorisations/{authorisationId}).
+# Default false: the call answers 403 SERVICE_BLOCKED. Answering the authorisation books the basket's payments
+# one after another and records, per payment, whether it was booked. Roll this out in stages: first the ownership
+# guard, then this, with a recovery rehearsal in between. Creating, reading, starting an authorisation on and
+# deleting baskets are not affected.
+#signing_basket_authorisation_enabled = false
+
+# How many times a payment of a signing basket that failed to book is claimed again, on the mapped connector only.
+# A payment on any other connector whose booking failed is left UNKNOWN for an operator.
+#signing_basket_member_max_attempts = 3
+
+# Resuming signing basket executions that stopped (for instance because the process booking the payments died).
+# The interval is in seconds (0 switches it off); the lease is how long a basket or member may sit without moving
+# before it counts as stopped. A member left executing past the lease becomes UNKNOWN and is reconciled by its
+# transaction id, or left for an operator.
+#signing_basket_resume_interval_in_seconds = 593
+#signing_basket_execution_lease_in_seconds = 300
+
 # Support multiple brands on one instance. Note this needs checking on a clustered environment
 #brands_enabled=false
 

obp-api/src/main/scala/bootstrap/liftweb/Boot.scala

@@ -125,7 +125,7 @@ import code.regulatedentities.attribute.RegulatedEntityAttribute
 import code.counterpartyattribute.{CounterpartyAttribute => CounterpartyAttributeMapper}
 import code.scheduler._
 import code.scope.{MappedScope, MappedUserScope, Scope}
-import code.signingbaskets.{MappedSigningBasket, MappedSigningBasketConsent, MappedSigningBasketPayment}
+import code.signingbaskets.{MappedSigningBasket, MappedSigningBasketConsent, MappedSigningBasketMemberClaim, MappedSigningBasketMemberExecution, MappedSigningBasketPayment}
 import code.socialmedia.MappedSocialMedia
 import code.standingorders.StandingOrder
 import code.taxresidence.MappedTaxResidence
@@ -643,6 +643,7 @@ class Boot extends MdcLoggable {
     }
     ConsentScheduler.startAll()
     TransactionScheduler.startAll()
+    SigningBasketScheduler.startAll()
 
 
     code.metrics.MetricsProps.enableMetricsScheduler match {
@@ -1004,6 +1005,8 @@ object ToSchemify extends MdcLoggable {
     MappedSigningBasket,
     MappedSigningBasketPayment,
     MappedSigningBasketConsent,
+    MappedSigningBasketMemberClaim,
+    MappedSigningBasketMemberExecution,
     MappedRegulatedEntity,
     AtmAttribute,
     AbacRule,

obp-api/src/main/scala/code/api/berlin/group/ConstantsBG.scala

@@ -47,5 +47,29 @@ object ConstantsBG {
     // 4) CANC (Cancelled) and
     // 5) RJCT (Rejected) are supported for signing baskets.
     val RCVD, PATC, ACTC, CANC, RJCT = Value
+
+    /**
+     * Stored between the moment a correct answer claims the basket and the moment its members have
+     * been dealt with. It is never reported: to a TPP a basket in this state is still RCVD, since the
+     * authorisation has not completed from its point of view.
+     */
+    val AUTHORISING_INTERNAL = "AUTHORISING"
+
+    /**
+     * Stored when the authorisation was answered correctly but not every member took effect: a payment
+     * failed, or an outcome is not known. Reported as RCVD, like AUTHORISING. The members' own results say
+     * what happened to each.
+     */
+    val EXECUTION_INCOMPLETE_INTERNAL = "EXECUTION_INCOMPLETE"
+
+    /**
+     * Stored when execution stopped and nothing that is left can be finished by running it again: every member
+     * that is not done has failed as often as it is allowed to. The basket is over, what it held is free,
+     * and it is no longer picked up. Reported as RCVD; the members' own results say what happened.
+     */
+    val EXECUTION_FAILED_INTERNAL = "EXECUTION_FAILED"
+
+    def external(storedStatus: String): String =
+      if (storedStatus == AUTHORISING_INTERNAL || storedStatus == EXECUTION_INCOMPLETE_INTERNAL || storedStatus == EXECUTION_FAILED_INTERNAL) RCVD.toString else storedStatus
   }
 }

obp-api/src/main/scala/code/api/berlin/group/v1_3/BerlinGroupConsentActivation.scala

@@ -0,0 +1,77 @@
+/**
+Open Bank Project - API
+Copyright (C) 2011-2026, TESOBE GmbH.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Email: contact@tesobe.com
+TESOBE GmbH.
+Osloer Strasse 16/17
+Berlin 13359, Germany
+
+This product includes software developed at
+TESOBE (http://www.tesobe.com/)
+
+  */
+
+package code.api.berlin.group.v1_3
+
+import code.api.util.APIUtil.unboxFullOrFail
+import code.api.util.ErrorMessages.{ConsentAccountAccessCannotBeGranted, ConsentUpdateStatusError}
+import code.api.util.{CallContext, Consent}
+import code.consent.{ConsentStatus, ConsentTrait, Consents}
+import com.openbankproject.commons.ExecutionContext.Implicits.global
+import com.openbankproject.commons.model.User
+
+import scala.concurrent.Future
+
+/**
+ * Makes a Berlin Group consent valid once the PSU's SCA for it has succeeded: grants the access an
+ * "allAccounts" consent leaves open, binds the consent to the PSU, and marks it valid.
+ *
+ * The consent authorisation (PUT /consents/{id}/authorisations/{id}) performs the same steps inline,
+ * interleaved with checking the answer. A signing basket checks the answer once for all its members and
+ * then activates each consent here.
+ *
+ * It can be run again after a stop at any point. The status changes last, so a stop leaves the consent
+ * `received`, with its access granted and perhaps already bound to the PSU, and running it again completes
+ * it. A consent made valid by an earlier version of this method, which set the status before binding, and
+ * left unbound by a stop between the two, is completed by binding it. A consent already valid and bound to
+ * this PSU is returned as it is.
+ */
+object BerlinGroupConsentActivation {
+
+  private def bound(consent: ConsentTrait): Option[String] = Consent.present(consent.userId)
+
+  /** Whether activating this consent for this PSU can still lead to a valid consent bound to them. */
+  def canActivate(consent: ConsentTrait, psuUserId: String): Boolean =
+    consent.status == ConsentStatus.received.toString ||
+      (consent.status == ConsentStatus.valid.toString && bound(consent).forall(_ == psuUserId))
+
+  def activate(consent: ConsentTrait, psu: User, callContext: Option[CallContext]): Future[ConsentTrait] =
+    if (consent.status == ConsentStatus.valid.toString) {
+      bound(consent) match {
+        case Some(user) if user == psu.userId => Future.successful(consent)
+        // Valid but not bound: a stop between the two steps. Bind it; the access was granted before.
+        case None => Consent.bindBerlinGroupConsentToPsu(consent.consentId, psu, callContext).map(_ => consent)
+        case Some(_) => Future.failed(new IllegalStateException(s"The consent is already valid for another PSU"))
+      }
+    } else for {
+      _ <- Consent.grantBerlinGroupAvailableAccountsAccess(psu, consent)
+        .map(unboxFullOrFail(_, callContext, ConsentAccountAccessCannotBeGranted))
+      _ <- Consent.bindBerlinGroupConsentToPsu(consent.consentId, psu, callContext)
+      valid <- Future(Consents.consentProvider.vend.updateConsentStatus(consent.consentId, ConsentStatus.valid))
+        .map(unboxFullOrFail(_, callContext, ConsentUpdateStatusError))
+    } yield valid
+}