` support handlers that require the bus's `ExecutionContext`. Bind their message and result types in a concrete subclass. Type resolution follows generic bindings through intermediate classes, reordered parameters, interfaces and standard generated subclasses; raw or unbound message variables do not become permission to dispatch every command/query through their upper bound. Construction still rejects an unresolved handler type. Parameterized result types expose their raw Java class for dispatch metadata, rather than a complete serialization schema. Context propagation does not authenticate an arbitrary context: populate identity through a trusted application boundary and enforce the operation's authorization policy. + Publishing a command result requires EDA, an `EventPublisherFactory` bean, the enabled CQRS/EDA bridge, and `@PublishDomainEvent` on the handler. Query invalidation requires a `QueryCacheAdapter` and a matching `@InvalidateCacheOn` event delivered through Spring's application-event path. Connect broker events to that local path explicitly. The onboarding tutorial configures neither bridge. These annotation declarations are API fragments for an application that has added that infrastructure: @@ -230,6 +232,14 @@ void shouldCreateFailedAuthorizationResultWithSingleError() { ## Detailed reference +### Explicit query cache identity + +`Query.getCacheKey()` returns `null` by default, so a handler executes without a cached result unless the query explicitly supplies a key. The former class-name/metadata-hash default could collapse different resources into the same entry. Updating the framework stops reading those implicit entries; existing explicit custom keys keep their own semantics and must be reviewed separately. + +A custom key must represent all result-affecting inputs and the relevant trusted tenant, subject, and policy scope, including resource identity, filters, pagination, and version where applicable. Do not put credentials or personal data in keys. If an execution context changes the result, either include its trusted scope in the query's explicit key or keep that handler uncached. The bus does not infer a complete authority boundary from an arbitrary context map. + +Both `query(query)` and `query(query, context)` require the query and handler to allow caching and a non-null explicit key. Validation and configured authorization precede cache access. `@QueryHandlerComponent(cacheable = false)` bypasses both cache reads and writes, including when a matching stale entry exists. Prefer that setting for authoritative workflow status/result reads without a qualified invalidation and freshness policy. + ### Configuration All properties live under the `firefly.cqrs.*` namespace. The values below are the configured defaults; you only need to set what you want to change. @@ -244,7 +254,7 @@ firefly: tracing-enabled: true query: timeout: 15s # default query processing timeout - caching-enabled: true # enable query-result caching by default + caching-enabled: true # provide caching; each query still needs an explicit key cache-ttl: 15m # default TTL for the dedicated CQRS query cache metrics-enabled: true tracing-enabled: true diff --git a/docs/site/modules/ecm-esignature-adobe-sign.md b/docs/site/modules/ecm-esignature-adobe-sign.md index db6506f5d..a13af4218 100644 --- a/docs/site/modules/ecm-esignature-adobe-sign.md +++ b/docs/site/modules/ecm-esignature-adobe-sign.md @@ -24,7 +24,7 @@ If onboarding includes a signed agreement, use the Adobe Sign adapter for the co``` diff --git a/docs/site/modules/ecm-esignature-docusign.md b/docs/site/modules/ecm-esignature-docusign.md index 53123f175..bcbe05f56 100644 --- a/docs/site/modules/ecm-esignature-docusign.md +++ b/docs/site/modules/ecm-esignature-docusign.md @@ -24,7 +24,7 @@ If onboarding includes a signed agreement, use the DocuSign adapter for the conf org.fireflyframework fireflyframework-ecm-esignature-adobe-sign -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/ecm-esignature-logalty.md b/docs/site/modules/ecm-esignature-logalty.md index 8e213add2..3b3b976b0 100644 --- a/docs/site/modules/ecm-esignature-logalty.md +++ b/docs/site/modules/ecm-esignature-logalty.md @@ -24,7 +24,7 @@ If onboarding includes a signed agreement, use the Logalty adapter for the confi org.fireflyframework fireflyframework-ecm-esignature-docusign -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/ecm-storage-aws.md b/docs/site/modules/ecm-storage-aws.md index 362859901..734edd296 100644 --- a/docs/site/modules/ecm-storage-aws.md +++ b/docs/site/modules/ecm-storage-aws.md @@ -24,7 +24,7 @@ Store an applicant's supporting document in Amazon S3 through the ECM content po org.fireflyframework fireflyframework-ecm-esignature-logalty -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/ecm-storage-azure.md b/docs/site/modules/ecm-storage-azure.md index 28a77db40..44ed5a9f5 100644 --- a/docs/site/modules/ecm-storage-azure.md +++ b/docs/site/modules/ecm-storage-azure.md @@ -24,7 +24,7 @@ Store an applicant's supporting document in Azure Blob Storage through the ECM c org.fireflyframework fireflyframework-ecm-storage-aws -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/ecm.md b/docs/site/modules/ecm.md index b1701e96d..c40d1d4ef 100644 --- a/docs/site/modules/ecm.md +++ b/docs/site/modules/ecm.md @@ -37,7 +37,7 @@ An applicant may upload supporting documents or sign an agreement after submissi org.fireflyframework fireflyframework-ecm-storage-azure -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/eda-kafka.md b/docs/site/modules/eda-kafka.md index f6c6036d0..737135868 100644 --- a/docs/site/modules/eda-kafka.md +++ b/docs/site/modules/eda-kafka.md @@ -24,7 +24,7 @@ Publish application lifecycle events to Kafka when a separate review worker cons org.fireflyframework fireflyframework-ecm -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/eda-postgres.md b/docs/site/modules/eda-postgres.md index beb2b57a7..b65f7b0db 100644 --- a/docs/site/modules/eda-postgres.md +++ b/docs/site/modules/eda-postgres.md @@ -24,7 +24,7 @@ A PostgreSQL delivery table can carry application lifecycle events without a sep org.fireflyframework fireflyframework-eda-kafka -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/eda-rabbitmq.md b/docs/site/modules/eda-rabbitmq.md index c1a045a56..c37b9cbe3 100644 --- a/docs/site/modules/eda-rabbitmq.md +++ b/docs/site/modules/eda-rabbitmq.md @@ -24,7 +24,7 @@ RabbitMQ can route application lifecycle events to the configured review queues. org.fireflyframework fireflyframework-eda-postgres -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/eda.md b/docs/site/modules/eda.md index 3b429185b..8814997c3 100644 --- a/docs/site/modules/eda.md +++ b/docs/site/modules/eda.md @@ -24,7 +24,7 @@ Once a submitted application must notify another service, define an application. org.fireflyframework fireflyframework-eda-rabbitmq -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/eventsourcing.md b/docs/site/modules/eventsourcing.md index 26b4e30ef..d1165753d 100644 --- a/docs/site/modules/eventsourcing.md +++ b/docs/site/modules/eventsourcing.md @@ -24,7 +24,7 @@ If the product needs application history as the source of truth, model submitted org.fireflyframework fireflyframework-eda -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/index.md b/docs/site/modules/index.md index d9d56beee..2fc34c530 100644 --- a/docs/site/modules/index.md +++ b/docs/site/modules/index.md @@ -115,6 +115,7 @@ Coordinate eligibility checks, capacity reservations, and review steps. [Follow | Component | Purpose | Role | | --- | --- | --- | | [Orchestration](orchestration.md) | Coordinate workflows, compensating sagas, and try-confirm-cancel operations in a reactive application. | application library | +| [Durable Batch](batch.md) | Run application-owned restartable jobs with explicit JDBC transactions and PostgreSQL checkpoints. | optional persistence foundation | ## Service operations diff --git a/docs/site/modules/kernel.md b/docs/site/modules/kernel.md index 76c3d84a2..9663d122f 100644 --- a/docs/site/modules/kernel.md +++ b/docs/site/modules/kernel.md @@ -24,7 +24,7 @@ A rejected application and an unavailable verification partner need different er org.fireflyframework fireflyframework-eventsourcing -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/notifications-core.md b/docs/site/modules/notifications-core.md index b8a43f6fc..6c1705f69 100644 --- a/docs/site/modules/notifications-core.md +++ b/docs/site/modules/notifications-core.md @@ -24,7 +24,7 @@ The onboarding service can use EmailService to confirm receipt or request missin org.fireflyframework fireflyframework-kernel -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/notifications-firebase.md b/docs/site/modules/notifications-firebase.md index 375046003..4eb3b2520 100644 --- a/docs/site/modules/notifications-firebase.md +++ b/docs/site/modules/notifications-firebase.md @@ -24,7 +24,7 @@ A mobile applicant portal can receive push notifications through Firebase. The a org.fireflyframework fireflyframework-notifications-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/notifications-resend.md b/docs/site/modules/notifications-resend.md index 1e4ecf195..7d74ad729 100644 --- a/docs/site/modules/notifications-resend.md +++ b/docs/site/modules/notifications-resend.md @@ -24,7 +24,7 @@ Resend is another email provider for application confirmations. Select it throug org.fireflyframework fireflyframework-notifications-firebase -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/notifications-sendgrid.md b/docs/site/modules/notifications-sendgrid.md index 0407f8491..c129a8c3e 100644 --- a/docs/site/modules/notifications-sendgrid.md +++ b/docs/site/modules/notifications-sendgrid.md @@ -24,7 +24,7 @@ SendGrid can deliver application confirmations through the email provider port. org.fireflyframework fireflyframework-notifications-resend -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/notifications-twilio.md b/docs/site/modules/notifications-twilio.md index 0554e5365..a1d9ac546 100644 --- a/docs/site/modules/notifications-twilio.md +++ b/docs/site/modules/notifications-twilio.md @@ -24,7 +24,7 @@ Use the Twilio SMS provider for an onboarding message when the product has colle org.fireflyframework fireflyframework-notifications-sendgrid -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/notifications.md b/docs/site/modules/notifications.md index 632286c8d..09d2a719b 100644 --- a/docs/site/modules/notifications.md +++ b/docs/site/modules/notifications.md @@ -49,7 +49,7 @@ This artifact has `pom` packaging and groups child projects. Select a concrete c org.fireflyframework fireflyframework-notifications-twilio -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/observability.md b/docs/site/modules/observability.md index bd4820688..89ce0f5a8 100644 --- a/docs/site/modules/observability.md +++ b/docs/site/modules/observability.md @@ -24,7 +24,7 @@ Measure application submissions and review latency with the metrics support, and org.fireflyframework fireflyframework-notifications-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/orchestration.md b/docs/site/modules/orchestration.md index 8eb2bc31b..9e069242b 100644 --- a/docs/site/modules/orchestration.md +++ b/docs/site/modules/orchestration.md @@ -39,7 +39,7 @@ An extended onboarding flow might verify documents, wait for a review, and provi org.fireflyframework fireflyframework-observability -26.10.21-SNAPSHOT +26.10.22 ``` @@ -138,6 +138,23 @@ tccEngine.execute(def, TccInputs.of(Map.of("debit", request, "credit", request)) }); ``` +### Require TCC persistence acknowledgments + +`BEST_EFFORT` is the default. Select `REQUIRED` when an execution must stop if its configured persistence provider does not acknowledge a save: + +```yaml +firefly: + orchestration: + tcc: + persistence-mode: REQUIRED +``` + +The native configuration gives the engine and orchestrator the same policy/provider. Manual assembly must do so explicitly; REQUIRED rejects absent or mismatched providers/policies. An initial save failure invokes no participant. A required checkpoint or final-save failure propagates `TccPersistenceException` with its correlation ID, save stage, attempted status, original cause, possible-effects flag and retained execution context. It cannot become a confirmed result, an error-suppressed success or an automatic compensation request. + +Overall completion observations, execution-completed publication and success callbacks—including asynchronous callback subscription—wait for the final acknowledgment. Participant/phase events still describe calls that already occurred and do not establish overall financial completion. Preserve the original operation/resource IDs when handling uncertainty; do not blindly retry execution or compensate based only on this error. + +An acknowledged save means the configured publisher completed. The default in-memory provider does not gain disk durability through this setting. Existing phase snapshots are not a participant journal or a durable pause/resume protocol: a CONFIRMED phase checkpoint can exist without an acknowledged final report. Actual provider storage, restart recovery and the product's financial action protocol need separate qualification. Existing participant-error cancellation rules remain; define participant timeout/uncertain-outcome semantics before using the example for money movement. + ### Workflow ```java diff --git a/docs/site/modules/parent.md b/docs/site/modules/parent.md index 6d876e953..005d429b6 100644 --- a/docs/site/modules/parent.md +++ b/docs/site/modules/parent.md @@ -24,7 +24,7 @@ The onboarding example inherits this parent and keeps its own application versio org.fireflyframework fireflyframework-orchestration -26.10.21-SNAPSHOT +26.10.22 ``` @@ -70,6 +70,17 @@ Versions below are read directly from the parent POM. Spring Boot supplies addit | `maven-failsafe-plugin.version` | `3.6.0` | | `native-maven-plugin.version` | `1.1.14` | +### Generated Java SDK object variants + +The managed OpenAPI Generator plugin accepts these opt-in Maven properties: + +| Property | Default | Generated behavior | +| --- | --- | --- | +| `openapi.use.oneof.interfaces` | `false` | Generates interfaces for supported `oneOf` object variants. | +| `openapi.use.oneof.deduction` | `false` | Enables Jackson deduction from variant field shapes in the qualified Java WebClient configuration. | + +Set both properties to `true` in the consuming SDK project after qualifying its schema, generator configuration, and runtime mapper. Test each variant's serialized shape and deserialization, including ambiguous objects and compatibility with the owning API. Continue to generate SDK code through the original OpenAPI lifecycle. These options preserve existing behavior by default and do not alter business admission rules or publish consumer artifacts. + ### Imported dependency platforms | BOM | Version expression | diff --git a/docs/site/modules/plugin-api.md b/docs/site/modules/plugin-api.md index dbe3cd6b3..541a1f78b 100644 --- a/docs/site/modules/plugin-api.md +++ b/docs/site/modules/plugin-api.md @@ -24,7 +24,7 @@ Define a host extension contract for an onboarding check and implement it in an org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/plugin-core.md b/docs/site/modules/plugin-core.md index 4ea8837d8..359d1fcd7 100644 --- a/docs/site/modules/plugin-core.md +++ b/docs/site/modules/plugin-core.md @@ -24,7 +24,7 @@ The onboarding host can load an application plugin class and start its contribut org.fireflyframework plugin-api -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/plugins.md b/docs/site/modules/plugins.md index bfc16f2d8..6578a8209 100644 --- a/docs/site/modules/plugins.md +++ b/docs/site/modules/plugins.md @@ -47,7 +47,7 @@ This artifact has `pom` packaging and groups child projects. Select a concrete c org.fireflyframework plugin-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/r2dbc.md b/docs/site/modules/r2dbc.md index 5ba6a75d3..e2751a90c 100644 --- a/docs/site/modules/r2dbc.md +++ b/docs/site/modules/r2dbc.md @@ -24,7 +24,7 @@ The tutorial saves applications in a process-local map. An application-owned R2D org.fireflyframework plugin-api -26.10.21-SNAPSHOT +26.10.22 ``` @@ -46,6 +46,18 @@ spring: password: secret ``` +### Opt-in bounded root transactions + +The source tree provides `BoundedRootTransaction` and `impl.DefaultBoundedRootTransaction` for an application-owned R2DBC transaction with a finite operation budget. Use a source build containing these classes until the corresponding release is published. This API does not register beans or change the default transaction manager. + +Build one `OwnedLeaseConnectionFactory` around the application connection factory, then construct the actual `DatabaseClient`, customized `R2dbcEntityTemplate`, Spring Data repositories, and `R2dbcTransactionManager` with that same wrapper. `BoundedR2dbcResources.verify` checks the factory/client/template/manager identities; preserve the host converter, callbacks, and repository configuration. The helper refuses an ambient transaction rather than joining another owner's work. It owns individual acquired leases and never closes the application pool. + +Capture the monotonic anchor before obtaining an original observation. Pass its original observation time, absolute expiration, explicit maximum duration, clock, and the same monotonic supplier to `R2dbcOperationBudget.observed`. Execute with a `PROPAGATION_REQUIRED` transaction definition and return one result; other propagation modes are refused. Compose an already observed dependency limit with `original.intersect(inherited)` before passing the budget to `execute`, so connection acquisition and transaction begin use the earlier bound. Use `RootTransactionScope.narrow` when a later native observation or dependency imposes an earlier bound. Original and narrowed budgets must share the identical `Clock` object and monotonic supplier. Budget checks include acquisition, transaction completion, and final cleanup; an expired budget does not grant a fresh timeout. Configure rollback and close limits separately with `CleanupPolicy`. + +`RootCompletion` exposes a value only after successful bounded completion. `RootTransactionException` exposes payload-free `TransactionFacts` and `CleanupFact`: commit acknowledgment and completed transaction boundary are separate observations. Timeout, cancellation, or cleanup failure can leave the business outcome uncertain. The application must recover through its authoritative action/receipt protocol. These technical facts establish no authentication, authorization, financial finality, or proof that no business effect occurred. + +Run the module and its reactor dependencies with `./mvnw -B -ntp -pl :fireflyframework-r2dbc -am verify`. The transaction tests exercise actual Spring transaction infrastructure with controlled R2DBC connections; native PostgreSQL and application journeys require separate integration tests. + ### 2. Define an entity, a filter DTO, and a paginated endpoint ```java @@ -185,6 +197,23 @@ Both beans use `matchIfMissing = true`, so they are active by default and back o This module ships an `application-logging.yml` profile fragment that sets `org.springframework.data.r2dbc` and `io.r2dbc` logging to `INFO`. +### Conditional database infrastructure + +R2DBC utility types can be present transitively in a service that owns no database. Their presence does not require that service to create a connection factory, entity template, or transaction manager. In this source version, database integration activates only when its runtime prerequisites exist: + +| Available infrastructure | Framework behavior | +| --- | --- | +| No R2DBC/transaction runtime classes | Database auto-configuration backs off before loading its dependent beans | +| Classes present, no entity template or connection factory | No database template initialization or transaction-manager creation | +| Application or Boot provides `R2dbcEntityTemplate` | Initialize `FilterUtils` after Boot's data R2DBC auto-configuration | +| One candidate `ConnectionFactory`, no existing reactive transaction manager | Provide `R2dbcTransactionManager` if Boot has not already supplied one | +| Existing `ReactiveTransactionManager` | Preserve the application's selected manager | +| Several connection factories without a primary candidate | Do not choose an arbitrary database; configure the intended transaction boundary explicitly | + +This conditional behavior concerns framework infrastructure. It does not disable Spring Boot's own database auto-configuration selected by a data starter. A service that includes that starter must still configure its database or deliberately exclude the unused Boot data configuration. Database-free services should depend only on the facilities they actually use. + +`FilterUtils` uses a process-wide template reference. Applications with multiple independently selected databases must define the supported filtering/database boundary explicitly; these conditions do not introduce multitenant database routing. + ### How It Works The module registers four Spring Boot auto-configurations (see `META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports`): diff --git a/docs/site/modules/rule-engine-core.md b/docs/site/modules/rule-engine-core.md index b41010306..af7598697 100644 --- a/docs/site/modules/rule-engine-core.md +++ b/docs/site/modules/rule-engine-core.md @@ -24,7 +24,7 @@ Evaluate a validated onboarding rule against application input through the parse org.fireflyframework fireflyframework-r2dbc -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/rule-engine-interfaces.md b/docs/site/modules/rule-engine-interfaces.md index f39853639..27d67674c 100644 --- a/docs/site/modules/rule-engine-interfaces.md +++ b/docs/site/modules/rule-engine-interfaces.md @@ -24,7 +24,7 @@ Choose the request DTO for plain YAML, encoded YAML, or a stored onboarding rule org.fireflyframework fireflyframework-rule-engine-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/rule-engine-models.md b/docs/site/modules/rule-engine-models.md index 879005184..142d0144c 100644 --- a/docs/site/modules/rule-engine-models.md +++ b/docs/site/modules/rule-engine-models.md @@ -24,7 +24,7 @@ Use these repositories when onboarding rules and constants need a stored, versio org.fireflyframework fireflyframework-rule-engine-interfaces -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/rule-engine-sdk.md b/docs/site/modules/rule-engine-sdk.md index 4e2ae8b39..a2b9d1243 100644 --- a/docs/site/modules/rule-engine-sdk.md +++ b/docs/site/modules/rule-engine-sdk.md @@ -24,7 +24,7 @@ A separate onboarding service can call the rule service through generated WebCli org.fireflyframework fireflyframework-rule-engine-models -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/rule-engine-web.md b/docs/site/modules/rule-engine-web.md index 7797055d4..805a4341f 100644 --- a/docs/site/modules/rule-engine-web.md +++ b/docs/site/modules/rule-engine-web.md @@ -24,7 +24,7 @@ Run the rule service when onboarding applications evaluate rules over HTTP or ma org.fireflyframework fireflyframework-rule-engine-sdk -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/rule-engine.md b/docs/site/modules/rule-engine.md index 8224ab2ea..08a38c46f 100644 --- a/docs/site/modules/rule-engine.md +++ b/docs/site/modules/rule-engine.md @@ -51,7 +51,7 @@ This artifact has `pom` packaging and groups child projects. Select a concrete c org.fireflyframework fireflyframework-rule-engine-web -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-adapter-cerbos.md b/docs/site/modules/security-adapter-cerbos.md index f5ad6d8bc..9322b98f1 100644 --- a/docs/site/modules/security-adapter-cerbos.md +++ b/docs/site/modules/security-adapter-cerbos.md @@ -24,7 +24,7 @@ A reviewer action can be checked against Cerbos when this adapter is the configu org.fireflyframework fireflyframework-rule-engine-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-adapter-opa.md b/docs/site/modules/security-adapter-opa.md index 274879b0e..3f2f4ca13 100644 --- a/docs/site/modules/security-adapter-opa.md +++ b/docs/site/modules/security-adapter-opa.md @@ -24,7 +24,7 @@ Delegate an onboarding authorization decision to a configured OPA policy through org.fireflyframework fireflyframework-security-adapter-cerbos -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-adapter-openfga.md b/docs/site/modules/security-adapter-openfga.md index 6486b6437..aefa308dc 100644 --- a/docs/site/modules/security-adapter-openfga.md +++ b/docs/site/modules/security-adapter-openfga.md @@ -24,7 +24,7 @@ Use RelationshipPort to check whether a reviewer is related to a particular appl org.fireflyframework fireflyframework-security-adapter-opa -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-adapter-r2dbc.md b/docs/site/modules/security-adapter-r2dbc.md index 65f432c4d..2c4d1cbe7 100644 --- a/docs/site/modules/security-adapter-r2dbc.md +++ b/docs/site/modules/security-adapter-r2dbc.md @@ -24,7 +24,7 @@ Store revoked token IDs when the application uses PostgreSQL for that security s org.fireflyframework fireflyframework-security-adapter-openfga -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-adapter-vault.md b/docs/site/modules/security-adapter-vault.md index 6cc7ee7a9..9fe33d1fd 100644 --- a/docs/site/modules/security-adapter-vault.md +++ b/docs/site/modules/security-adapter-vault.md @@ -24,7 +24,7 @@ Resolve a verification partner's credential through SecretsPort at runtime. Prov org.fireflyframework fireflyframework-security-adapter-r2dbc -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-api.md b/docs/site/modules/security-api.md index 2fe43b4ea..f104d9562 100644 --- a/docs/site/modules/security-api.md +++ b/docs/site/modules/security-api.md @@ -24,7 +24,7 @@ Represent an authenticated applicant or reviewer with SecurityPrincipal and eval org.fireflyframework fireflyframework-security-adapter-vault -26.10.21-SNAPSHOT +26.10.22 ``` @@ -208,7 +208,7 @@ Runtime footprint is intentionally tiny (see `pom.xml`): | `org.projectlombok:lombok` | provided | `@Builder` on `SecurityPrincipal`. | | `junit-jupiter`, `assertj-core`, `reactor-test` | test | Unit tests. | -Version and parent are managed by `fireflyframework-parent` (CalVer `26.10.21-SNAPSHOT`); consumers inheriting the parent/BOM can omit the version. +Version and parent are managed by `fireflyframework-parent` (CalVer `26.10.22`); consumers inheriting the parent/BOM can omit the version. ```xml org.fireflyframework fireflyframework-security-api -26.10.21-SNAPSHOT +26.10.22 diff --git a/docs/site/modules/security-authorization-server.md b/docs/site/modules/security-authorization-server.md index 30bb79b29..090fee29e 100644 --- a/docs/site/modules/security-authorization-server.md +++ b/docs/site/modules/security-authorization-server.md @@ -24,7 +24,7 @@ Use this separate servlet service when the deployment needs to issue its own OAu ``` diff --git a/docs/site/modules/security-bff.md b/docs/site/modules/security-bff.md index c42d5d626..f4afa4cda 100644 --- a/docs/site/modules/security-bff.md +++ b/docs/site/modules/security-bff.md @@ -24,7 +24,7 @@ A browser applicant portal can keep tokens in a server-side session and present org.fireflyframework fireflyframework-security-authorization-server -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-core.md b/docs/site/modules/security-core.md index ba4701400..3b82885d8 100644 --- a/docs/site/modules/security-core.md +++ b/docs/site/modules/security-core.md @@ -40,7 +40,7 @@ Map the authenticated applicant/reviewer claims and evaluate the configured auth org.fireflyframework fireflyframework-security-bff -26.10.21-SNAPSHOT +26.10.22 ``` @@ -187,7 +187,7 @@ Supporting types: `SecureRequirement` (a framework-neutral view of `@Secure`, bu The engine is deliberately fail-closed at every decision point: - `SecureAuthorizationEvaluator`. a `null` (unauthenticated) principal is denied. Declared dimensions (roles, scopes, permissions, expression) are AND-ed; within a dimension, `requireAll*` selects ALL vs ANY (ANY by default). An empty requirement permits any authenticated principal ("require authentication"). A SpEL expression that evaluates false, or throws / is malformed, denies. -- `EmbeddedPolicyDecisionAdapter`. deny-overrides: any rule DENY wins; rules registered but none permitting yields DENY; a rule that errors becomes INDETERMINATE (never PERMIT). With no rules registered it permits, deferring to RBAC at the PEP (ABAC simply not in use). +- `EmbeddedPolicyDecisionAdapter`. The original constructor preserves legacy deny-overrides semantics and permits with no rules to defer to a separate RBAC gate. The `(List org.fireflyframework fireflyframework-security-core -26.10.21-SNAPSHOT +26.10.22 , boolean strict)` overload with `true` denies missing rules, failed/empty publishers, malformed decisions without an effect and explicit denials, requires an explicit permit, and retains permit obligations in rule order. Normal indeterminate abstention may coexist with an explicit permit. An enforcing caller must fulfill returned obligations or deny; a provider bean is not evidence of an institution policy. The resource-server binding selects this overload with `firefly.security.resource-server.strict-policy: true`. - `InMemoryKeyManagementAdapter`. flagged not-for-production; the autoconfigure layer fails closed at startup if no real key source resolves under prod profiles. - `CaffeineIntrospectionCache`. TTL is bounded by the token's own expiry, so a revoked-by-expiry token cannot outlive its `exp`. diff --git a/docs/site/modules/security-idp-aws-cognito.md b/docs/site/modules/security-idp-aws-cognito.md index 655b5cc62..32221d89f 100644 --- a/docs/site/modules/security-idp-aws-cognito.md +++ b/docs/site/modules/security-idp-aws-cognito.md @@ -24,7 +24,7 @@ Use an existing Cognito user pool for applicant identities. Configure its region ``` diff --git a/docs/site/modules/security-idp-azure-ad.md b/docs/site/modules/security-idp-azure-ad.md index af4432758..bd88811b1 100644 --- a/docs/site/modules/security-idp-azure-ad.md +++ b/docs/site/modules/security-idp-azure-ad.md @@ -24,7 +24,7 @@ Connect applicant or reviewer identity operations to the configured Entra ID or org.fireflyframework fireflyframework-security-idp-aws-cognito -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-idp-internal-db.md b/docs/site/modules/security-idp-internal-db.md index eeb0845d5..69bef8db4 100644 --- a/docs/site/modules/security-idp-internal-db.md +++ b/docs/site/modules/security-idp-internal-db.md @@ -24,7 +24,7 @@ Use the internal database adapter when this service owns its identity store. Con org.fireflyframework fireflyframework-security-idp-azure-ad -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-idp-keycloak.md b/docs/site/modules/security-idp-keycloak.md index fbcdc065b..e234ef842 100644 --- a/docs/site/modules/security-idp-keycloak.md +++ b/docs/site/modules/security-idp-keycloak.md @@ -24,7 +24,7 @@ Manage applicant identities in the configured Keycloak realm. Set its server and org.fireflyframework fireflyframework-security-idp-internal-db -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-idp.md b/docs/site/modules/security-idp.md index f86acf5dd..6b95dcad7 100644 --- a/docs/site/modules/security-idp.md +++ b/docs/site/modules/security-idp.md @@ -24,7 +24,7 @@ Manage applicant identities through IdpAdapter and a selected provider. Keep ide org.fireflyframework fireflyframework-security-idp-keycloak -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-method-policy.md b/docs/site/modules/security-method-policy.md index 0c305a35e..94732ca4f 100644 --- a/docs/site/modules/security-method-policy.md +++ b/docs/site/modules/security-method-policy.md @@ -24,7 +24,7 @@ Protect a reactive review method on a Spring bean with the required reviewer aut org.fireflyframework fireflyframework-security-idp -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-oauth2-client.md b/docs/site/modules/security-oauth2-client.md index c010d3f84..a0e42e73c 100644 --- a/docs/site/modules/security-oauth2-client.md +++ b/docs/site/modules/security-oauth2-client.md @@ -24,7 +24,7 @@ The onboarding service may need an authorized outbound call to a protected partn org.fireflyframework fireflyframework-security-method-policy -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/security-resource-server.md b/docs/site/modules/security-resource-server.md index 0442df474..e535ab4c7 100644 --- a/docs/site/modules/security-resource-server.md +++ b/docs/site/modules/security-resource-server.md @@ -24,7 +24,7 @@ Protect onboarding HTTP routes with JWTs from the configured identity authority. org.fireflyframework fireflyframework-security-oauth2-client -26.10.21-SNAPSHOT +26.10.22 ``` @@ -36,12 +36,13 @@ This binding configures JWT validation. Opaque-token introspection ports exist e Set `OIDC_ISSUER` to the expected token issuer and `OIDC_JWK_SET_URI` to that provider's JWKS endpoint. For an external identity provider, configure both: `issuer` validates the `iss` claim; it does not discover signing keys. Without `jwk-set-uri`, the decoder uses the configured `KeyManagementPort` signing key instead. -With the module on the classpath, the resource server is active with zero code. Every route is locked down. Mark public routes explicitly and read the validated principal directly: +With the module on the classpath, protected routes require a validated bearer token. Authentication alone does not grant financial or insurance operations. Enable strict policy mode, contribute the current project policy, and mark public routes explicitly: ```yaml firefly: security: resource-server: + strict-policy: true permit-matchers: - /actuator/health - /public/** @@ -80,12 +81,29 @@ PolicyRule denyReports() { } ``` +## Header bearer authentication and CSRF + +The default API chain authenticates protected routes through an explicit `Authorization: Bearer org.fireflyframework fireflyframework-security-resource-server -26.10.21-SNAPSHOT +26.10.22 ` header and never restores authentication from a session cookie. Basic, form, query-token and token-cookie credentials are not alternate authorities. Native Spring Security CSRF protection remains enabled with its bearer exemption: legitimate signed bearer POST/PUT/PATCH/DELETE requests do not need CSRF tokens, while unsafe non-bearer requests do. A missing CSRF token can return 403 before authentication, including on explicitly allowlisted browser routes. A valid CSRF token does not authenticate a request. + +Applications that need session login supply their own filter chain through the existing bean-backoff seam and retain appropriate CSRF controls. CSRF state may use WebSession without making that session an API authentication authority. The [canonical header bearer and browser CSRF contract](https://github.com/fireflyframework/fireflyframework-jafly/blob/main/modules/security-resource-server/docs/bearer-api-and-csrf.md) documents exact transport/status compatibility, permitted browser token delivery, and the maintained real-HTTP boundaries. + +## Require an explicit current policy + +`firefly.security.resource-server.strict-policy: true` makes the fallback embedded policy deny when no rules are configured. A failed or empty rule publisher, or a decision without an effect, denies even when another rule permits. Normal `INDETERMINATE` results may abstain, but at least one explicit permit is required and any denial wins. The embedded provider retains permit obligations; an enforcing business boundary must fulfill them or deny access. + +The default URL chain cannot perform row filtering, masking, approval, or other policy obligations. In strict mode it rejects obligation-bearing permits and bounds the current policy publisher to five seconds. A project-owned `PolicyDecisionPort` remains the single authority and replaces the fallback provider; its errors, empty results, unresolved obligations, and timeouts cannot authorize an exchange. A custom filter chain must enforce those checks itself and preserve a catchall denial for unmatched routes. + +The property defaults to `false` for compatibility. In that legacy mode, the embedded provider with no rules permits in order to defer to a separate RBAC gate. Do not confuse a default provider bean with an approved current resource decision. Financial and insurance integrations that require strict mode must also reject missing or false configuration at their own business entry point. Public `permit-matchers` remain explicit authentication exceptions; do not put protected business paths there. + +A signed token proves its validated claims. It does not supply tenant ownership, staff impersonation approval, contract authority, a model approval, or permission to retry a financial effect. Bind each business request to its exact trusted principal/resource and evaluate current authority again on replay and recovery. Keep actor selection out of untrusted headers and request bodies. + ## Configuration and wiring The `firefly.security.resource-server` namespace is bound by [ResourceServerProperties.java](https://github.com/fireflyframework/fireflyframework-jafly/blob/main/modules/security-resource-server/src/main/java/org/fireflyframework/security/rs/ResourceServerProperties.java). Nested configuration is declared in the linked class. Defaults are the field initializers. | Setting | Java type | Source default | | --- | --- | --- | +| `firefly.security.resource-server.strict-policy` | `boolean` | `false` | | `firefly.security.resource-server.permit-matchers` | `List ` | `new ArrayList<>()` | | `firefly.security.resource-server.issuer` | `String` | `not initialized` | | `firefly.security.resource-server.audiences` | `List ` | `new ArrayList<>()` | @@ -177,10 +195,10 @@ This module depends only on `security-webflux` (which transitively brings `secur `ResourceServerAutoConfiguration` contributes, each gated by `@ConditionalOnMissingBean`: - A verifying JWT decoder. `NimbusReactiveJwtDecoder.withPublicKey(...)` built from the active RSA `SigningKey` resolved through `KeyManagementPort.activeSigningKey()`, pinned to `RS256`. If no RSA key resolves, the context fails to start (fail-closed) rather than booting an unprotected server. -- A validator chain. A `DelegatingOAuth2TokenValidator ` always enforcing `JwtTimestampValidator` (exp/nbf with skew), plus `JwtIssuerValidator` when `issuer` is configured and a custom audience validator when `audiences` is non-empty. +- A validator chain. A `DelegatingOAuth2TokenValidator ` always enforcing required expiration and timestamp validation through `AccessTokenValidators.timestamps()` (including nbf/skew validation), plus `JwtIssuerValidator` when `issuer` is configured and a custom audience validator when `audiences` is non-empty. - Claim-to-authority mapping. An `AuthorityMappingPort` (default `ConfigurableAuthorityMapper`) driven by configurable role/scope claim paths and authority prefix, normalizing provider-specific claims (Keycloak/Cognito/Entra) into a uniform authority/scope set. - A framework principal. `JwtToFireflyPrincipalConverter` replaces Spring's default `JwtAuthenticationToken` with a `FireflyAuthenticationToken` whose principal is a `SecurityPrincipal` (built by `PrincipalFactory.fromClaims(...)`), so `@AuthenticationPrincipal SecurityPrincipal` works with its default configuration. -- A default-deny filter chain. `fireflySecurityWebFilterChain` disables CSRF, HTTP-Basic, form-login, and logout; permits only the explicitly configured `permitMatchers`; and routes `anyExchange()` through a `PolicyAuthorizationManager` backed by `PolicyDecisionPort`. Unconfigured routes are denied. +- An authenticated policy filter chain. `fireflySecurityWebFilterChain` disables CSRF, HTTP-Basic, form-login, and logout; permits only the explicitly configured `permitMatchers`; and routes `anyExchange()` through a `PolicyAuthorizationManager` backed by `PolicyDecisionPort`. Anonymous requests to unconfigured routes are denied. Authenticated authorization depends on the configured policy mode; strict mode requires an explicit permit. - Hardened response headers on every response: `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer`, a strict `Content-Security-Policy` (`default-src 'none'; frame-ancestors 'none'`), and HSTS (`includeSubDomains`, 365-day max-age). - Framework defaults for the remaining ports. `EmbeddedPolicyDecisionAdapter` (collecting any `PolicyRule` beans), `LoggingAuditEventAdapter`, and `ReactorSecurityContextAdapter`. all overridable. @@ -202,7 +220,8 @@ All keys live under `firefly.security.resource-server`: | Property | Default | Description | | --- | --- | --- | -| `permit-matchers` | _(empty)_ | Ant-style path patterns served without authentication. The only opt-out from default-deny. | +| `strict-policy` | `false` | Require an explicit permit; reject failed/empty rules and unsupported URL obligations. | +| `permit-matchers` | _(empty)_ | Ant-style path patterns served without authentication. | | `issuer` | _(none)_ | Expected `iss`; when set, a `JwtIssuerValidator` is enforced. | | `audiences` | _(empty)_ | Acceptable `aud` values; when non-empty, an audience validator is enforced. | | `authority-prefix` | `""` | Prefix applied to mapped authorities (e.g. `ROLE_`). | @@ -218,7 +237,7 @@ Bearer token → fireflySecurityWebFilterChain → NimbusReactiveJwtDecoder (RS256, key from KeyManagementPort) → DelegatingOAuth2TokenValidator (timestamp [+ issuer] [+ audience]) → JwtToFireflyPrincipalConverter → FireflyAuthenticationToken(SecurityPrincipal) - → authorizeExchange: permitMatchers → permitAll, else PolicyAuthorizationManager (PolicyDecisionPort, default-deny) + → authorizeExchange: permitMatchers → permitAll, else PolicyAuthorizationManager (PolicyDecisionPort, explicit strict mode or legacy RBAC deferral) → controller / handler ``` diff --git a/docs/site/modules/security-spi.md b/docs/site/modules/security-spi.md index d28ab2976..62f860caa 100644 --- a/docs/site/modules/security-spi.md +++ b/docs/site/modules/security-spi.md @@ -24,7 +24,7 @@ Connect the onboarding service's policy, relationship, revocation, key, or secre ``` diff --git a/docs/site/modules/security-test.md b/docs/site/modules/security-test.md index 8d1544611..207c841f8 100644 --- a/docs/site/modules/security-test.md +++ b/docs/site/modules/security-test.md @@ -24,7 +24,7 @@ Create applicant/reviewer test principals and deterministic policy outcomes for org.fireflyframework fireflyframework-security-spi -26.10.21-SNAPSHOT +26.10.22 ``` @@ -192,7 +192,7 @@ If you are not inheriting the Firefly parent, pin the version explicitly: org.fireflyframework fireflyframework-security-test -26.10.21-SNAPSHOT +26.10.22 test ``` diff --git a/docs/site/modules/security-webflux.md b/docs/site/modules/security-webflux.md index b8502e342..16638dba3 100644 --- a/docs/site/modules/security-webflux.md +++ b/docs/site/modules/security-webflux.md @@ -24,7 +24,7 @@ Project the authenticated reviewer into the framework principal and connect URL org.fireflyframework fireflyframework-security-test -26.10.21-SNAPSHOT +26.10.22 test ``` diff --git a/docs/site/modules/starter-application.md b/docs/site/modules/starter-application.md index 5aec520d9..c184b2666 100644 --- a/docs/site/modules/starter-application.md +++ b/docs/site/modules/starter-application.md @@ -24,7 +24,7 @@ An application-tier onboarding API can use these request/context conventions whi org.fireflyframework fireflyframework-security-webflux -26.10.21-SNAPSHOT +26.10.22 ``` @@ -48,6 +48,10 @@ public class AccountController extends AbstractApplicationController { } ``` +## Application metadata discovery + +Application metadata discovery reads predictable bean types without creating application beans, FactoryBeans or their products. Declare the `@FireflyApplication` main class through a known bean type; the existing immediate-superclass annotation lookup is retained. An unknown product type stays untouched, with existing fallback metadata when no annotation can be resolved. This prevents metadata discovery from reentering Actuator endpoint construction. The guarantee applies to metadata discovery; other integrations retain their own registration and initialization lifecycles. + ## Configuration and wiring The `firefly.application` namespace is bound by [ApplicationLayerProperties.java](https://github.com/fireflyframework/fireflyframework-jafly/blob/main/modules/starter-application/src/main/java/org/fireflyframework/common/application/config/ApplicationLayerProperties.java). Nested configuration is declared in the linked class. Defaults are the field initializers. diff --git a/docs/site/modules/starter-core.md b/docs/site/modules/starter-core.md index 5b4aa43fc..d0ef270a3 100644 --- a/docs/site/modules/starter-core.md +++ b/docs/site/modules/starter-core.md @@ -26,7 +26,7 @@ A larger onboarding service may adopt this bundle for shared HTTP-client and ope org.fireflyframework fireflyframework-starter-application -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/starter-data.md b/docs/site/modules/starter-data.md index fec54dc7f..54d50bb58 100644 --- a/docs/site/modules/starter-data.md +++ b/docs/site/modules/starter-data.md @@ -24,7 +24,7 @@ Background document enrichment or verification jobs may need the data-job and pr org.fireflyframework fireflyframework-starter-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/starter-domain.md b/docs/site/modules/starter-domain.md index 43f084563..46859a638 100644 --- a/docs/site/modules/starter-domain.md +++ b/docs/site/modules/starter-domain.md @@ -24,7 +24,7 @@ Use this dependency bundle when application submission/review grows into a domai org.fireflyframework fireflyframework-starter-data -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/utils.md b/docs/site/modules/utils.md index 3718d112a..5c2e011f6 100644 --- a/docs/site/modules/utils.md +++ b/docs/site/modules/utils.md @@ -24,7 +24,7 @@ After accepting an application, the service can render a confirmation template o org.fireflyframework fireflyframework-starter-domain -26.10.21-SNAPSHOT +26.10.22 ``` @@ -32,6 +32,27 @@ After accepting an application, the service can render a confirmation template o The Java examples in this section are API fragments for application code. Use the stated imports and supply the application-owned collaborators described alongside them. +### Render a restricted published fragment + +Use `RestrictedHtmlTemplateRenderer` for the closed `RESTRICTED_XHTML_V1` profile. Its independent configuration accepts a deeply copied JSON-compatible model, exact BigInteger/BigDecimal and integral numbers; it rejects arbitrary Java/FreeMarker objects and floating-point values. Model strings remain data. Existing `TemplateRenderUtil` retains its application-owned classpath/file/PDF behavior. + +```java +import org.fireflyframework.utils.template.RestrictedHtmlTemplateRenderer; +import java.util.Map; + +var renderer = new RestrictedHtmlTemplateRenderer(); +var result = renderer.render(" org.fireflyframework fireflyframework-utils -26.10.21-SNAPSHOT +26.10.22 ${recipient}
", + Map.of("recipient", "Synthetic & Bank")); +if (!result.success()) { + throw new IllegalArgumentException(result.code()); +} +String fragment = result.html(); //Synthetic & Bank
+``` + +The renderer forces HTML escaping, disables API/class construction and external template loaders, then validates a well-formed XHTML fragment with a closed print tag/attribute profile. Scripts, styles, links, images, custom/external entity declarations, DTDs and unknown markup reject the entire output. No partial document is returned. Successful output must contain visible text. Application-owned branding/shells and financial facts remain separate responsibilities; this profile does not render arbitrary CSS/assets or PDF. + +`RestrictedTemplateLimits.defaults()` bounds template256KiB, output1MiB, model5000nodes/depth32, strings65536UTF16 units,10000template elements and5000ms cooperative checks. Custom limits are immutable per instance. Codes are `UNSUPPORTED_TEMPLATE`, `UNSUPPORTED_MODEL`, `RENDER_LIMIT`, `EMPTY_OUTPUT`, `INVALID_HTML` and `RENDER_FAILED`; they do not disclose source/model fragments. Cooperative tracing cannot interrupt one expensive expression or establish a hard heap/RSS/CPU boundary. A service accepting published content must supply its separately qualified worker-process lifetime/heap controls when required. This utility alone does not provide that isolation. + ### Render a template to HTML Place `invoice.ftl` under `src/main/resources/templates/`: diff --git a/docs/site/modules/validators.md b/docs/site/modules/validators.md index 59448d004..5c7952dd8 100644 --- a/docs/site/modules/validators.md +++ b/docs/site/modules/validators.md @@ -24,7 +24,7 @@ The tutorial validates name and email with Jakarta constraints. Later onboarding``` diff --git a/docs/site/modules/web-idempotency.md b/docs/site/modules/web-idempotency.md new file mode 100644 index 000000000..1a55c43d9 --- /dev/null +++ b/docs/site/modules/web-idempotency.md @@ -0,0 +1,57 @@ +# HTTP idempotency and durable business effects + +HTTP response reuse is an optimization around an authorized request. The business service remains responsible for recording a durable operation key and applying its effects exactly once within its own transaction boundary. A cached response is not an operation ledger. + +## Configure the boundary + +1. Set `spring.application.name` or `firefly.web.idempotency.namespace`. Use an explicit service/environment namespace whenever deployments share a cache backend. +2. Authenticate and authorize the request before `IdempotencyWebFilter` (order `-80`). The default resolver requires an authenticated Spring `Authentication` containing a Jafly `SecurityPrincipal` with issuer and subject. It resolves tenant through the existing `TenantResolverPort`, or through the principal's nonblank `tenantId` when that port is absent. Unresolved trust bypasses response reuse. +3. For another authentication stack, implement `IdempotencyScopeResolver` as a bean. Return a stable namespace derived from verified issuer, caller, authorized tenant, and authorization state. Return `Mono.empty()` when trust cannot be resolved. Never derive authority from unvalidated request headers. +4. Select and qualify the backing cache using `firefly.cache.*`. The HTTP cache has its own manager/prefix. Namespace isolation is still required when multiple deployments use the same backend. +5. Keep the durable operation key inside the business command. Validate ownership and authorization on every request, including retries. If authorization exists only inside the handler, disable HTTP response reuse for that endpoint or move its policy check before this filter. + +## Request and response contract + +| Input or condition | Behavior | +| --- | --- | +| No retry header | Handler executes normally | +| `GET`, `HEAD`, `OPTIONS`, `TRACE` | Header is ignored; no mutation-cache read or write | +| Missing application namespace or unresolved trusted caller/tenant | Handler executes without response reuse | +| Multiple, blank, or longer-than-256-character keys on an eligible mutation | `400`, before handler execution | +| Same scoped key and byte-identical request fingerprint | Successful response is replayed | +| Same scoped key with changed body, content type, accepted representation/encoding/language, content encoding, or `If-Match`/`If-None-Match` | `409`, before another handler execution | +| Concurrent identical requests in one filter instance | One handler execution; other callers await its result | +| Concurrent conflicting input | Immediate `409`, without waiting for the original response | +| Successful empty response, including `204` | Empty response is reusable and waiting callers complete | +| Handled non-success response | Concurrent waiters receive it; later retries execute again | +| Handler failure/cancellation | Waiting callers terminate; local in-flight ownership is released | +| Oversized request body | `413`, before handler execution | +| Oversized response body | Buffering fails; no successful cache entry is recorded. The business effect may already exist and must be recovered through its durable key | + +The scoped key includes application namespace, trusted identity namespace, method, raw URI path, raw query, and the client key. Its `v2` prefix and SHA-256 digest avoid replaying legacy unscoped entries or storing raw identity/key material as cache keys. Default trusted identity includes issuer, subject, resolved tenant, and sorted current roles/scopes. + +The request fingerprint is SHA-256 over the exact request bytes plus `Content-Type`, `Accept`, `Content-Encoding`, `Accept-Encoding`, `Accept-Language`, `If-Match`, and `If-None-Match`. Semantically equivalent JSON with different bytes conflicts deliberately. Use a different retry key for changed input. Every ordered value of each selected header is included, including later field lines. Names, cardinalities, and values are framed with explicit UTF-8 byte lengths; combining two field lines into one comma-separated line may conservatively conflict even when HTTP semantics are equivalent. + +Only `2xx` responses are retained. Replays preserve status, body, content type, `Location`, `ETag`, `Content-Language`, `Content-Encoding`, `Content-Location`, `Content-Disposition`, `Last-Modified`, and `Vary`. Application-produced encoded bytes retain their complete `Content-Encoding` metadata; accepted encodings participate in the fingerprint. Cookies and authentication headers are not replayed. `Vary` is supported only when every named field belongs to the selected fingerprint headers. A response with `Vary: *` or an unsupported field is not retained; an already waiting duplicate receives `409` and must use durable business recovery. Later requests execute through the normal business handler. This includes CORS policies that add `Vary: Origin`, since Origin is outside this selected representation contract; do not assume that supplying a key makes retention active for such a route. Cache retention is configured in hours (24 by default); body limits default to 1 MiB for requests and 4 MiB for responses. This filter buffers a complete response and is unsuitable for streaming: opt out with `@DisableIdempotency` or disable the filter. + +## Failure, restart, and multiple processes + +Bounded response bytes are captured while writing, but status, content type, and safe replay headers are read only after successful delegate completion, including Spring `beforeCommit` callbacks. Retention decisions and waiting duplicates therefore use final committed metadata. A failed commit cannot complete a waiter with an earlier successful snapshot. + +Cache writes participate in the reactive request completion; they are not detached subscriptions. The local in-flight claim remains owned until cache completion, and a second lookup closes the completed-owner handoff race. A failed best-effort cache write can still leave the original business effect committed. Subsequent requests must recover from that effect's durable operation key. + +A second filter instance can replay an existing cache entry, including its request fingerprint. In-flight claims are held in memory and are not shared between processes. Two instances receiving the same key concurrently can both reach the business handler; a shared cache does not make those executions atomic. Crashes, cache outages, cache expiration, and a response limit failure require durable business reconciliation. Do not describe this feature as distributed exactly-once processing. + +Restart qualification of a business journey must verify its database operation claim, original effect, status history, and recovery response separately. The framework tests exercise scoped cache metadata and HTTP behavior, not bank ledger or provider transaction finality. + +## Qualification + +Run from the Jafly repository root with JDK 25: + +```bash +./mvnw -pl :fireflyframework-web -am verify +``` + +`ScopedIdempotencyWebFilterTest` covers route/method/query and issuer/subject/tenant/service isolation, changed-input rejection, concurrent same-input execution, body preservation, empty responses, failure/cancellation cleanup, delayed cache persistence, and bounded buffers. `IdempotencyHttpIntegrationTest` runs actual Reactor Netty HTTP against Spring WebFlux controller methods on a private loopback port. It verifies body delivery, cross-route and read/mutation separation, payload conflict, actual `@DisableIdempotency` behavior, raw-wire gzip decoding on original/replay, safe metadata and credential/cookie exclusions, complete multi-value header conflicts, field boundaries, unsupported Vary behavior, and final status/encoding/Vary from `beforeCommit` callbacks. Deterministic concurrent HTTP fixtures verify pending duplicates and failed commits without production test hooks. `IdempotencyAutoConfigurationTest` exercises real Caffeine cache wiring, application namespace fallback, and a custom trusted resolver. + +These are local framework qualification cases. Consumer adoption requires building the exact framework source artifact, qualifying each service's authentication/tenant/authorization configuration, and repeating its real experience-to-core journey tests. A framework build alone does not establish that deployed consumers use it. diff --git a/docs/site/modules/web.md b/docs/site/modules/web.md index 2aa68ee47..a7180591a 100644 --- a/docs/site/modules/web.md +++ b/docs/site/modules/web.md @@ -24,7 +24,7 @@ The onboarding HTTP boundary needs consistent error responses and carefully chos org.fireflyframework fireflyframework-validators -26.10.21-SNAPSHOT +26.10.22 ``` @@ -88,19 +88,35 @@ A typical error response (RFC 7807 Problem Details): ### Idempotent requests -Clients send a unique `X-Idempotency-Key` header on a write request; retries with the same key replay the original cached response instead of re-executing the handler. Opt an endpoint out with `@DisableIdempotency`: +The optional `X-Idempotency-Key` header enables bounded HTTP response reuse for `POST`, `PUT`, `PATCH`, and `DELETE`. `GET`, `HEAD`, `OPTIONS`, and `TRACE` always pass through. The [HTTP idempotency guide](web-idempotency.md) describes setup, scope, recovery, and the executable qualification cases. + +Configure a stable application/environment namespace and authenticated tenant context before relying on response reuse: + +```yaml +spring: + application: + name: payment-service +firefly: + web: + idempotency: + namespace: payment-service:sandbox + max-request-body-bytes: 1048576 + max-response-body-bytes: 4194304 +``` + +The default resolver uses an authenticated Jafly `SecurityPrincipal` and the existing `TenantResolverPort` when available. Without that port, it requires `SecurityPrincipal.tenantId()`. Missing trusted identity, tenant, or application namespace bypasses HTTP response reuse. A caller-supplied tenant header never establishes identity. Services using a different authentication model can provide an `IdempotencyScopeResolver` bean derived from their verified server-side context. + +A key is scoped by service/environment namespace, issuer, subject, tenant, current roles/scopes, HTTP method, raw path, and raw query. A different payload or relevant representation/precondition header under that scope returns `409` before another handler execution. Byte-identical input replays a successful status/body and selected representation headers, including `Location`. JSON whitespace and field order count as input changes. + +Enforce authorization before the filter. HTTP response reuse cannot replace resource authorization, durable business idempotency, transaction isolation, or distributed coordination. The cache can expire or be unavailable, and in-flight coordination belongs to one filter instance. Financial effects must retain their authoritative durable business key even when a cache is enabled. + +Opt streaming or other unsuitable controller methods out with `@DisableIdempotency`; auto-configuration resolves actual controller metadata before response reuse. Manually constructed filters must supply handler mappings to honor that annotation. Disable the entire filter with `firefly.web.idempotency.enabled=false` when endpoint-specific metadata is unavailable. ```java import org.fireflyframework.web.idempotency.annotation.DisableIdempotency; -@PostMapping("/payments") -public Mono org.fireflyframework fireflyframework-web -26.10.21-SNAPSHOT +26.10.22 pay(@RequestBody PaymentRequest req) { - // Send header "X-Idempotency-Key: "; a retry returns the same response. - return paymentService.charge(req); -} - @PostMapping("/heartbeat") -@DisableIdempotency // never deduplicated, header hidden from OpenAPI +@DisableIdempotency public Mono heartbeat() { return monitor.ping(); } @@ -133,6 +149,8 @@ The `firefly.web.cors` namespace is bound by [CorsProperties.java](https://githu The `firefly.web.error-handling` namespace is bound by [ErrorHandlingProperties.java](https://github.com/fireflyframework/fireflyframework-jafly/blob/main/modules/web/src/main/java/org/fireflyframework/web/error/config/ErrorHandlingProperties.java). Nested configuration is declared in the linked class. Defaults are the field initializers. +Source coordinates and exception type attached by the metadata aspect follow `include-debug-info`; cause type/message follow `include-exception-cause`. Both are omitted by default and suppressed under the `prod` or `production` profile even when the flags are enabled. This applies to business, converted HTTP-status, and validation metadata. The response filter copies metadata, preserving the exception's internal diagnostic record and ordinary business fields such as an operation ID or retry delay. These controls cover the framework's known diagnostic metadata keys; application-provided messages and custom metadata still need an appropriate disclosure policy. + | Setting | Java type | Source default | | --- | --- | --- | | `firefly.web.error-handling.include-stack-trace` | `boolean` | `false` | @@ -156,6 +174,9 @@ The `firefly.web.idempotency` namespace is bound by [IdempotencyProperties.java] | --- | --- | --- | | `firefly.web.idempotency.enabled` | `boolean` | `true` | | `firefly.web.idempotency.header-name` | `String` | `"X-Idempotency-Key"` | +| `firefly.web.idempotency.namespace` | `String` | unset; auto-configuration uses `spring.application.name` | +| `firefly.web.idempotency.max-request-body-bytes` | `int` | `1048576` | +| `firefly.web.idempotency.max-response-body-bytes` | `int` | `4194304` | | `firefly.web.idempotency.cache` | `Cache` | `new Cache()` | The `firefly.web.http-request-logging` namespace is bound by [HttpRequestLoggingProperties.java](https://github.com/fireflyframework/fireflyframework-jafly/blob/main/modules/web/src/main/java/org/fireflyframework/web/logging/config/HttpRequestLoggingProperties.java). Nested configuration is declared in the linked class. Defaults are the field initializers. @@ -309,6 +330,9 @@ firefly: idempotency: enabled: true # matchIfMissing=true (on by default) header-name: X-Idempotency-Key + namespace: payment-service:sandbox # unique service/environment sharing a cache backend + max-request-body-bytes: 1048576 + max-response-body-bytes: 4194304 cache: ttl-hours: 24 # TTL for cached idempotent responses @@ -409,6 +433,43 @@ public class OpenApiGenApplication { The Maven profile and plugin configuration are inherited from `fireflyframework-parent`. +### Named Instant schemas + +Use the named scalar when an owner field is a real `java.time.Instant` and its generated client must retain the instant, offset interpretation, and nanoseconds. Existing `LocalDateTime` fields can keep their current SDK configuration. + +Use a framework revision containing this registrar; earlier artifacts do not implement the flag. The capability is disabled by default. Enable it explicitly in the application configuration **and** in the original producer application's `openapi-gen` profile: + +```yaml +firefly: + openapi: + instant-scalar: + enabled: true +``` + +Normal auto-configuration and `@EnableOpenApiGen` import the same property-gated registrar. A coincidentally named domain schema has no effect while this property is missing or false. + +Annotate only the actual Instant field: + +```java +import io.swagger.v3.oas.annotations.media.Schema; +import java.time.Instant; + +public record Observation( + @Schema(ref = "#/components/schemas/FireflyInstant") Instant recordedAt) {} +``` + +When an enabled document references this reserved name, the registrar adds `FireflyInstant` as the primitive `type: string`, `format: date-time` schema. An existing different definition fails generation instead of being overwritten. The registrar does not change runtime JSON serializers or the global ModelConverters registry. It visits modeled OpenAPI 3.0/3.1 schema locations, including path parameters, webhooks, component path items, and encoding headers; examples, descriptions, and arbitrary extension data are not opt-in signals, and remote references are not resolved. + +In the owner SDK's original `openapi-generator-maven-plugin` execution, use the standard mapping **list** supported by generator 7.26: + +```xml + + +``` + +Keep an existing `FireflyInstant=java.time.Instant +java8-localdatetime ` for legacy fields. The explicit named mapping applies to the annotated scalar, rather than changing all date-time models. Generate the schema and SDK through the owner's original producer; do not edit the generated schema or Java files. Inspect generated field types and verify client serialization/deserialization against the real owner response before adopting this in a service. Framework source tests and generator contract vectors do not establish that service's producer, transport, release, or deployment readiness. + ## Continue the application Use the [Security Resource Server guide](security-resource-server.md) for the next part of this integration. The runnable submission/query service is in [Getting started](../getting-started.md). diff --git a/docs/site/modules/webhooks-core.md b/docs/site/modules/webhooks-core.md index 421d82cc8..a9ec48aeb 100644 --- a/docs/site/modules/webhooks-core.md +++ b/docs/site/modules/webhooks-core.md @@ -24,7 +24,7 @@ Validate/enrich the inbound verification-provider request and publish its config``` diff --git a/docs/site/modules/webhooks-interfaces.md b/docs/site/modules/webhooks-interfaces.md index e4025d5c5..11daa02e4 100644 --- a/docs/site/modules/webhooks-interfaces.md +++ b/docs/site/modules/webhooks-interfaces.md @@ -24,7 +24,7 @@ Carry the provider name, JSON payload, event ID, and request metadata from verif org.fireflyframework fireflyframework-webhooks-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/webhooks-processor.md b/docs/site/modules/webhooks-processor.md index db07da6d7..51875f3ae 100644 --- a/docs/site/modules/webhooks-processor.md +++ b/docs/site/modules/webhooks-processor.md @@ -34,7 +34,7 @@ Implement the worker that turns a verified provider event into an application-st org.fireflyframework fireflyframework-webhooks-interfaces -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/webhooks-sdk.md b/docs/site/modules/webhooks-sdk.md index 3cde40628..41dda4b37 100644 --- a/docs/site/modules/webhooks-sdk.md +++ b/docs/site/modules/webhooks-sdk.md @@ -24,7 +24,7 @@ An integration client can use the generated ingestion contract when sending prov org.fireflyframework fireflyframework-webhooks-processor -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/webhooks-web.md b/docs/site/modules/webhooks-web.md index a5cf1947d..80b5896ba 100644 --- a/docs/site/modules/webhooks-web.md +++ b/docs/site/modules/webhooks-web.md @@ -24,7 +24,7 @@ Expose the verification-provider ingestion endpoint through this service after i org.fireflyframework fireflyframework-webhooks-sdk -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/modules/webhooks.md b/docs/site/modules/webhooks.md index 642a92d61..8502d5163 100644 --- a/docs/site/modules/webhooks.md +++ b/docs/site/modules/webhooks.md @@ -51,7 +51,7 @@ This artifact has `pom` packaging and groups child projects. Select a concrete c org.fireflyframework fireflyframework-webhooks-web -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/docs/site/releases.md b/docs/site/releases.md index 61daac0cb..b411028d6 100644 --- a/docs/site/releases.md +++ b/docs/site/releases.md @@ -2,7 +2,7 @@  -The parent, BOM, and libraries share one version train. The working version is `26.10.21-SNAPSHOT`. See GitHub Releases and the registries for published artifacts. +The parent, BOM, and libraries share one version train. The working source version is `26.10.22`; publication is a separate release-workflow operation. See GitHub Releases and the registries for published artifacts. ## Workflows @@ -81,3 +81,13 @@ actionlint .github/workflows/*.yml ``` Unit tests cover valid and invalid version inheritance, tag/commit mismatches, mixed versions, parent/BOM cycles, duplicate GAVs, missing projects, path traversal, required attachments, signed bundle completeness, and remote HTTP/digest failures using an isolated local HTTP server. These tests do not publish artifacts or inspect credentials. + +## Source-owned Batch namespace intent + +The current reactor has 87 projects and 85 publishable artifacts, requiring all 316 POM/JAR/source/Javadoc files. The historical canonical map still records 84 publication homes; its captured association file remains unchanged, including additional historical package rows. `scripts/initial-package-intent.json` is a separate source-owned intention for `org.fireflyframework:fireflyframework-batch` in the public `fireflyframework/fireflyframework-jafly` repository, ID 1423952543. It does not describe an existing package, allocate a namespace, grant write permission, select a version, or reserve a patch counter. + +An explicit `github_packages.py plan --initial-intent scripts/initial-package-intent.json` composes all 85 homes and requires a complete 316-file inventory. The plan retains the exact intent digest and historical association digest separately. It stages local files but produces no publisher POM, settings or deploy goals. Omitting the intent still refuses the unmapped Batch coordinate. + +Execution revalidates the committed intent and requires an actual HTTP 200 response for the existing Batch package with the exact Maven name, public visibility, owner and repository name/ID before creating publisher/settings files or invoking Maven. HTTP 404 is `UNKNOWN` and refuses; an inaccessible namespace is never treated as absence or creation authority. Conflicting metadata, authorization failures and HTTP 409 also refuse. The 84 historical bootstrap policies remain unchanged. + +This source intent does not make first publication runnable. An independently authorized provider-side allocation and exact readable existing-package association are still prerequisites. No package is created or uploaded by preparing the intent; no GitHub association or first-publication qualification follows from mocked source tests. All85 artifacts must be accounted for; Batch is never silently omitted. The human-selected `YY.MM.PATCH` counter policy is unchanged. diff --git a/docs/site/versioning.md b/docs/site/versioning.md index 84e8cbdd0..17fec0b8b 100644 --- a/docs/site/versioning.md +++ b/docs/site/versioning.md @@ -2,13 +2,13 @@  -Jafly uses padded CalVer versions: `YY.MM.PATCH` for a final release and `YY.MM.PATCH-SNAPSHOT` during development. The final component counts releases within the month; it does not represent a date. The current development version is `26.10.21-SNAPSHOT`. +Jafly uses `YY.MM.PATCH` for a final release and `YY.MM.PATCH-SNAPSHOT` during development. The year and month use two digits. The final component is a nonnegative release counter within the month, not a day, and has no two-digit limit: `26.10.6`, `26.10.32`, and `26.10.100` are valid. Existing two-digit zero-padding, such as `26.10.06`, remains supported. The current development version is `26.10.22`. ## What the version identifies A framework train covers the consumer parent, independent BOM, libraries, adapters, and component aggregators. Nested projects may inherit their effective version from a local framework parent. Internal framework dependencies select that same train; third-party libraries retain their own managed versions. -An application's version remains separate. An application at `1.0.0` can use framework train `26.10.21-SNAPSHOT`; the public parent/BOM use `fireflyframework.version` rather than the consumer's `${project.version}` for internal dependency management. +An application's version remains separate. An application at `1.0.0` can use framework train `26.10.22`; the public parent/BOM use `fireflyframework.version` rather than the consumer's `${project.version}` for internal dependency management. ## Development snapshots @@ -23,7 +23,7 @@ The manual snapshot workflow publishes to GitHub Packages only after verificatio ## Change the train ```sh -python3 scripts/set_version.py 26.10.21-SNAPSHOT +python3 scripts/set_version.py 26.10.22 python3 scripts/release_tools.py check python3 -m unittest discover -s tests -v python3 -m unittest discover -s tests/release -v diff --git a/mkdocs.yml b/mkdocs.yml index 5e5feaf46..fa381a8b1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -184,9 +184,11 @@ nav: - "ECM eSignature: Logalty": modules/ecm-esignature-logalty.md - "Business coordination": - "Orchestration": modules/orchestration.md + - "Durable batch": modules/batch.md - "Service operations": - "Observability": modules/observability.md - "Web": modules/web.md + - "HTTP idempotency": modules/web-idempotency.md - "Config server": modules/config-server.md - "Team-wide architecture": - "Build parent": modules/parent.md diff --git a/modules/agentic-bridge/README.md b/modules/agentic-bridge/README.md index 30ef4a103..9db9784e8 100644 --- a/modules/agentic-bridge/README.md +++ b/modules/agentic-bridge/README.md @@ -31,7 +31,7 @@ This artifact has `pom` packaging and groups child projects. Select a concrete c org.fireflyframework fireflyframework-webhooks-interfaces -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/modules/agentic-bridge/docs/QUICK_START.md b/modules/agentic-bridge/docs/QUICK_START.md index 8027bd875..3a114cd16 100644 --- a/modules/agentic-bridge/docs/QUICK_START.md +++ b/modules/agentic-bridge/docs/QUICK_START.md @@ -13,7 +13,7 @@ Spring Boot service in five minutes. org.fireflyframework fireflyframework-agentic-bridge-core -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/README.md b/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/README.md index 311f47b96..2aa39c0c0 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/README.md +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/README.md @@ -20,7 +20,7 @@ Configure the primary agent endpoint and authentication for the reviewer-assista org.fireflyframework fireflyframework-agentic-bridge-starter -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/pom.xml b/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/pom.xml index 6978b9572..cd1398213 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/pom.xml +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-autoconfigure/pom.xml @@ -7,7 +7,7 @@ org.fireflyframework fireflyframework-agentic-bridge-autoconfigure -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-core/README.md b/modules/agentic-bridge/fireflyframework-agentic-bridge-core/README.md index a60613b71..2a5e64b7d 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-core/README.md +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-core/README.md @@ -20,7 +20,7 @@ Call a remote agent from the onboarding service when a configured assistant is n org.fireflyframework fireflyframework-agentic-bridge -26.10.21-SNAPSHOT +26.10.22 ../pom.xml ``` diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-core/pom.xml b/modules/agentic-bridge/fireflyframework-agentic-bridge-core/pom.xml index 7172a0e05..0ac923662 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-core/pom.xml +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-core/pom.xml @@ -7,7 +7,7 @@ org.fireflyframework fireflyframework-agentic-bridge-core -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-samples/pom.xml b/modules/agentic-bridge/fireflyframework-agentic-bridge-samples/pom.xml index c7b2a0cf8..e19ded244 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-samples/pom.xml +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-samples/pom.xml @@ -7,7 +7,7 @@ org.fireflyframework fireflyframework-agentic-bridge -26.10.21-SNAPSHOT +26.10.22 ../pom.xml diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/README.md b/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/README.md index 0d902bc4e..e9fac7547 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/README.md +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/README.md @@ -20,7 +20,7 @@ Add this bundle if the onboarding service needs Spring-managed agent client/brid org.fireflyframework fireflyframework-agentic-bridge -26.10.21-SNAPSHOT +26.10.22 ../pom.xml ``` diff --git a/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/pom.xml b/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/pom.xml index 279cea7e6..9157c23c1 100644 --- a/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/pom.xml +++ b/modules/agentic-bridge/fireflyframework-agentic-bridge-starter/pom.xml @@ -7,7 +7,7 @@ org.fireflyframework fireflyframework-agentic-bridge-starter -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/agentic-bridge/pom.xml b/modules/agentic-bridge/pom.xml index 2cd1c3eb2..49a2c930f 100644 --- a/modules/agentic-bridge/pom.xml +++ b/modules/agentic-bridge/pom.xml @@ -7,13 +7,13 @@ org.fireflyframework fireflyframework-agentic-bridge -26.10.21-SNAPSHOT +26.10.22 ../pom.xml org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml org.fireflyframework fireflyframework-agentic-bridge -26.10.21-SNAPSHOT +26.10.22 pom Firefly Framework - Agentic Bridge diff --git a/modules/backoffice/README.md b/modules/backoffice/README.md index a994f56d7..1dfeee15f 100644 --- a/modules/backoffice/README.md +++ b/modules/backoffice/README.md @@ -20,7 +20,7 @@ A reviewer portal can use backoffice contexts and controller bases to expose int``` diff --git a/modules/backoffice/pom.xml b/modules/backoffice/pom.xml index fca732422..d95af1a99 100644 --- a/modules/backoffice/pom.xml +++ b/modules/backoffice/pom.xml @@ -7,12 +7,12 @@ org.fireflyframework fireflyframework-backoffice -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml fireflyframework-backoffice -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Common Backoffice Library diff --git a/modules/batch/README.md b/modules/batch/README.md new file mode 100644 index 000000000..1d0521654 --- /dev/null +++ b/modules/batch/README.md @@ -0,0 +1,131 @@ + + +# Native Durable Batch + +[Site guide](https://fireflyframework.github.io/fireflyframework-jafly/modules/batch/) · [Business coordination](https://fireflyframework.github.io/fireflyframework-jafly/capabilities/business-coordination/) + +Optional Spring Batch JDBC infrastructure for application-owned restartable jobs. This module uses the framework's Boot-managed Spring Batch 6.0.5, a PostgreSQL JobRepository, explicit JDBC transactions, and a bounded application-owned executor. + +Module qualification has passed 56 tests, including actual PostgreSQL rollback/checkpoints and separate packaged JVM stop, hard-kill/recovery/restart, and concurrent launch. Whole-framework compatibility and independent review remain pending. No published availability or product adoption is claimed. + +## Add the module explicitly + +```xml ++ +``` + +The framework parent/BOM manage this optional coordinate. Existing starters do not include it. The module does not bring Boot's Batch or JDBC starters, create a DataSource, start a scheduler, launch a job, or apply database DDL. An application supplies its PostgreSQL JDBC driver and the normal resources below. + +## Select three resources and disable startup jobs + +```yaml +spring: + batch: + job: + enabled: false +firefly: + batch: + durable: + enabled: true + data-source-ref: batchMetadataDataSource + transaction-manager-ref: batchMetadataTransactions + task-executor-ref: batchWorkers +``` + +The feature defaults to disabled. Enabled mode requires all three exact bean names and explicit `spring.batch.job.enabled=false`. A conventional name, `@Primary`, another available database, or an R2DBC connection factory does not select a fallback. + +The transaction manager must be a `DataSourceTransactionManager` or its `JdbcTransactionManager` subtype whose DataSource is the same selected object. The executor must be an initialized `ThreadPoolTaskExecutor` with equal core/max sizes from 1 to 32, queue capacity zero, and `ThreadPoolExecutor.AbortPolicy`. The application owns its lifecycle. Resource validation happens before a metadata connection is opened. + +This application configuration fragment uses an application-provided PostgreSQL driver and environment-bound connection properties: + +```java +import java.util.concurrent.ThreadPoolExecutor; +import javax.sql.DataSource; +import org.postgresql.ds.PGSimpleDataSource; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.core.env.Environment; +import org.springframework.jdbc.support.JdbcTransactionManager; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; + +@Configuration(proxyBeanMethods = false) +class BatchResources { + @Bean + DataSource batchMetadataDataSource(Environment environment) { + var source = new PGSimpleDataSource(); + source.setURL(environment.getRequiredProperty("application.batch.jdbc-url")); + source.setUser(environment.getRequiredProperty("application.batch.username")); + source.setPassword(environment.getRequiredProperty("application.batch.password")); + return source; + } + + @Bean + JdbcTransactionManager batchMetadataTransactions( + @Qualifier("batchMetadataDataSource") DataSource source) { + return new JdbcTransactionManager(source); + } + + @Bean + ThreadPoolTaskExecutor batchWorkers() { + var executor = new ThreadPoolTaskExecutor(); + executor.setCorePoolSize(2); + executor.setMaxPoolSize(2); + executor.setQueueCapacity(0); + executor.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy()); + executor.setThreadNamePrefix("application-batch-"); + return executor; + } +} +``` + +Use the host application's normal secret configuration. JDBC work is blocking: launch and operate it from appropriate worker threads, without blocking a reactive HTTP event loop. + +## Provision the exact native schema + +The resource `org/fireflyframework/batch/jdbc/schema-postgresql-6.0.5.sql` is an unchanged copy of the [Spring Batch 6.0.5 PostgreSQL schema](https://github.com/spring-projects/spring-batch/blob/v6.0.5/spring-batch-core/src/main/resources/org/springframework/batch/core/schema-postgresql.sql). Its SHA-256 is `f2ac75cae966e4eb1cd97ee6b11f130495b7109153754a7983befb87614d6e38`. + +Have the deployment owner apply it explicitly through the application's controlled migration process. It is outside Flyway's default migration location and has no automatic initializer. Select the intended schema through the DataSource. The initial supported table prefix is `BATCH_`; the three sequence names are `BATCH_JOB_INSTANCE_SEQ`, `BATCH_JOB_EXECUTION_SEQ`, and `BATCH_STEP_EXECUTION_SEQ`. + +The runtime role needs schema usage, native table read/insert/update access, and sequence usage/read access. It does not need schema ownership or CREATE permission. Readiness checks PostgreSQL identity, required columns/types, native sequences, and access without inserting a job or advancing a sequence. These checks do not certify storage hardware, backup policy, or every migration constraint. Native administrative deletion APIs additionally require deletion privileges if an owner elects to expose them. + +## Use native steps and native operator APIs + +`JdbcBatchSteps.chunk(name, chunkSize)` returns the native Batch 6 `ChunkOrientedStepBuilder`. `tasklet(name, tasklet)` returns its native tasklet builder. Both bind the explicitly selected JDBC transaction manager. Normal readers, processors, writers, streams, listeners, and job builders remain native Spring Batch APIs. + +A restartable reader must save a stable cursor in its `ExecutionContext` and reopen from that cursor. A committed checkpoint must refer to an immutable input snapshot, with a stable order and item identity. Re-querying today's changing account set under an old cursor does not establish a valid restart. + +A JDBC writer participates in the local transaction only when it uses the same selected DataSource through transaction-aware Spring JDBC access such as `JdbcTemplate`/`DataSourceUtils`. A different database, independent connection, R2DBC operation, or remote SDK call is outside that local transaction. A durable metadata repository alone does not establish atomic item writes: the native chunk builder defaults to a resource-free transaction manager unless explicitly configured. + +Use Batch 6 object-based `JobOperator.start(Job, JobParameters)`, `stop(JobExecution)`, `restart(JobExecution)`, and `recover(JobExecution)`, then re-read stored state through `JobRepository`. Keep the job registered under its stable name. A stop acknowledgment is a request; the current chunk may still commit. Wait for the stored stopped state and inspect its actual checkpoint before restart. A failed attempt and a new successful attempt remain separate executions of the same instance. + +There is no automatic recover/restart worker. Before native recovery after a hard failure, establish that the previous process cannot continue; for a distributed business effect, also establish the owner's lease/fence and producer safety. Native recovery changes metadata and does not fence an old remote request or prove that an external effect did not occur. + +## Project-owned providers + +Ordinary project repository/operator beans replace defaults without `@Primary` or special method names. A project `DefaultBatchConfiguration` or `@EnableBatchProcessing` configuration backs off the default provider group. Deferred Boot defaults preserve either registration order and actual project component scans. Enabled-mode resource/schema/readiness checks still run, require a single repository and operator, and reject known resource-free repositories. + +An opaque custom provider must use the selected JDBC resources and pass actual persistence/restart qualification. An interface or bean name cannot prove its storage behavior. Disabled mode leaves the application's existing Batch configuration alone. + +## Bind a business run outside the framework + +The application owns a durable run claim containing the exact tenant, book, job kind, cutoff, input snapshot, model/policy/currency revisions, and full intent digest. Exact retries must retain that claim; changed intent must conflict. Use identifying native parameters for stable tenant/run/intent references. Do not add a random run incrementer to turn a conflicting retry into a new accepted run. + +Do not persist bearer tokens, caller security contexts, credentials, personal facts, or large business payloads in job parameters or execution contexts. Store small versioned references and cursors. The native metadata schema has no built-in tenant authority: an exposed execution ID must first resolve to its owning run and current caller permission. The application must supply current service authority before each new business effect, including after restart. + +For an external producer, durably establish a stable item action before I/O, call the genuine owner SDK with current service credentials, and retain its authoritative receipt. Unknown outcomes stay unresolved until the producer supplies an observed effect or an authoritative no-effect closure under the same immutable intent/deadline. A metadata checkpoint or empty lookup is not such a receipt. Reconciliation exceptions need an owner, reason, retained evidence, and permitted human resolution; a batch retry must not invent a financial adjustment. + +This module supplies execution infrastructure. It does not implement bank/provider reconciliation, a balanced general ledger, IFRS 9 classification/measurement/ECL/hedge accounting, IFRS 17 measurement/reinsurance, statutory reporting, or a complete banking/insurance journey. + +## Qualification + +Module tests use disposable PostgreSQL and synthetic item rows. Configuration/classpath/resource guards, native schema access, project-provider backoff, transactional rollback, checkpoint persistence, process death/restart, concurrent launch, executor admission, and an owner-run/current-authority recipe are separate gates. The focused module gate currently passes 56 tests with zero failures, errors, or skips. Its owner-run recipe is test-owned application code using an explicit synthetic deployment principal and an actual strict in-process PolicyDecisionPort. It proves full-intent replay checks, current policy before new fixture effects, and owner OPEN state despite native COMPLETED when stored acknowledgments are missing. It does not qualify a real external financial producer or a deployed identity provider. Whole-framework and independent final review remain pending. + +Run the module through the framework's normal Maven verification with the configured dependency repository. PostgreSQL/container absence is a failed infrastructure gate, not a skipped durability test. Full framework compatibility and independent review are required before adoption. Publication planning currently fails closed for the new unmapped native coordinate until a reviewed package home/preflight is supplied; historical package associations remain unchanged. + + +The maintained executable fixtures are [checkpoint rollback/restart](src/test/java/org/fireflyframework/batch/jdbc/JdbcCheckpointDatabaseTest.java), [packaged process lifecycle/concurrent admission](src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchProcessDatabaseIT.java), and [application-owned run/current-policy recipe](src/test/java/org/fireflyframework/batch/jdbc/OwnedRunBoundaryDatabaseTest.java). The latter uses only a local in-process PDP inside its synthetic JDBC writer; an application must keep remote policy or producer network I/O outside held database locks and implement its own durable effect protocol. Do not copy the fixture's trusted principal construction into an HTTP authentication boundary. diff --git a/modules/batch/pom.xml b/modules/batch/pom.xml new file mode 100644 index 000000000..6f702ed7a --- /dev/null +++ b/modules/batch/pom.xml @@ -0,0 +1,100 @@ + +org.fireflyframework +fireflyframework-batch +26.10.22 ++ diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableBatchAutoConfiguration.java b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableBatchAutoConfiguration.java new file mode 100644 index 000000000..da4829730 --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableBatchAutoConfiguration.java @@ -0,0 +1,131 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import org.springframework.beans.factory.BeanFactory; +import org.springframework.beans.factory.InitializingBean; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.beans.factory.SmartInitializingSingleton; +import org.springframework.beans.factory.config.BeanFactoryPostProcessor; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.DependsOn; +import org.springframework.context.annotation.Import; +import org.springframework.core.env.Environment; +import org.springframework.util.ClassUtils; + +/** Opt-in infrastructure for explicitly configured durable JDBC jobs. */ +@AutoConfiguration( + beforeName = { + "org.springframework.boot.batch.autoconfigure.BatchAutoConfiguration", + "org.springframework.boot.batch.autoconfigure.BatchJobLauncherAutoConfiguration" + }) +@ConditionalOnProperty(prefix = "firefly.batch.durable", name = "enabled", havingValue = "true") +@EnableConfigurationProperties(DurableBatchProperties.class) +@Import(DurableBatchAutoConfiguration.JdbcInfrastructure.class) +public class DurableBatchAutoConfiguration { + @Bean + static BeanFactoryPostProcessor durableBatchClasspathGuard() { + return beanFactory -> { + for (String required : + new String[] { + "org.springframework.batch.core.repository.JobRepository", + "org.springframework.batch.core.configuration.support.JdbcDefaultBatchConfiguration", + "org.springframework.jdbc.datasource.DataSourceTransactionManager", + "org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" + }) { + if (!ClassUtils.isPresent(required, beanFactory.getBeanClassLoader())) { + throw new IllegalStateException("Durable Batch requires " + required); + } + } + }; + } + + @Bean + InitializingBean durableBatchSettingsGuard( + DurableBatchProperties properties, Environment environment) { + return () -> { + if (!Boolean.FALSE.equals( + environment.getProperty("spring.batch.job.enabled", Boolean.class))) { + throw new IllegalStateException( + "Durable Batch requires explicit spring.batch.job.enabled=false"); + } + requireReference(properties.getDataSourceRef(), "data-source-ref"); + requireReference(properties.getTransactionManagerRef(), "transaction-manager-ref"); + requireReference(properties.getTaskExecutorRef(), "task-executor-ref"); + }; + } + + @Bean + @DependsOn("durableBatchSettingsGuard") + JdbcBatchResources durableJdbcBatchResources( + DurableBatchProperties properties, BeanFactory beans) { + return JdbcBatchResources.resolve(properties, beans); + } + + @Configuration(proxyBeanMethods = false) + @ConditionalOnClass( + name = + "org.springframework.batch.core.configuration.support.JdbcDefaultBatchConfiguration") + @Import(JdbcReadiness.class) + static class JdbcInfrastructure {} + + @Configuration(proxyBeanMethods = false) + static class JdbcReadiness { + @Bean + @org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean( + org.fireflyframework.batch.jdbc.JdbcBatchSteps.class) + org.fireflyframework.batch.jdbc.JdbcBatchSteps durableJdbcBatchSteps( + org.springframework.batch.core.repository.JobRepository repository, + JdbcBatchResources resources) { + return new org.fireflyframework.batch.jdbc.JdbcBatchSteps( + repository, resources.transactionManager()); + } + + @Bean + SmartInitializingSingleton durableBatchReadiness( + JdbcBatchResources resources, + ObjectProvider4.0.0 ++ +org.fireflyframework +fireflyframework-parent +26.10.22 +../../build/parent/pom.xml +fireflyframework-batch +Firefly Framework - Durable Batch +Opt-in native Spring Batch JDBC infrastructure + ++ + ++ +org.springframework.batch +spring-batch-core ++ +org.springframework +spring-jdbc ++ +org.springframework.boot +spring-boot-autoconfigure ++ +org.springframework.boot +spring-boot-starter-test +test ++ +org.springframework.boot +spring-boot-batch +test ++ +org.springframework +spring-r2dbc +test ++ +org.postgresql +postgresql +test ++ +org.testcontainers +testcontainers-postgresql +test ++ +org.testcontainers +testcontainers-junit-jupiter +test ++ +org.fireflyframework +fireflyframework-security-core +test ++ ++ ++ +org.apache.maven.plugins +maven-failsafe-plugin ++ ++ ++ integration-test verify + ++ +${project.build.directory}/${project.build.finalName}.jar ++ +org.apache.maven.plugins +maven-dependency-plugin +3.7.0 ++ ++ +runtime-test-classpath +generate-test-resources ++ build-classpath + +runtime +${project.build.directory}/runtime-classpath.txt ++ repositories, + ObjectProvider operators) { + return () -> { + var selected = repositories.stream().toList(); + if (selected.size() != 1 || operators.stream().count() != 1) { + throw new IllegalStateException( + "Durable Batch requires exactly one JobRepository and JobOperator"); + } + if (selected.getFirst() + instanceof + org.springframework.batch.core.repository.support + .ResourcelessJobRepository) { + throw new IllegalStateException( + "Durable Batch cannot use a ResourcelessJobRepository"); + } + JdbcBatchSchemaValidator.validate(resources.dataSource()); + }; + } + } + + private static void requireReference(String value, String property) { + if (value == null || value.isBlank()) { + throw new IllegalStateException("firefly.batch.durable." + property + " is required"); + } + } +} diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableBatchProperties.java b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableBatchProperties.java new file mode 100644 index 000000000..3f5bfc5a3 --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableBatchProperties.java @@ -0,0 +1,59 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import org.springframework.boot.context.properties.ConfigurationProperties; + +/** Names of application-owned resources; no database or executor is created implicitly. */ +@ConfigurationProperties("firefly.batch.durable") +public class DurableBatchProperties { + private boolean enabled; + private String dataSourceRef; + private String transactionManagerRef; + private String taskExecutorRef; + + public boolean isEnabled() { + return enabled; + } + + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } + + public String getDataSourceRef() { + return dataSourceRef; + } + + public void setDataSourceRef(String dataSourceRef) { + this.dataSourceRef = dataSourceRef; + } + + public String getTransactionManagerRef() { + return transactionManagerRef; + } + + public void setTransactionManagerRef(String transactionManagerRef) { + this.transactionManagerRef = transactionManagerRef; + } + + public String getTaskExecutorRef() { + return taskExecutorRef; + } + + public void setTaskExecutorRef(String taskExecutorRef) { + this.taskExecutorRef = taskExecutorRef; + } +} diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableJdbcBatchAutoConfiguration.java b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableJdbcBatchAutoConfiguration.java new file mode 100644 index 000000000..842923eda --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/DurableJdbcBatchAutoConfiguration.java @@ -0,0 +1,47 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.Import; + +/** Default providers are deferred until ordinary project configurations have been registered. */ +@AutoConfiguration( + after = DurableBatchAutoConfiguration.class, + beforeName = { + "org.springframework.boot.batch.autoconfigure.BatchAutoConfiguration", + "org.springframework.boot.batch.autoconfigure.BatchJobLauncherAutoConfiguration" + }) +@ConditionalOnClass( + name = "org.springframework.batch.core.configuration.support.JdbcDefaultBatchConfiguration") +@ConditionalOnProperty(prefix = "firefly.batch.durable", name = "enabled", havingValue = "true") +@Import(DurableJdbcBatchAutoConfiguration.DefaultProviders.class) +public class DurableJdbcBatchAutoConfiguration { + @Configuration(proxyBeanMethods = false) + @ConditionalOnMissingBean( + value = + org.springframework.batch.core.configuration.support.DefaultBatchConfiguration + .class, + annotation = + org.springframework.batch.core.configuration.annotation.EnableBatchProcessing + .class) + @Import(JdbcBatchConfiguration.class) + static class DefaultProviders {} +} diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchConfiguration.java b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchConfiguration.java new file mode 100644 index 000000000..dce6f7417 --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchConfiguration.java @@ -0,0 +1,66 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import org.springframework.batch.core.configuration.support.JdbcDefaultBatchConfiguration; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.core.task.TaskExecutor; +import org.springframework.transaction.PlatformTransactionManager; + +import javax.sql.DataSource; + +@Configuration(proxyBeanMethods = false) +class JdbcBatchConfiguration extends JdbcDefaultBatchConfiguration { + private final JdbcBatchResources resources; + + JdbcBatchConfiguration(JdbcBatchResources resources) { + this.resources = resources; + } + + @Bean + @Override + @ConditionalOnMissingBean(value = JobRepository.class, name = "jobRepository") + public JobRepository jobRepository() { + JdbcBatchSchemaValidator.validate(resources.dataSource()); + return super.jobRepository(); + } + + @Bean + @Override + @ConditionalOnMissingBean(value = JobOperator.class, name = "jobOperator") + public JobOperator jobOperator(JobRepository repository) { + return super.jobOperator(repository); + } + + @Override + protected DataSource getDataSource() { + return resources.dataSource(); + } + + @Override + protected PlatformTransactionManager getTransactionManager() { + return resources.transactionManager(); + } + + @Override + protected TaskExecutor getTaskExecutor() { + return resources.taskExecutor(); + } +} diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchResources.java b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchResources.java new file mode 100644 index 000000000..4fdf67fb4 --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchResources.java @@ -0,0 +1,95 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import org.springframework.beans.BeansException; +import org.springframework.beans.factory.BeanFactory; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; + +import java.util.concurrent.SynchronousQueue; +import java.util.concurrent.ThreadPoolExecutor; + +import javax.sql.DataSource; + +record JdbcBatchResources( + DataSource dataSource, + DataSourceTransactionManager transactionManager, + ThreadPoolTaskExecutor taskExecutor) { + static JdbcBatchResources resolve(DurableBatchProperties properties, BeanFactory beans) { + DataSource source = + selected(beans, properties.getDataSourceRef(), DataSource.class, "data-source-ref"); + DataSourceTransactionManager manager = + selected( + beans, + properties.getTransactionManagerRef(), + DataSourceTransactionManager.class, + "transaction-manager-ref"); + if (manager.getDataSource() != source) { + throw new IllegalStateException( + "Durable Batch manager must own the same DataSource selected by" + + " data-source-ref"); + } + ThreadPoolTaskExecutor executor = + selected( + beans, + properties.getTaskExecutorRef(), + ThreadPoolTaskExecutor.class, + "task-executor-ref"); + if (executor.getCorePoolSize() < 1 + || executor.getCorePoolSize() > 32 + || executor.getCorePoolSize() != executor.getMaxPoolSize()) { + throw new IllegalStateException( + "Durable Batch task-executor-ref requires core=max in 1..32"); + } + if (executor.getQueueCapacity() != 0) { + throw new IllegalStateException( + "Durable Batch task-executor-ref requires queueCapacity=0"); + } + ThreadPoolExecutor initialized = executor.getThreadPoolExecutor(); + if (initialized.getCorePoolSize() < 1 + || initialized.getCorePoolSize() > 32 + || initialized.getCorePoolSize() != initialized.getMaximumPoolSize()) { + throw new IllegalStateException( + "Durable Batch initialized executor requires core=max in 1..32"); + } + if (!(initialized.getQueue() instanceof SynchronousQueue)) { + throw new IllegalStateException( + "Durable Batch initialized executor requires a SynchronousQueue"); + } + if (!(initialized.getRejectedExecutionHandler() + instanceof ThreadPoolExecutor.AbortPolicy)) { + throw new IllegalStateException("Durable Batch task-executor-ref requires AbortPolicy"); + } + return new JdbcBatchResources(source, manager, executor); + } + + private static T selected(BeanFactory beans, String name, Class type, String property) { + try { + return beans.getBean(name, type); + } catch (BeansException error) { + throw new IllegalStateException( + "firefly.batch.durable." + + property + + " must select a " + + type.getSimpleName() + + " bean named '" + + name + + "'", + error); + } + } +} diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchSchemaValidator.java b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchSchemaValidator.java new file mode 100644 index 000000000..6b557c5fd --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/autoconfigure/JdbcBatchSchemaValidator.java @@ -0,0 +1,148 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Types; +import java.util.Map; +import java.util.Set; + +import javax.sql.DataSource; + +final class JdbcBatchSchemaValidator { + private static final Map COLUMNS = + Map.of( + "BATCH_JOB_INSTANCE", "JOB_INSTANCE_ID,VERSION,JOB_NAME,JOB_KEY", + "BATCH_JOB_EXECUTION", + "JOB_EXECUTION_ID,VERSION,JOB_INSTANCE_ID,CREATE_TIME,START_TIME,END_TIME,STATUS,EXIT_CODE,EXIT_MESSAGE,LAST_UPDATED", + "BATCH_JOB_EXECUTION_PARAMS", + "JOB_EXECUTION_ID,PARAMETER_NAME,PARAMETER_TYPE,PARAMETER_VALUE,IDENTIFYING", + "BATCH_STEP_EXECUTION", + "STEP_EXECUTION_ID,VERSION,STEP_NAME,JOB_EXECUTION_ID,CREATE_TIME,START_TIME,END_TIME,STATUS,COMMIT_COUNT,READ_COUNT,FILTER_COUNT,WRITE_COUNT,READ_SKIP_COUNT,WRITE_SKIP_COUNT,PROCESS_SKIP_COUNT,ROLLBACK_COUNT,EXIT_CODE,EXIT_MESSAGE,LAST_UPDATED", + "BATCH_STEP_EXECUTION_CONTEXT", + "STEP_EXECUTION_ID,SHORT_CONTEXT,SERIALIZED_CONTEXT", + "BATCH_JOB_EXECUTION_CONTEXT", + "JOB_EXECUTION_ID,SHORT_CONTEXT,SERIALIZED_CONTEXT"); + private static final Set NUMBERS = + Set.of( + "JOB_INSTANCE_ID", + "VERSION", + "JOB_EXECUTION_ID", + "STEP_EXECUTION_ID", + "COMMIT_COUNT", + "READ_COUNT", + "FILTER_COUNT", + "WRITE_COUNT", + "READ_SKIP_COUNT", + "WRITE_SKIP_COUNT", + "PROCESS_SKIP_COUNT", + "ROLLBACK_COUNT"); + private static final Set TIMES = + Set.of("CREATE_TIME", "START_TIME", "END_TIME", "LAST_UPDATED"); + private static final String[] SEQUENCES = { + "BATCH_JOB_INSTANCE_SEQ", "BATCH_JOB_EXECUTION_SEQ", "BATCH_STEP_EXECUTION_SEQ" + }; + + private JdbcBatchSchemaValidator() {} + + static void validate(DataSource source) { + try (Connection connection = source.getConnection()) { + if (!"PostgreSQL".equals(connection.getMetaData().getDatabaseProductName())) { + throw new IllegalStateException( + "Durable Batch currently requires PostgreSQL metadata"); + } + boolean wasReadOnly = connection.isReadOnly(); + try { + connection.setReadOnly(true); + for (var table : COLUMNS.entrySet()) { + try (var statement = connection.createStatement(); + var result = + statement.executeQuery( + "SELECT " + + table.getValue() + + " FROM " + + table.getKey() + + " WHERE 1=0")) { + var metadata = result.getMetaData(); + String[] columns = table.getValue().split(","); + for (int index = 0; index < columns.length; index++) { + String column = columns[index]; + int expected = + NUMBERS.contains(column) + ? Types.BIGINT + : TIMES.contains(column) + ? Types.TIMESTAMP + : column.equals("IDENTIFYING") + ? Types.CHAR + : Types.VARCHAR; + if (metadata.getColumnType(index + 1) != expected) { + throw new IllegalStateException( + "Durable Batch has incompatible native column: " + + table.getKey() + + "." + + column); + } + } + } + try (var statement = + connection.prepareStatement( + "SELECT has_table_privilege(current_user,CAST(? AS" + + " regclass),'SELECT') AND" + + " has_table_privilege(current_user,CAST(? AS" + + " regclass),'INSERT') AND" + + " has_table_privilege(current_user,CAST(? AS" + + " regclass),'UPDATE')")) { + for (int index = 1; index <= 3; index++) { + statement.setString( + index, table.getKey().toLowerCase(java.util.Locale.ROOT)); + } + try (var result = statement.executeQuery()) { + if (!result.next() || !result.getBoolean(1)) { + throw new IllegalStateException( + "Durable Batch lacks native table access: " + + table.getKey()); + } + } + } + } + for (String sequence : SEQUENCES) { + try (var statement = + connection.prepareStatement( + "SELECT c.relkind='S' AND" + + " has_sequence_privilege(current_user,c.oid,'USAGE') AND" + + " has_sequence_privilege(current_user,c.oid,'SELECT')" + + " FROM pg_catalog.pg_class c WHERE c.oid=CAST(? AS" + + " regclass)")) { + statement.setString(1, sequence.toLowerCase(java.util.Locale.ROOT)); + try (var result = statement.executeQuery()) { + if (!result.next() || !result.getBoolean(1)) { + throw new IllegalStateException( + "Durable Batch lacks native sequence access: " + sequence); + } + } + } + } + } finally { + connection.setReadOnly(wasReadOnly); + } + } catch (SQLException error) { + throw new IllegalStateException( + "Durable Batch requires the provisioned Spring Batch 6.0.5 PostgreSQL schema", + error); + } + } +} diff --git a/modules/batch/src/main/java/org/fireflyframework/batch/jdbc/JdbcBatchSteps.java b/modules/batch/src/main/java/org/fireflyframework/batch/jdbc/JdbcBatchSteps.java new file mode 100644 index 000000000..aec731a8d --- /dev/null +++ b/modules/batch/src/main/java/org/fireflyframework/batch/jdbc/JdbcBatchSteps.java @@ -0,0 +1,49 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.jdbc; + +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.core.step.builder.ChunkOrientedStepBuilder; +import org.springframework.batch.core.step.builder.StepBuilder; +import org.springframework.batch.core.step.builder.TaskletStepBuilder; +import org.springframework.batch.core.step.tasklet.Tasklet; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.util.Assert; + +/** Native JDBC step-builder entry point. */ +public final class JdbcBatchSteps { + private final JobRepository repository; + private final DataSourceTransactionManager manager; + + public JdbcBatchSteps(JobRepository repository, DataSourceTransactionManager manager) { + Assert.notNull(repository, "JobRepository is required"); + Assert.notNull(manager, "JDBC transaction manager is required"); + this.repository = repository; + this.manager = manager; + } + + public ChunkOrientedStepBuilder chunk(String name, int size) { + Assert.hasText(name, "Step name is required"); + Assert.isTrue(size > 0, "Chunk size must be positive"); + return new StepBuilder(name, repository).chunk(size).transactionManager(manager); + } + + public TaskletStepBuilder tasklet(String name, Tasklet tasklet) { + Assert.hasText(name, "Step name is required"); + Assert.notNull(tasklet, "Tasklet is required"); + return new StepBuilder(name, repository).tasklet(tasklet, manager); + } +} diff --git a/modules/batch/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports b/modules/batch/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports new file mode 100644 index 000000000..f07a9f3ba --- /dev/null +++ b/modules/batch/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports @@ -0,0 +1,2 @@ +org.fireflyframework.batch.autoconfigure.DurableBatchAutoConfiguration +org.fireflyframework.batch.autoconfigure.DurableJdbcBatchAutoConfiguration diff --git a/modules/batch/src/main/resources/org/fireflyframework/batch/jdbc/schema-postgresql-6.0.5.sql b/modules/batch/src/main/resources/org/fireflyframework/batch/jdbc/schema-postgresql-6.0.5.sql new file mode 100644 index 000000000..097a9d6c1 --- /dev/null +++ b/modules/batch/src/main/resources/org/fireflyframework/batch/jdbc/schema-postgresql-6.0.5.sql @@ -0,0 +1,78 @@ +-- Autogenerated: do not edit this file + +CREATE TABLE BATCH_JOB_INSTANCE ( + JOB_INSTANCE_ID BIGINT NOT NULL PRIMARY KEY, + VERSION BIGINT, + JOB_NAME VARCHAR(100) NOT NULL, + JOB_KEY VARCHAR(32) NOT NULL, + constraint JOB_INST_UN unique (JOB_NAME, JOB_KEY) +) ; + +CREATE TABLE BATCH_JOB_EXECUTION ( + JOB_EXECUTION_ID BIGINT NOT NULL PRIMARY KEY, + VERSION BIGINT, + JOB_INSTANCE_ID BIGINT NOT NULL, + CREATE_TIME TIMESTAMP NOT NULL, + START_TIME TIMESTAMP DEFAULT NULL, + END_TIME TIMESTAMP DEFAULT NULL, + STATUS VARCHAR(10), + EXIT_CODE VARCHAR(2500), + EXIT_MESSAGE VARCHAR(2500), + LAST_UPDATED TIMESTAMP, + constraint JOB_INST_EXEC_FK foreign key (JOB_INSTANCE_ID) + references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID) +) ; + +CREATE TABLE BATCH_JOB_EXECUTION_PARAMS ( + JOB_EXECUTION_ID BIGINT NOT NULL, + PARAMETER_NAME VARCHAR(100) NOT NULL, + PARAMETER_TYPE VARCHAR(100) NOT NULL, + PARAMETER_VALUE VARCHAR(2500), + IDENTIFYING CHAR(1) NOT NULL, + constraint JOB_EXEC_PARAMS_FK foreign key (JOB_EXECUTION_ID) + references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) +) ; + +CREATE TABLE BATCH_STEP_EXECUTION ( + STEP_EXECUTION_ID BIGINT NOT NULL PRIMARY KEY, + VERSION BIGINT NOT NULL, + STEP_NAME VARCHAR(100) NOT NULL, + JOB_EXECUTION_ID BIGINT NOT NULL, + CREATE_TIME TIMESTAMP NOT NULL, + START_TIME TIMESTAMP DEFAULT NULL, + END_TIME TIMESTAMP DEFAULT NULL, + STATUS VARCHAR(10), + COMMIT_COUNT BIGINT, + READ_COUNT BIGINT, + FILTER_COUNT BIGINT, + WRITE_COUNT BIGINT, + READ_SKIP_COUNT BIGINT, + WRITE_SKIP_COUNT BIGINT, + PROCESS_SKIP_COUNT BIGINT, + ROLLBACK_COUNT BIGINT, + EXIT_CODE VARCHAR(2500), + EXIT_MESSAGE VARCHAR(2500), + LAST_UPDATED TIMESTAMP, + constraint JOB_EXEC_STEP_FK foreign key (JOB_EXECUTION_ID) + references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) +) ; + +CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT ( + STEP_EXECUTION_ID BIGINT NOT NULL PRIMARY KEY, + SHORT_CONTEXT VARCHAR(2500) NOT NULL, + SERIALIZED_CONTEXT TEXT, + constraint STEP_EXEC_CTX_FK foreign key (STEP_EXECUTION_ID) + references BATCH_STEP_EXECUTION(STEP_EXECUTION_ID) +) ; + +CREATE TABLE BATCH_JOB_EXECUTION_CONTEXT ( + JOB_EXECUTION_ID BIGINT NOT NULL PRIMARY KEY, + SHORT_CONTEXT VARCHAR(2500) NOT NULL, + SERIALIZED_CONTEXT TEXT, + constraint JOB_EXEC_CTX_FK foreign key (JOB_EXECUTION_ID) + references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) +) ; + +CREATE SEQUENCE BATCH_STEP_EXECUTION_SEQ MAXVALUE 9223372036854775807 NO CYCLE; +CREATE SEQUENCE BATCH_JOB_EXECUTION_SEQ MAXVALUE 9223372036854775807 NO CYCLE; +CREATE SEQUENCE BATCH_JOB_INSTANCE_SEQ MAXVALUE 9223372036854775807 NO CYCLE; diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/DurableBatchAutoConfigurationTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/DurableBatchAutoConfigurationTest.java new file mode 100644 index 000000000..b2c37e8ee --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/DurableBatchAutoConfigurationTest.java @@ -0,0 +1,115 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.boot.autoconfigure.AutoConfigurations; +import org.springframework.boot.test.context.FilteredClassLoader; +import org.springframework.boot.test.context.runner.ApplicationContextRunner; + +class DurableBatchAutoConfigurationTest { + private final ApplicationContextRunner runner = + new ApplicationContextRunner() + .withConfiguration( + AutoConfigurations.of( + DurableBatchAutoConfiguration.class, + DurableJdbcBatchAutoConfiguration.class)); + + @Test + void absentModeCreatesNoBatchInfrastructure() { + runner.run( + context -> { + assertThat(context).hasNotFailed(); + assertThat(context) + .doesNotHaveBean(JobRepository.class) + .doesNotHaveBean(JobOperator.class); + assertThat(context).doesNotHaveBean(javax.sql.DataSource.class); + }); + } + + @Test + void falseModeIgnoresIncompleteDurableSettings() { + runner.withPropertyValues( + "firefly.batch.durable.enabled=false", + "firefly.batch.durable.data-source-ref=not-present") + .run( + context -> { + assertThat(context).hasNotFailed(); + assertThat(context) + .doesNotHaveBean(JobRepository.class) + .doesNotHaveBean(JobOperator.class); + }); + } + + @Test + void disabledModeToleratesMissingBatchClasses() { + runner.withClassLoader(new FilteredClassLoader("org.springframework.batch")) + .run(context -> assertThat(context).hasNotFailed()); + } + + @Test + void disabledModeToleratesMissingJdbcClasses() { + runner.withClassLoader(new FilteredClassLoader("org.springframework.jdbc")) + .run(context -> assertThat(context).hasNotFailed()); + } + + @Test + void explicitModeCannotSilentlyIgnoreMissingJdbcClasses() { + runner.withClassLoader(new FilteredClassLoader("org.springframework.jdbc")) + .withPropertyValues( + "firefly.batch.durable.enabled=true", "spring.batch.job.enabled=false") + .run(context -> assertThat(context).hasFailed()); + } + + @Test + void explicitModeCannotSilentlyIgnoreMissingBatchClasses() { + runner.withClassLoader(new FilteredClassLoader("org.springframework.batch")) + .withPropertyValues( + "firefly.batch.durable.enabled=true", "spring.batch.job.enabled=false") + .run(context -> assertThat(context).hasFailed()); + } + + @Test + void explicitModeRequiresDataSourceReference() { + runner.withPropertyValues( + "firefly.batch.durable.enabled=true", "spring.batch.job.enabled=false") + .run( + context -> { + assertThat(context).hasFailed(); + assertThat(context.getStartupFailure()) + .hasStackTraceContaining("data-source-ref"); + }); + } + + @Test + void explicitModeRequiresStartupRunnerToBeDisabled() { + runner.withPropertyValues( + "firefly.batch.durable.enabled=true", + "firefly.batch.durable.data-source-ref=ownedDataSource", + "firefly.batch.durable.transaction-manager-ref=ownedManager", + "firefly.batch.durable.task-executor-ref=ownedExecutor") + .run( + context -> { + assertThat(context).hasFailed(); + assertThat(context.getStartupFailure()) + .hasStackTraceContaining("spring.batch.job.enabled=false"); + }); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/DurableBatchBootCoexistenceTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/DurableBatchBootCoexistenceTest.java new file mode 100644 index 000000000..ccea2c7ce --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/DurableBatchBootCoexistenceTest.java @@ -0,0 +1,248 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.fireflyframework.batch.fixture.PostgresBatchFixture; +import org.fireflyframework.batch.fixture.project.ScannedNativeProviders; +import org.junit.jupiter.api.Test; +import org.springframework.batch.core.configuration.support.JdbcDefaultBatchConfiguration; +import org.springframework.batch.core.configuration.support.MapJobRegistry; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.launch.support.JobOperatorFactoryBean; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.core.repository.support.JdbcJobRepositoryFactoryBean; +import org.springframework.batch.core.repository.support.ResourcelessJobRepository; +import org.springframework.boot.autoconfigure.AutoConfigurations; +import org.springframework.boot.batch.autoconfigure.BatchAutoConfiguration; +import org.springframework.boot.batch.autoconfigure.BatchJobLauncherAutoConfiguration; +import org.springframework.boot.batch.autoconfigure.JobLauncherApplicationRunner; +import org.springframework.boot.test.context.runner.ApplicationContextRunner; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.ComponentScan; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.Import; +import org.springframework.core.task.TaskExecutor; +import org.springframework.jdbc.support.JdbcTransactionManager; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; +import org.springframework.transaction.PlatformTransactionManager; + +import javax.sql.DataSource; + +class DurableBatchBootCoexistenceTest extends PostgresBatchFixture { + private ApplicationContextRunner enabled(DataSource source) { + return new ApplicationContextRunner() + .withConfiguration( + AutoConfigurations.of( + DurableBatchAutoConfiguration.class, + DurableJdbcBatchAutoConfiguration.class, + BatchAutoConfiguration.class, + BatchJobLauncherAutoConfiguration.class)) + .withPropertyValues( + "firefly.batch.durable.enabled=true", + "spring.batch.job.enabled=false", + "firefly.batch.durable.data-source-ref=ownedDataSource", + "firefly.batch.durable.transaction-manager-ref=ownedManager", + "firefly.batch.durable.task-executor-ref=ownedExecutor") + .withBean("ownedDataSource", DataSource.class, () -> source) + .withBean( + "ownedManager", + JdbcTransactionManager.class, + () -> new JdbcTransactionManager(source)) + .withBean( + "ownedExecutor", + ThreadPoolTaskExecutor.class, + PostgresBatchFixture::executor); + } + + @Test + void realBootConfigurationCannotCreateResourceFreeRepositoryOrStartupRunner() throws Exception { + enabled(schema("complete")) + .run( + context -> { + assertThat(context) + .hasNotFailed() + .hasSingleBean(JobRepository.class) + .hasSingleBean(JobOperator.class); + assertThat(context.getBean(JobRepository.class)) + .isNotInstanceOf(ResourcelessJobRepository.class); + assertThat(context).doesNotHaveBean(JobLauncherApplicationRunner.class); + }); + } + + @Test + void ordinaryProjectBeansReplaceNativeDefaultsWithoutPrimary() throws Exception { + assertPlainProject(PlainProject.class); + } + + @Test + void plainProjectImportFirstStillReplacesDefaults() throws Exception { + assertPlainProject(ProjectFirst.class); + } + + @Test + void plainProjectImportLastStillReplacesDefaults() throws Exception { + assertPlainProject(ProjectLast.class); + } + + @Test + void actualProjectComponentScanReplacesProvidersWithoutPrimary() throws Exception { + assertPlainProject(ScannedProject.class); + } + + private void assertPlainProject(Class> project) throws Exception { + DataSource source = schema("complete"); + enabled(source) + .withUserConfiguration(project) + .run( + context -> { + assertThat(context) + .hasNotFailed() + .hasSingleBean(JobRepository.class) + .hasSingleBean(JobOperator.class); + assertThat(context.getBean(JobRepository.class)) + .isSameAs(context.getBean("projectRepository")); + assertThat(context.getBean(JobOperator.class)) + .isSameAs(context.getBean("projectOperator")); + context.getBean(JobRepository.class) + .createJobInstance( + "project-owned-job", + new JobParametersBuilder() + .addString("tenant", "owned", true) + .toJobParameters()); + }); + assertThat( + new org.springframework.jdbc.core.JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isEqualTo(1); + } + + @Test + void projectJdbcConfigurationBacksOffDefaultGroup() throws Exception { + enabled(schema("complete")) + .withUserConfiguration(ProjectJdbc.class) + .run( + context -> { + assertThat(context) + .hasNotFailed() + .hasSingleBean(JobRepository.class) + .hasSingleBean(JobOperator.class); + assertThat(context.getBeansOfType(JdbcDefaultBatchConfiguration.class)) + .hasSize(1); + assertThat(context.getBean(JdbcDefaultBatchConfiguration.class)) + .isInstanceOf(ProjectJdbc.class); + context.getBean(JobRepository.class) + .createJobInstance( + "project-config-job", + new JobParametersBuilder().toJobParameters()); + }); + } + + @Test + void enabledProjectResourceFreeRepositoryIsRejected() throws Exception { + enabled(schema("complete")) + .withUserConfiguration(ResourceFreeProject.class) + .run(context -> assertThat(context).hasFailed()); + } + + @Test + void disabledModuleLeavesProjectResourceFreeLegacyConfigurationAlone() { + new ApplicationContextRunner() + .withConfiguration( + AutoConfigurations.of( + DurableBatchAutoConfiguration.class, + DurableJdbcBatchAutoConfiguration.class)) + .withUserConfiguration(ResourceFreeProject.class) + .run( + context -> { + assertThat(context).hasNotFailed().hasSingleBean(JobRepository.class); + assertThat(context.getBean(JobRepository.class)) + .isInstanceOf(ResourcelessJobRepository.class); + }); + } + + @Configuration(proxyBeanMethods = false) + @ComponentScan(basePackageClasses = ScannedNativeProviders.class) + static class ScannedProject {} + + @Configuration(proxyBeanMethods = false) + static class PlainProject { + @Bean + JobRepository projectRepository(JdbcBatchResources resources) throws Exception { + var factory = new JdbcJobRepositoryFactoryBean(); + factory.setDataSource(resources.dataSource()); + factory.setTransactionManager(resources.transactionManager()); + factory.setDatabaseType("POSTGRES"); + factory.afterPropertiesSet(); + return factory.getObject(); + } + + @Bean + JobOperator projectOperator(JobRepository repository, JdbcBatchResources resources) + throws Exception { + var factory = new JobOperatorFactoryBean(); + factory.setJobRepository(repository); + factory.setJobRegistry(new MapJobRegistry()); + factory.setTransactionManager(resources.transactionManager()); + factory.setTaskExecutor(resources.taskExecutor()); + factory.afterPropertiesSet(); + return factory.getObject(); + } + } + + @Configuration(proxyBeanMethods = false) + @Import({PlainProject.class, DurableBatchAutoConfiguration.class}) + static class ProjectFirst {} + + @Configuration(proxyBeanMethods = false) + @Import({DurableBatchAutoConfiguration.class, PlainProject.class}) + static class ProjectLast {} + + @Configuration(proxyBeanMethods = false) + static class ProjectJdbc extends JdbcDefaultBatchConfiguration { + private final JdbcBatchResources resources; + + ProjectJdbc(JdbcBatchResources resources) { + this.resources = resources; + } + + @Override + protected DataSource getDataSource() { + return resources.dataSource(); + } + + @Override + protected PlatformTransactionManager getTransactionManager() { + return resources.transactionManager(); + } + + @Override + protected TaskExecutor getTaskExecutor() { + return resources.taskExecutor(); + } + } + + @Configuration(proxyBeanMethods = false) + static class ResourceFreeProject { + @Bean + JobRepository projectRepository() { + return new ResourcelessJobRepository(); + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/JdbcBatchResourceSelectionTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/JdbcBatchResourceSelectionTest.java new file mode 100644 index 000000000..93af8ed03 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/JdbcBatchResourceSelectionTest.java @@ -0,0 +1,273 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.Mockito.mock; + +import io.r2dbc.spi.ConnectionFactory; + +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.support.DefaultListableBeanFactory; +import org.springframework.boot.autoconfigure.AutoConfigurations; +import org.springframework.boot.test.context.runner.ApplicationContextRunner; +import org.springframework.core.task.SyncTaskExecutor; +import org.springframework.jdbc.datasource.AbstractDataSource; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.r2dbc.connection.R2dbcTransactionManager; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; + +import java.sql.Connection; +import java.util.concurrent.SynchronousQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.atomic.AtomicInteger; + +import javax.sql.DataSource; + +class JdbcBatchResourceSelectionTest { + private final AtomicInteger connections = new AtomicInteger(); + private final DataSource dataSource = + new AbstractDataSource() { + @Override + public Connection getConnection() { + connections.incrementAndGet(); + throw new AssertionError("Invalid resources must fail before JDBC acquisition"); + } + + @Override + public Connection getConnection(String username, String password) { + return getConnection(); + } + }; + + private ApplicationContextRunner enabled() { + return new ApplicationContextRunner() + .withConfiguration( + AutoConfigurations.of( + DurableBatchAutoConfiguration.class, + DurableJdbcBatchAutoConfiguration.class)) + .withPropertyValues( + "firefly.batch.durable.enabled=true", + "spring.batch.job.enabled=false", + "firefly.batch.durable.data-source-ref=ownedDataSource", + "firefly.batch.durable.transaction-manager-ref=ownedManager", + "firefly.batch.durable.task-executor-ref=ownedExecutor"); + } + + private ThreadPoolTaskExecutor executor() { + var executor = new ThreadPoolTaskExecutor(); + executor.setCorePoolSize(2); + executor.setMaxPoolSize(2); + executor.setQueueCapacity(0); + return executor; + } + + private ApplicationContextRunner resources(Object source, Object manager, Object executor) { + return enabled() + .withBean("ownedDataSource", Object.class, () -> source) + .withBean("ownedManager", Object.class, () -> manager) + .withBean("ownedExecutor", Object.class, () -> executor); + } + + private void rejects(ApplicationContextRunner runner, String reason) { + runner.run( + context -> { + assertThat(context).hasFailed(); + assertThat(context.getStartupFailure()).hasStackTraceContaining(reason); + assertThat(connections).hasValue(0); + }); + } + + @Test + void namedResourcesMustActuallyExist() { + rejects(enabled(), "data-source-ref"); + } + + @Test + void selectedDataSourceMustBeJdbc() { + rejects( + resources( + "not-a-datasource", + new DataSourceTransactionManager(dataSource), + executor()), + "data-source-ref"); + } + + @Test + void selectedManagerMustNotBeReactive() { + rejects( + resources( + dataSource, + new R2dbcTransactionManager(mock(ConnectionFactory.class)), + executor()), + "transaction-manager-ref"); + } + + @Test + void managerMustOwnTheExactSelectedDataSource() { + DataSource other = mock(DataSource.class); + rejects( + resources(dataSource, new DataSourceTransactionManager(other), executor()), + "same DataSource"); + } + + @Test + void executorCannotRunJobsSynchronouslyOnCaller() { + rejects( + resources( + dataSource, + new DataSourceTransactionManager(dataSource), + new SyncTaskExecutor()), + "task-executor-ref"); + } + + @Test + void executorCannotHaveAnUnboundedMaximum() { + var executor = executor(); + executor.setMaxPoolSize(Integer.MAX_VALUE); + rejects( + resources(dataSource, new DataSourceTransactionManager(dataSource), executor), + "core=max"); + } + + @Test + void executorCannotQueueUnboundedWork() { + var executor = executor(); + executor.setQueueCapacity(Integer.MAX_VALUE); + rejects( + resources(dataSource, new DataSourceTransactionManager(dataSource), executor), + "queueCapacity=0"); + } + + @Test + void callerRunsRejectionIsNotAllowed() { + var executor = executor(); + executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy()); + rejects( + resources(dataSource, new DataSourceTransactionManager(dataSource), executor), + "AbortPolicy"); + } + + private JdbcBatchResources selectedInitializedExecutor(ThreadPoolTaskExecutor executor) { + var beans = new DefaultListableBeanFactory(); + beans.registerSingleton("ownedDataSource", dataSource); + beans.registerSingleton("ownedManager", new DataSourceTransactionManager(dataSource)); + beans.registerSingleton("ownedExecutor", executor); + var properties = new DurableBatchProperties(); + properties.setDataSourceRef("ownedDataSource"); + properties.setTransactionManagerRef("ownedManager"); + properties.setTaskExecutorRef("ownedExecutor"); + return JdbcBatchResources.resolve(properties, beans); + } + + private void rejectsInitializedExecutor(ThreadPoolTaskExecutor executor, String reason) { + try { + assertThatThrownBy(() -> selectedInitializedExecutor(executor)) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining(reason); + assertThat(connections).hasValue(0); + } finally { + executor.shutdown(); + } + } + + @Test + void changedCapacityCannotHideInitializedUnboundedQueue() { + var executor = executor(); + executor.setQueueCapacity(Integer.MAX_VALUE); + executor.initialize(); + executor.setQueueCapacity(0); + assertThat(executor.getQueueCapacity()).isZero(); + assertThat(executor.getThreadPoolExecutor().getQueue().remainingCapacity()) + .isEqualTo(Integer.MAX_VALUE); + rejectsInitializedExecutor(executor, "SynchronousQueue"); + } + + @Test + void fullBoundedQueueCannotPassAsZeroCapacityHandoff() { + var executor = executor(); + executor.setQueueCapacity(1); + executor.initialize(); + executor.getThreadPoolExecutor().getQueue().add(() -> {}); + executor.setQueueCapacity(0); + assertThat(executor.getThreadPoolExecutor().getQueue().remainingCapacity()).isZero(); + rejectsInitializedExecutor(executor, "SynchronousQueue"); + } + + @Test + void backingMaximumCannotDifferFromFixedConfiguredLimit() { + var executor = executor(); + executor.initialize(); + executor.getThreadPoolExecutor().setMaximumPoolSize(Integer.MAX_VALUE); + assertThat(executor.getMaxPoolSize()).isEqualTo(2); + rejectsInitializedExecutor(executor, "core=max"); + } + + @Test + void backingCoreCannotBeZeroDespiteConfiguredLimit() { + var executor = executor(); + executor.initialize(); + executor.getThreadPoolExecutor().setCorePoolSize(0); + assertThat(executor.getCorePoolSize()).isEqualTo(2); + rejectsInitializedExecutor(executor, "core=max"); + } + + @Test + void backingFixedPoolCannotExceedThirtyTwoDespiteConfiguredLimit() { + var executor = executor(); + executor.initialize(); + executor.getThreadPoolExecutor().setMaximumPoolSize(33); + executor.getThreadPoolExecutor().setCorePoolSize(33); + assertThat(executor.getCorePoolSize()).isEqualTo(2); + assertThat(executor.getMaxPoolSize()).isEqualTo(2); + rejectsInitializedExecutor(executor, "core=max"); + } + + @Test + void backingCallerRunsPolicyIsRejected() { + var executor = executor(); + executor.initialize(); + executor.getThreadPoolExecutor() + .setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy()); + rejectsInitializedExecutor(executor, "AbortPolicy"); + } + + @Test + void actualSingleWorkerSynchronousExecutorIsAccepted() { + acceptsFixedSynchronousExecutor(1); + } + + @Test + void actualThirtyTwoWorkerSynchronousExecutorIsAccepted() { + acceptsFixedSynchronousExecutor(32); + } + + private void acceptsFixedSynchronousExecutor(int size) { + var executor = executor(); + executor.setCorePoolSize(size); + executor.setMaxPoolSize(size); + executor.initialize(); + try { + assertThat(executor.getThreadPoolExecutor().getQueue()) + .isInstanceOf(SynchronousQueue.class); + assertThat(selectedInitializedExecutor(executor).taskExecutor()).isSameAs(executor); + assertThat(connections).hasValue(0); + } finally { + executor.shutdown(); + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/JdbcBatchSchemaTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/JdbcBatchSchemaTest.java new file mode 100644 index 000000000..e306e8fcd --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/JdbcBatchSchemaTest.java @@ -0,0 +1,186 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.fireflyframework.batch.fixture.PostgresBatchFixture; +import org.junit.jupiter.api.Test; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.core.repository.support.ResourcelessJobRepository; +import org.springframework.boot.autoconfigure.AutoConfigurations; +import org.springframework.boot.test.context.runner.ApplicationContextRunner; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.support.JdbcTransactionManager; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; + +import javax.sql.DataSource; + +class JdbcBatchSchemaTest extends PostgresBatchFixture { + private ApplicationContextRunner configured(DataSource source) { + return new ApplicationContextRunner() + .withConfiguration( + AutoConfigurations.of( + DurableBatchAutoConfiguration.class, + DurableJdbcBatchAutoConfiguration.class)) + .withPropertyValues( + "firefly.batch.durable.enabled=true", + "spring.batch.job.enabled=false", + "firefly.batch.durable.data-source-ref=ownedDataSource", + "firefly.batch.durable.transaction-manager-ref=ownedManager", + "firefly.batch.durable.task-executor-ref=ownedExecutor") + .withBean("ownedDataSource", DataSource.class, () -> source) + .withBean( + "ownedManager", + JdbcTransactionManager.class, + () -> new JdbcTransactionManager(source)) + .withBean( + "ownedExecutor", + ThreadPoolTaskExecutor.class, + PostgresBatchFixture::executor); + } + + @Test + void genuineNativeRepositoryPersistsWithoutRuntimeSchemaOwnership() throws Exception { + DataSource source = schema("complete"); + configured(source) + .run( + context -> { + assertThat(context) + .hasNotFailed() + .hasSingleBean(JobRepository.class) + .hasSingleBean(JobOperator.class); + JobRepository repository = context.getBean(JobRepository.class); + assertThat(repository).isNotInstanceOf(ResourcelessJobRepository.class); + var parameters = + new JobParametersBuilder() + .addString("tenant", "owned-tenant", true) + .toJobParameters(); + var instance = repository.createJobInstance("owned-job", parameters); + assertThat(instance.getId()).isPositive(); + assertThat(repository.getJobInstance("owned-job", parameters).getId()) + .isEqualTo(instance.getId()); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT" + + " has_schema_privilege(current_user,current_schema(),'CREATE')", + Boolean.class)) + .isFalse(); + }); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE WHERE" + + " JOB_NAME='owned-job'", + Integer.class)) + .isEqualTo(1); + } + + @Test + void enabledNativeContextSuppliesJdbcBoundStepBuilderHelper() throws Exception { + configured(schema("complete")) + .run( + context -> { + assertThat(context) + .hasNotFailed() + .hasSingleBean( + org.fireflyframework.batch.jdbc.JdbcBatchSteps.class); + }); + } + + @Test + void missingSchemaCannotBeAcceptedAsDurable() throws Exception { + configured(schema("empty")).run(context -> assertThat(context).hasFailed()); + } + + @Test + void missingNativeColumnFailsBeforeJobCreation() throws Exception { + DataSource source = schema("missing-column"); + configured(source).run(context -> assertThat(context).hasFailed()); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isZero(); + } + + @Test + void incompatibleNativeColumnTypeFailsBeforeJobCreation() throws Exception { + DataSource source = schema("wrong-column-type"); + configured(source).run(context -> assertThat(context).hasFailed()); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isZero(); + } + + @Test + void readOnlyMetadataGrantCannotBeAcceptedAsDurableWriter() throws Exception { + configured(schema("read-only")).run(context -> assertThat(context).hasFailed()); + } + + @Test + void missingNativeSequenceFailsBeforeJobCreation() throws Exception { + DataSource source = schema("missing-sequence"); + configured(source).run(context -> assertThat(context).hasFailed()); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isZero(); + } + + @Test + void inaccessibleSchemaFailsReadiness() throws Exception { + configured(schema("no-access")).run(context -> assertThat(context).hasFailed()); + } + + @Test + void explicitNamesIgnoreAnotherPrimaryResource() throws Exception { + DataSource selected = schema("complete"); + DataSource other = schema("empty"); + configured(selected) + .withBean( + "anotherDataSource", + DataSource.class, + () -> other, + definition -> definition.setPrimary(true)) + .withBean( + "anotherManager", + JdbcTransactionManager.class, + () -> new JdbcTransactionManager(other), + definition -> definition.setPrimary(true)) + .run( + context -> { + assertThat(context).hasNotFailed().hasSingleBean(JobRepository.class); + context.getBean(JobRepository.class) + .createJobInstance( + "explicit-selected-job", + new JobParametersBuilder() + .addString("run", "owned-run", true) + .toJobParameters()); + }); + assertThat( + new JdbcTemplate(selected) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isEqualTo(1); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/MinimalBootRuntimeTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/MinimalBootRuntimeTest.java new file mode 100644 index 000000000..0cb6444c8 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/MinimalBootRuntimeTest.java @@ -0,0 +1,67 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.concurrent.TimeUnit; + +class MinimalBootRuntimeTest { + @TempDir Path temporary; + + @Test + void actualDefaultBootRuntimeHasNoDatabaseOrJobInfrastructure() throws Exception { + String dependencies = Files.readString(Path.of("target/runtime-classpath.txt")).trim(); + assertThat(dependencies) + .doesNotContain("spring-boot-batch-") + .doesNotContain("spring-boot-jdbc-") + .doesNotContain("spring-boot-starter-batch-"); + String classpath = + Path.of("target/test-classes").toAbsolutePath() + + File.pathSeparator + + Path.of("target/classes").toAbsolutePath() + + File.pathSeparator + + dependencies; + Path log = temporary.resolve("minimal-boot.log"); + Process process = + new ProcessBuilder( + Path.of(System.getProperty("java.home"), "bin", "java").toString(), + "-cp", + classpath, + "org.fireflyframework.batch.fixture.MinimalBootMain") + .redirectErrorStream(true) + .redirectOutput(log.toFile()) + .start(); + try { + assertThat(process.waitFor(Duration.ofSeconds(25).toMillis(), TimeUnit.MILLISECONDS)) + .isTrue(); + assertThat(process.exitValue()).withFailMessage(Files.readString(log)).isZero(); + assertThat(Files.readString(log)).contains("OWNED_MINIMAL_BOOT_NO_RESOURCES=true"); + } finally { + if (process.isAlive()) { + process.destroyForcibly(); + process.waitFor(10, TimeUnit.SECONDS); + } + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/MinimalPackagedBootIT.java b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/MinimalPackagedBootIT.java new file mode 100644 index 000000000..b971c0887 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/autoconfigure/MinimalPackagedBootIT.java @@ -0,0 +1,119 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.autoconfigure; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.fireflyframework.batch.fixture.MinimalBootMain; +import org.junit.jupiter.api.Test; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; + +class MinimalPackagedBootIT { + private void run(boolean successful, String... arguments) throws Exception { + Path parent = Path.of("target/durable-process-evidence"); + Files.createDirectories(parent); + Path evidence = Files.createTempDirectory(parent, "minimal-").toAbsolutePath(); + Path bootstrap = evidence.resolve("bootstrap"); + for (String name : + List.of( + MinimalBootMain.class.getName(), + MinimalBootMain.class.getName() + "$Application")) { + String resource = name.replace('.', '/') + ".class"; + Path target = bootstrap.resolve(resource); + Files.createDirectories(target.getParent()); + try (var input = MinimalBootMain.class.getClassLoader().getResourceAsStream(resource)) { + if (input == null) throw new IllegalStateException("Owned bootstrap missing"); + Files.write(target, input.readAllBytes()); + } + } + Path artifact = Path.of(System.getProperty("durable.batch.jar")); + assertThat(artifact).isRegularFile(); + String dependencies = Files.readString(Path.of("target/runtime-classpath.txt")).trim(); + assertThat(dependencies) + .doesNotContain( + "spring-boot-batch-", + "spring-boot-jdbc-", + "testcontainers", + "junit", + "target/classes"); + String classpath = + bootstrap + + java.io.File.pathSeparator + + artifact + + java.io.File.pathSeparator + + dependencies; + var command = + new ArrayList<>( + List.of( + Path.of(System.getProperty("java.home"), "bin/java").toString(), + "-Xms16m", + "-Xmx192m", + "-cp", + classpath, + MinimalBootMain.class.getName())); + command.addAll(List.of(arguments)); + Path log = evidence.resolve("java.log"); + var child = + new ProcessBuilder(command) + .redirectErrorStream(true) + .redirectOutput(log.toFile()) + .start(); + try { + assertThat(child.waitFor(25, TimeUnit.SECONDS)).isTrue(); + String output = Files.readString(log); + if (successful) { + assertThat(child.exitValue()).withFailMessage(output).isZero(); + assertThat(output) + .contains( + "OWNED_MINIMAL_BOOT_NO_RESOURCES=true", + "OWNED_MODULE_ORIGIN=" + artifact.toUri().toURL()); + } else { + assertThat(child.exitValue()).withFailMessage(output).isNotZero(); + assertThat(output).contains("data-source-ref"); + } + } finally { + if (child.isAlive()) child.destroyForcibly(); + assertThat(child.waitFor(10, TimeUnit.SECONDS)).isTrue(); + Files.writeString( + evidence.resolve("cleanup.txt"), + "pid=" + child.pid() + "\nalive=" + child.isAlive()); + assertThat(child.isAlive()).isFalse(); + } + } + + @Test + void absentModeHasNoDatabaseEvenWithPackagedModulePresent() throws Exception { + run(true); + } + + @Test + void explicitFalseModeIgnoresIncompleteResourceNames() throws Exception { + run( + true, + "--firefly.batch.durable.enabled=false", + "--firefly.batch.durable.data-source-ref=not-present"); + } + + @Test + void explicitTrueModeRejectsMissingResourcesInActualPackagedBoot() throws Exception { + run(false, "--firefly.batch.durable.enabled=true", "--spring.batch.job.enabled=false"); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/BatchProcessMain.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/BatchProcessMain.java new file mode 100644 index 000000000..53a84e480 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/BatchProcessMain.java @@ -0,0 +1,131 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture; + +import org.postgresql.ds.PGSimpleDataSource; +import org.springframework.batch.core.BatchStatus; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; + +/** Test-owned native consumer; its runtime classpath has no test framework or containers. */ +public final class BatchProcessMain { + public static void main(String[] args) throws Exception { + var source = new PGSimpleDataSource(); + source.setURL(required("OWNED_BATCH_URL")); + source.setUser(required("OWNED_BATCH_USER")); + source.setPassword(required("OWNED_BATCH_PASSWORD")); + source.setCurrentSchema(required("OWNED_BATCH_SCHEMA")); + System.out.println( + "OWNED_MODULE_ORIGIN=" + + org.fireflyframework.batch.jdbc.JdbcBatchSteps.class + .getProtectionDomain() + .getCodeSource() + .getLocation()); + String mode = args[0]; + String run = required("OWNED_BATCH_RUN"); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var operator = context.getBean(JobOperator.class); + var job = + CheckpointJobFixture.job( + context, + source, + run, + false, + false, + () -> { + String release = System.getenv("OWNED_BATCH_RELEASE_FILE"); + if (release != null) { + System.out.println("OWNED_HOLD_BEFORE_FOUR=true"); + System.out.flush(); + long deadline = + System.nanoTime() + Duration.ofSeconds(60).toNanos(); + while (!Files.exists(Path.of(release))) { + if (System.nanoTime() > deadline) + throw new IllegalStateException( + "Owned barrier timed out"); + try { + Thread.sleep(20); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new IllegalStateException( + "Owned barrier interrupted", interrupted); + } + } + } + }); + if (mode.equals("stop")) { + var execution = repository.getJobExecution(Long.parseLong(args[1])); + System.out.println("OWNED_STOP_ACK=" + operator.stop(execution)); + return; + } + org.springframework.batch.core.job.JobExecution started; + if (mode.equals("recover-restart") || mode.equals("restart")) { + var original = repository.getJobExecution(Long.parseLong(args[1])); + if (mode.equals("recover-restart")) { + operator.recover(original); + var observed = repository.getJobExecution(original.getId()); + if (observed.getStatus() != BatchStatus.FAILED) + throw new AssertionError("Recovery was not persisted"); + System.out.println("OWNED_RECOVERED_STORED_STATUS=" + observed.getStatus()); + original = observed; + } + started = operator.restart(original); + } else if (mode.equals("start")) { + String startFile = System.getenv("OWNED_BATCH_START_FILE"); + System.out.println("OWNED_READY_TO_START=true"); + System.out.flush(); + if (startFile != null) { + long deadline = System.nanoTime() + Duration.ofSeconds(30).toNanos(); + while (!Files.exists(Path.of(startFile))) { + if (System.nanoTime() > deadline) + throw new IllegalStateException("Owned launch barrier timed out"); + Thread.sleep(20); + } + } + started = + operator.start( + job, + new JobParametersBuilder() + .addString("tenant", required("OWNED_BATCH_TENANT"), true) + .addString("run", run, true) + .toJobParameters()); + } else { + throw new IllegalArgumentException("Unknown owned process mode"); + } + System.out.println("OWNED_EXECUTION_ID=" + started.getId()); + System.out.flush(); + var completed = CheckpointJobFixture.await(repository, started.getId()); + System.out.println("OWNED_FINAL_STATUS=" + completed.getStatus()); + if (completed.getStatus() != BatchStatus.COMPLETED + && completed.getStatus() != BatchStatus.STOPPED) { + throw new AssertionError("Owned execution failed: " + completed.getStatus()); + } + } + } + + private static String required(String name) { + String value = System.getenv(name); + if (value == null || value.isBlank()) + throw new IllegalStateException("Missing owned fixture setting: " + name); + return value; + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/CheckpointJobFixture.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/CheckpointJobFixture.java new file mode 100644 index 000000000..520b5511a --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/CheckpointJobFixture.java @@ -0,0 +1,176 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture; + +import org.fireflyframework.batch.autoconfigure.DurableBatchAutoConfiguration; +import org.fireflyframework.batch.autoconfigure.DurableJdbcBatchAutoConfiguration; +import org.fireflyframework.batch.jdbc.JdbcBatchSteps; +import org.springframework.batch.core.configuration.support.MapJobRegistry; +import org.springframework.batch.core.job.Job; +import org.springframework.batch.core.job.JobExecution; +import org.springframework.batch.core.job.builder.JobBuilder; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.core.step.builder.StepBuilder; +import org.springframework.batch.infrastructure.item.ExecutionContext; +import org.springframework.batch.infrastructure.item.ItemStreamReader; +import org.springframework.context.annotation.AnnotationConfigApplicationContext; +import org.springframework.core.env.MapPropertySource; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.support.JdbcTransactionManager; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; + +import java.time.Duration; + +import javax.sql.DataSource; + +public final class CheckpointJobFixture { + private CheckpointJobFixture() {} + + public static AnnotationConfigApplicationContext open(DataSource source) { + var context = new AnnotationConfigApplicationContext(); + context.getEnvironment() + .getPropertySources() + .addFirst( + new MapPropertySource( + "owned-batch", + java.util.Map.of( + "firefly.batch.durable.enabled", + "true", + "spring.batch.job.enabled", + "false", + "firefly.batch.durable.data-source-ref", + "ownedDataSource", + "firefly.batch.durable.transaction-manager-ref", + "ownedManager", + "firefly.batch.durable.task-executor-ref", + "ownedExecutor"))); + context.registerBean("ownedDataSource", DataSource.class, () -> source); + context.registerBean( + "ownedManager", + JdbcTransactionManager.class, + () -> new JdbcTransactionManager(source)); + context.registerBean( + "ownedExecutor", + ThreadPoolTaskExecutor.class, + () -> { + var executor = new ThreadPoolTaskExecutor(); + executor.setCorePoolSize(2); + executor.setMaxPoolSize(2); + executor.setQueueCapacity(0); + executor.setThreadNamePrefix("owned-batch-"); + return executor; + }); + context.registerBean("ownedRegistry", MapJobRegistry.class, MapJobRegistry::new); + context.register( + DurableBatchAutoConfiguration.class, DurableJdbcBatchAutoConfiguration.class); + try { + context.refresh(); + return context; + } catch (RuntimeException | Error failure) { + context.close(); + throw failure; + } + } + + public static Job job( + AnnotationConfigApplicationContext context, + DataSource source, + String run, + boolean failAtFive, + boolean omittedManagerControl) + throws Exception { + return job(context, source, run, failAtFive, omittedManagerControl, () -> {}); + } + + public static Job job( + AnnotationConfigApplicationContext context, + DataSource source, + String run, + boolean failAtFive, + boolean omittedManagerControl, + Runnable beforeFour) + throws Exception { + var repository = context.getBean(JobRepository.class); + var reader = new CursorReader(beforeFour); + var helper = context.getBean(JdbcBatchSteps.class); + var builder = + omittedManagerControl + ? new StepBuilder("owned-chunk", repository). chunk(3) + : helper. chunk("owned-chunk", 3); + var jdbc = new JdbcTemplate(source); + var step = + builder.reader(reader) + .writer( + items -> { + for (Integer item : items) { + jdbc.update( + "INSERT INTO owned_items(run_id,item_no) VALUES" + + " (?,?)", + run, + item); + if (failAtFive && item == 5) + throw new IllegalStateException( + "Owned fixture writer failure at five"); + } + }) + .build(); + Job job = new JobBuilder("owned-checkpoint-job", repository).start(step).build(); + context.getBean(MapJobRegistry.class).register(job); + return job; + } + + public static JobExecution await(JobRepository repository, long executionId) + throws InterruptedException { + long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos(); + while (System.nanoTime() < deadline) { + JobExecution current = repository.getJobExecution(executionId); + if (current != null && !current.isRunning() && current.getEndTime() != null) + return current; + Thread.sleep(20); + } + throw new AssertionError( + "Owned execution did not reach a stored terminal state: " + executionId); + } + + public static final class CursorReader implements ItemStreamReader { + private int cursor; + private final Runnable beforeFour; + + public CursorReader() { + this(() -> {}); + } + + public CursorReader(Runnable beforeFour) { + this.beforeFour = beforeFour; + } + + @Override + public void open(ExecutionContext context) { + cursor = context.getInt("owned.cursor", 0); + } + + @Override + public Integer read() { + if (cursor == 3) beforeFour.run(); + return cursor >= 10 ? null : ++cursor; + } + + @Override + public void update(ExecutionContext context) { + context.putInt("owned.cursor", cursor); + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/MinimalBootMain.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/MinimalBootMain.java new file mode 100644 index 000000000..44efce46e --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/MinimalBootMain.java @@ -0,0 +1,56 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture; + +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.boot.SpringBootConfiguration; +import org.springframework.boot.WebApplicationType; +import org.springframework.boot.autoconfigure.EnableAutoConfiguration; +import org.springframework.boot.builder.SpringApplicationBuilder; + +import javax.sql.DataSource; + +public class MinimalBootMain { + @SpringBootConfiguration + @EnableAutoConfiguration + static class Application {} + + public static void main(String[] args) { + try (var context = + new SpringApplicationBuilder(Application.class) + .web(WebApplicationType.NONE) + .properties("spring.main.banner-mode=off", "logging.level.root=WARN") + .run(args)) { + if (!context.getBeansOfType(DataSource.class).isEmpty() + || !context.getBeansOfType(JobRepository.class).isEmpty() + || !context.getBeansOfType(JobOperator.class).isEmpty()) { + throw new AssertionError( + "Disabled module created database or Batch infrastructure"); + } + System.out.println( + "OWNED_MODULE_ORIGIN=" + + org.fireflyframework.batch.jdbc.JdbcBatchSteps.class + .getProtectionDomain() + .getCodeSource() + .getLocation()); + System.out.println("OWNED_MINIMAL_BOOT_NO_RESOURCES=true"); + } catch (RuntimeException failure) { + failure.printStackTrace(System.err); + throw failure; + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/OwnedRunFixture.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/OwnedRunFixture.java new file mode 100644 index 000000000..1a820ac78 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/OwnedRunFixture.java @@ -0,0 +1,199 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture; + +import org.fireflyframework.batch.jdbc.JdbcBatchSteps; +import org.fireflyframework.security.api.domain.SecurityPrincipal; +import org.fireflyframework.security.spi.PolicyDecisionPort; +import org.springframework.batch.core.configuration.support.MapJobRegistry; +import org.springframework.batch.core.job.Job; +import org.springframework.batch.core.job.builder.JobBuilder; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.context.annotation.AnnotationConfigApplicationContext; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.transaction.support.TransactionTemplate; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.util.HexFormat; +import java.util.Map; + +import javax.sql.DataSource; + +/** Application-owned synthetic recipe, not a framework financial run authority. */ +public final class OwnedRunFixture { + public record Intent(String tenant, String run, String canonical) { + public String digest() { + try { + return HexFormat.of() + .formatHex( + MessageDigest.getInstance("SHA-256") + .digest(canonical.getBytes(StandardCharsets.UTF_8))); + } catch (java.security.NoSuchAlgorithmException impossible) { + throw new IllegalStateException(impossible); + } + } + } + + private final JdbcTemplate jdbc; + private final TransactionTemplate transaction; + private final PolicyDecisionPort policy; + private final SecurityPrincipal servicePrincipal; + + public OwnedRunFixture( + DataSource source, + DataSourceTransactionManager manager, + PolicyDecisionPort policy, + SecurityPrincipal servicePrincipal) { + this.jdbc = new JdbcTemplate(source); + this.transaction = new TransactionTemplate(manager); + this.policy = policy; + this.servicePrincipal = servicePrincipal; + } + + public String claim(Intent intent) { + authorize(intent); + return transaction.execute( + status -> { + int inserted = + jdbc.update( + "INSERT INTO" + + " owned_run_claim(tenant,run_id,canonical,digest,state)" + + " VALUES (?,?,?,?,'OPEN') ON CONFLICT(tenant,run_id) DO" + + " NOTHING", + intent.tenant(), + intent.run(), + intent.canonical(), + intent.digest()); + var stored = + jdbc.queryForMap( + "SELECT canonical,digest FROM owned_run_claim WHERE tenant=?" + + " AND run_id=? FOR UPDATE", + intent.tenant(), + intent.run()); + match(intent, stored); + return inserted == 1 ? "CREATED" : "REPLAY"; + }); + } + + private void authorize(Intent intent) { + if (servicePrincipal == null + || !intent.tenant().equals(servicePrincipal.tenantId()) + || !servicePrincipal.hasScope("owned-run:execute")) { + throw new IllegalStateException("Current owner service tenant or scope denied"); + } + try { + var publisher = + policy.authorize( + servicePrincipal, + "owned-run:execute", + "owned-run:" + intent.tenant() + ":" + intent.run(), + Map.of( + "tenant", + intent.tenant(), + "run", + intent.run(), + "intentDigest", + intent.digest())); + var decision = + publisher == null ? null : publisher.block(java.time.Duration.ofSeconds(1)); + if (decision == null || !decision.granted() || !decision.obligations().isEmpty()) { + throw new IllegalStateException("Current owner service policy denied"); + } + } catch (RuntimeException failure) { + throw new IllegalStateException("Current owner service policy denied", failure); + } + } + + private void requireMatchingClaim(Intent intent) { + match( + intent, + jdbc.queryForMap( + "SELECT canonical,digest FROM owned_run_claim WHERE tenant=? AND run_id=?", + intent.tenant(), + intent.run())); + } + + private static void match(Intent intent, Map stored) { + if (!intent.canonical().equals(stored.get("canonical")) + || !intent.digest().equals(stored.get("digest"))) { + throw new IllegalStateException("Owner run intent conflicts with immutable claim"); + } + } + + public Job job(AnnotationConfigApplicationContext context, Intent intent, boolean failAtFive) + throws Exception { + requireMatchingClaim(intent); + var repository = context.getBean(JobRepository.class); + var reader = new CheckpointJobFixture.CursorReader(); + var step = + context.getBean(JdbcBatchSteps.class) + . chunk("owned-run-step", 3) + .reader(reader) + .writer( + items -> { + for (Integer item : items) { + authorize(intent); + requireMatchingClaim(intent); + jdbc.update( + "INSERT INTO owned_run_items(tenant,run_id,item_no)" + + " VALUES (?,?,?)", + intent.tenant(), + intent.run(), + item); + if (failAtFive && item == 5) + throw new IllegalStateException( + "Owned recipe fixture failure at five"); + } + }) + .build(); + Job job = new JobBuilder("owned-run-job", repository).start(step).build(); + context.getBean(MapJobRegistry.class).register(job); + return job; + } + + public boolean completeOnlyWithStoredAcknowledgments(Intent intent) { + authorize(intent); + return transaction.execute( + status -> { + requireMatchingClaim(intent); + Integer items = + jdbc.queryForObject( + "SELECT count(*) FROM owned_run_items WHERE tenant=? AND" + + " run_id=?", + Integer.class, + intent.tenant(), + intent.run()); + Integer missing = + jdbc.queryForObject( + "SELECT count(*) FROM owned_run_items i LEFT JOIN" + + " owned_run_receipts r ON r.tenant=i.tenant AND" + + " r.run_id=i.run_id AND r.item_no=i.item_no WHERE" + + " i.tenant=? AND i.run_id=? AND r.item_no IS NULL", + Integer.class, + intent.tenant(), + intent.run()); + if (items != 10 || missing != 0) return false; + jdbc.update( + "UPDATE owned_run_claim SET state='COMPLETED' WHERE tenant=? AND" + + " run_id=?", + intent.tenant(), + intent.run()); + return true; + }); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/PostgresBatchFixture.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/PostgresBatchFixture.java new file mode 100644 index 000000000..9b91e6e02 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/PostgresBatchFixture.java @@ -0,0 +1,167 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture; + +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.postgresql.ds.PGSimpleDataSource; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; +import org.testcontainers.containers.PostgreSQLContainer; + +import java.nio.charset.StandardCharsets; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.UUID; + +import javax.sql.DataSource; + +public abstract class PostgresBatchFixture { + protected static PostgreSQLContainer> database; + protected static String role; + protected static String password; + + @BeforeAll + static void openPrivateDatabase() throws Exception { + database = + new PostgreSQLContainer<>("postgres:17-alpine") + .withDatabaseName("owned_batch_fixture") + .withUsername("owned_bootstrap") + .withPassword(UUID.randomUUID().toString()); + database.start(); + role = "batch_runtime_" + UUID.randomUUID().toString().replace("-", ""); + password = UUID.randomUUID().toString(); + try (var connection = owner().getConnection()) { + createRuntimeRole(connection, role, password); + } + } + + static void createRuntimeRole(Connection connection, String roleName, String rolePassword) + throws SQLException { + if (invalidCredential(roleName) + || roleName.getBytes(StandardCharsets.UTF_8).length > 63 + || invalidCredential(rolePassword)) { + throw new IllegalArgumentException("Invalid runtime role credentials"); + } + try { + try (var statement = connection.createStatement()) { + statement.execute( + """ + CREATE OR REPLACE FUNCTION pg_temp.create_batch_runtime_role(role_name text, role_password text) + RETURNS void + LANGUAGE plpgsql + SECURITY INVOKER + SET search_path = pg_catalog + AS $role_setup$ + BEGIN + EXECUTE pg_catalog.format( + 'CREATE ROLE %I LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD %L', + role_name, role_password); + END + $role_setup$ + """); + } + try (var query = + connection.prepareStatement("SELECT pg_temp.create_batch_runtime_role(?, ?)")) { + query.setString(1, roleName); + query.setString(2, rolePassword); + query.execute(); + } + } catch (SQLException failure) { + throw new SQLException( + "Runtime role provisioning failed", + failure.getSQLState(), + failure.getErrorCode()); + } + } + + private static boolean invalidCredential(String value) { + return value == null + || value.isBlank() + || value.indexOf('\0') >= 0 + || value.codePoints() + .anyMatch( + codePoint -> + codePoint >= Character.MIN_SURROGATE + && codePoint <= Character.MAX_SURROGATE); + } + + @AfterAll + static void closePrivateDatabase() { + if (database != null) database.stop(); + } + + protected static PGSimpleDataSource owner() { + var source = new PGSimpleDataSource(); + source.setURL(database.getJdbcUrl()); + source.setUser(database.getUsername()); + source.setPassword(database.getPassword()); + return source; + } + + protected static DataSource schema(String variant) throws Exception { + String name = "owned_" + UUID.randomUUID().toString().replace("-", ""); + String ddl; + try (var resource = + PostgresBatchFixture.class.getResourceAsStream( + "/org/springframework/batch/core/schema-postgresql.sql")) { + if (resource == null) + throw new IllegalStateException("Native Batch PostgreSQL schema missing"); + ddl = new String(resource.readAllBytes(), StandardCharsets.UTF_8); + } + if (variant.equals("missing-column")) ddl = ddl.replace("\tFILTER_COUNT BIGINT,", ""); + if (variant.equals("wrong-column-type")) + ddl = ddl.replace("\tFILTER_COUNT BIGINT,", "\tFILTER_COUNT VARCHAR(100),"); + if (variant.equals("missing-sequence")) + ddl = + ddl.replace( + "CREATE SEQUENCE BATCH_JOB_INSTANCE_SEQ MAXVALUE 9223372036854775807 NO" + + " CYCLE;", + ""); + try (var connection = owner().getConnection(); + var statement = connection.createStatement()) { + statement.execute("CREATE SCHEMA " + name); + connection.setSchema(name); + if (!variant.equals("empty")) statement.execute(ddl); + if (!variant.equals("no-access")) { + statement.execute("GRANT USAGE ON SCHEMA " + name + " TO " + role); + statement.execute( + "GRANT " + + (variant.equals("read-only") ? "SELECT" : "SELECT,INSERT,UPDATE") + + " ON ALL TABLES IN SCHEMA " + + name + + " TO " + + role); + statement.execute( + "GRANT USAGE,SELECT ON ALL SEQUENCES IN SCHEMA " + name + " TO " + role); + } + } + var runtime = new PGSimpleDataSource(); + runtime.setURL(database.getJdbcUrl()); + runtime.setUser(role); + runtime.setPassword(password); + runtime.setCurrentSchema(name); + return runtime; + } + + protected static ThreadPoolTaskExecutor executor() { + var executor = new ThreadPoolTaskExecutor(); + executor.setCorePoolSize(2); + executor.setMaxPoolSize(2); + executor.setQueueCapacity(0); + executor.setThreadNamePrefix("owned-batch-"); + return executor; + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/PostgresBatchRoleProvisioningTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/PostgresBatchRoleProvisioningTest.java new file mode 100644 index 000000000..0c27d0fb7 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/PostgresBatchRoleProvisioningTest.java @@ -0,0 +1,195 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatExceptionOfType; +import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException; + +import org.junit.jupiter.api.Test; +import org.postgresql.ds.PGSimpleDataSource; + +import java.nio.charset.StandardCharsets; +import java.sql.SQLException; +import java.util.UUID; + +class PostgresBatchRoleProvisioningTest extends PostgresBatchFixture { + private static String freshRole() { + return "batch_role_" + UUID.randomUUID().toString().replace("-", ""); + } + + private static String freshPassword() { + return UUID.randomUUID().toString(); + } + + private void provision(String name, String secret) throws Exception { + try (var connection = owner().getConnection()) { + createRuntimeRole(connection, name, secret); + } + } + + private long roleCount() throws Exception { + try (var connection = owner().getConnection(); + var statement = connection.createStatement(); + var rows = statement.executeQuery("SELECT count(*) FROM pg_catalog.pg_roles")) { + rows.next(); + return rows.getLong(1); + } + } + + private void assertLoginAndPrivileges(String name, String secret) throws Exception { + var runtime = new PGSimpleDataSource(); + runtime.setURL(database.getJdbcUrl()); + runtime.setUser(name); + runtime.setPassword(secret); + try (var connection = runtime.getConnection(); + var query = + connection.prepareStatement( + """ + SELECT current_user, rolcanlogin, rolsuper, rolbypassrls, + rolcreatedb, rolcreaterole, rolreplication, + (SELECT count(*) FROM pg_catalog.pg_auth_members WHERE member = r.oid) + FROM pg_catalog.pg_roles r WHERE rolname = ? + """)) { + query.setString(1, name); + try (var rows = query.executeQuery()) { + assertThat(rows.next()).isTrue(); + assertThat(rows.getString(1)).isEqualTo(name); + assertThat(rows.getBoolean(2)).isTrue(); + for (int column = 3; column <= 7; column++) { + assertThat(rows.getBoolean(column)).isFalse(); + } + assertThat(rows.getLong(8)).isZero(); + assertThat(rows.next()).isFalse(); + } + try (var forbidden = connection.createStatement()) { + assertThatExceptionOfType(SQLException.class) + .isThrownBy( + () -> forbidden.execute("CREATE ROLE forbidden_fixture_privilege")) + .satisfies(failure -> assertThat(failure.getSQLState()).isEqualTo("42501")); + assertThatExceptionOfType(SQLException.class) + .isThrownBy( + () -> + forbidden.execute( + "CREATE DATABASE forbidden_fixture_database")) + .satisfies(failure -> assertThat(failure.getSQLState()).isEqualTo("42501")); + } + } + } + + @Test + void ordinaryRuntimeRoleLogsInWithoutElevatedPrivileges() throws Exception { + String name = freshRole(); + String secret = freshPassword(); + provision(name, secret); + assertLoginAndPrivileges(name, secret); + } + + @Test + void quotedRoleAndPasswordRemainDataAndCreateExactlyOneRole() throws Exception { + String name = "batch_😀\"';--" + UUID.randomUUID().toString().replace("-", ""); + String secret = freshPassword() + "'\"; Ω😀"; + long before = roleCount(); + provision(name, secret); + assertThat(roleCount()).isEqualTo(before + 1); + assertLoginAndPrivileges(name, secret); + } + + @Test + void identifierByteLimitPreservesExactMultibyteRole() throws Exception { + String[] names = {"b".repeat(61) + "é", "c".repeat(59) + "😀"}; + for (String name : names) { + assertThat(name.getBytes(StandardCharsets.UTF_8)).hasSize(63); + String secret = freshPassword(); + provision(name, secret); + assertLoginAndPrivileges(name, secret); + } + } + + @Test + void invalidRoleNamesFailBeforeAnyRoleEffect() throws Exception { + long before = roleCount(); + String[] invalid = { + null, + "", + " ", + "a\0b", + "b".repeat(64), + "b".repeat(62) + "é", + "b".repeat(60) + "😀", + "bad\uD800" + }; + for (String name : invalid) { + assertThatIllegalArgumentException() + .isThrownBy(() -> provision(name, freshPassword())) + .withMessage("Invalid runtime role credentials"); + assertThat(roleCount()).isEqualTo(before); + } + } + + @Test + void invalidPasswordsFailBeforeAnyRoleEffect() throws Exception { + long before = roleCount(); + String[] invalid = {null, "", " ", "a\0b", "bad\uDC00"}; + for (String secret : invalid) { + assertThatIllegalArgumentException() + .isThrownBy(() -> provision(freshRole(), secret)) + .withMessage("Invalid runtime role credentials"); + assertThat(roleCount()).isEqualTo(before); + } + } + + @Test + void nativeProvisioningFailureRetainsSqlStateWithoutCredentialDiagnostics() throws Exception { + String name = freshRole(); + String secret = freshPassword(); + provision(name, secret); + assertThatExceptionOfType(SQLException.class) + .isThrownBy(() -> provision(name, secret)) + .satisfies( + failure -> { + assertThat(failure.getSQLState()).isEqualTo("42710"); + var output = new java.io.StringWriter(); + failure.printStackTrace(new java.io.PrintWriter(output)); + assertThat(output.toString()).doesNotContain(secret); + assertThat(failure.getMessage()) + .isEqualTo("Runtime role provisioning failed"); + }); + assertLoginAndPrivileges(name, secret); + } + + @Test + void provisioningFunctionIsTemporaryInvokerWithFixedSearchPath() throws Exception { + try (var connection = owner().getConnection()) { + createRuntimeRole(connection, freshRole(), freshPassword()); + try (var statement = + connection.prepareStatement( + """ + SELECT p.prosecdef, p.proconfig, n.nspname + FROM pg_catalog.pg_proc p JOIN pg_catalog.pg_namespace n ON p.pronamespace = n.oid + WHERE p.oid = pg_catalog.to_regprocedure('pg_temp.create_batch_runtime_role(text,text)') + """); + var rows = statement.executeQuery()) { + assertThat(rows.next()).isTrue(); + assertThat(rows.getBoolean(1)).isFalse(); + assertThat((String[]) rows.getArray(2).getArray()) + .containsExactly("search_path=pg_catalog"); + assertThat(rows.getString(3)).startsWith("pg_temp_"); + assertThat(rows.next()).isFalse(); + } + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/fixture/project/ScannedNativeProviders.java b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/project/ScannedNativeProviders.java new file mode 100644 index 000000000..1c188872f --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/fixture/project/ScannedNativeProviders.java @@ -0,0 +1,60 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.fixture.project; + +import org.springframework.batch.core.configuration.support.MapJobRegistry; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.launch.support.JobOperatorFactoryBean; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.core.repository.support.JdbcJobRepositoryFactoryBean; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; +import org.springframework.transaction.PlatformTransactionManager; + +import javax.sql.DataSource; + +@Configuration(proxyBeanMethods = false) +public class ScannedNativeProviders { + @Bean + JobRepository projectRepository( + @Qualifier("ownedDataSource") DataSource source, + @Qualifier("ownedManager") PlatformTransactionManager manager) + throws Exception { + var factory = new JdbcJobRepositoryFactoryBean(); + factory.setDataSource(source); + factory.setTransactionManager(manager); + factory.setDatabaseType("POSTGRES"); + factory.afterPropertiesSet(); + return factory.getObject(); + } + + @Bean + JobOperator projectOperator( + JobRepository repository, + @Qualifier("ownedManager") PlatformTransactionManager manager, + @Qualifier("ownedExecutor") ThreadPoolTaskExecutor executor) + throws Exception { + var factory = new JobOperatorFactoryBean(); + factory.setJobRepository(repository); + factory.setJobRegistry(new MapJobRegistry()); + factory.setTransactionManager(manager); + factory.setTaskExecutor(executor); + factory.afterPropertiesSet(); + return factory.getObject(); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchConcurrencyTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchConcurrencyTest.java new file mode 100644 index 000000000..85ccde2d3 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchConcurrencyTest.java @@ -0,0 +1,157 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.jdbc; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import org.fireflyframework.batch.fixture.CheckpointJobFixture; +import org.fireflyframework.batch.fixture.PostgresBatchFixture; +import org.junit.jupiter.api.Test; +import org.springframework.batch.core.BatchStatus; +import org.springframework.batch.core.configuration.support.MapJobRegistry; +import org.springframework.batch.core.job.Job; +import org.springframework.batch.core.job.JobExecution; +import org.springframework.batch.core.job.builder.JobBuilder; +import org.springframework.batch.core.job.parameters.JobParameters; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.infrastructure.repeat.RepeatStatus; +import org.springframework.jdbc.core.JdbcTemplate; + +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; + +class JdbcBatchConcurrencyTest extends PostgresBatchFixture { + private JobParameters parameters(String tenant, String run) { + return new JobParametersBuilder() + .addString("tenant", tenant, true) + .addString("run", run, true) + .toJobParameters(); + } + + @Test + void runningAndCompletedDuplicateLaunchesCannotExecuteAgain() throws Exception { + var source = schema("complete"); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var operator = context.getBean(JobOperator.class); + var entered = new CountDownLatch(1); + var release = new CountDownLatch(1); + var effects = new java.util.concurrent.atomic.AtomicInteger(); + var helper = context.getBean(JdbcBatchSteps.class); + var step = + helper.tasklet( + "owned-admission-step", + (contribution, chunk) -> { + entered.countDown(); + if (!release.await(10, TimeUnit.SECONDS)) + throw new IllegalStateException( + "Owned barrier expired"); + effects.incrementAndGet(); + return RepeatStatus.FINISHED; + }) + .build(); + Job job = new JobBuilder("owned-admission-job", repository).start(step).build(); + context.getBean(MapJobRegistry.class).register(job); + var parameters = parameters("owned-a", "stable-run"); + JobExecution first = operator.start(job, parameters); + try { + assertThat(entered.await(10, TimeUnit.SECONDS)).isTrue(); + assertThatThrownBy(() -> operator.start(job, parameters)) + .isInstanceOf( + org.springframework.batch.core.launch + .JobExecutionAlreadyRunningException.class); + assertThat(effects.get()).isZero(); + } finally { + release.countDown(); + } + assertThat(CheckpointJobFixture.await(repository, first.getId()).getStatus()) + .isEqualTo(BatchStatus.COMPLETED); + assertThatThrownBy(() -> operator.start(job, parameters)) + .isInstanceOf( + org.springframework.batch.core.launch + .JobInstanceAlreadyCompleteException.class); + assertThat(effects.get()).isEqualTo(1); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", + Integer.class)) + .isEqualTo(1); + assertThat(repository.getJobExecutions(first.getJobInstance())).hasSize(1); + var otherTenant = + repository.createJobInstance( + job.getName(), parameters("owned-b", "stable-run")); + assertThat(otherTenant.getId()).isNotEqualTo(first.getJobInstanceId()); + } + } + + @Test + void fullBoundedExecutorRejectsThirdExecutionWithoutCallerThreadEffects() throws Exception { + var source = schema("complete"); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var operator = context.getBean(JobOperator.class); + var entered = new CountDownLatch(2); + var release = new CountDownLatch(1); + var rejectedEntered = new AtomicBoolean(); + var effects = new java.util.concurrent.atomic.AtomicInteger(); + var helper = context.getBean(JdbcBatchSteps.class); + var step = + helper.tasklet( + "owned-bound-step", + (contribution, chunk) -> { + if (contribution + .getStepExecution() + .getJobParameters() + .getString("run") + .equals("third")) { + rejectedEntered.set(true); + } + entered.countDown(); + if (!release.await(10, TimeUnit.SECONDS)) + throw new IllegalStateException( + "Owned barrier expired"); + effects.incrementAndGet(); + return RepeatStatus.FINISHED; + }) + .build(); + Job job = new JobBuilder("owned-bound-job", repository).start(step).build(); + context.getBean(MapJobRegistry.class).register(job); + JobExecution first = operator.start(job, parameters("owned", "first")); + JobExecution second = operator.start(job, parameters("owned", "second")); + try { + assertThat(entered.await(10, TimeUnit.SECONDS)).isTrue(); + JobExecution third = operator.start(job, parameters("owned", "third")); + assertThat(repository.getJobExecution(third.getId()).getStatus()) + .isEqualTo(BatchStatus.FAILED); + assertThat(rejectedEntered.get()).isFalse(); + assertThat(effects.get()).isZero(); + } finally { + release.countDown(); + } + assertThat(CheckpointJobFixture.await(repository, first.getId()).getStatus()) + .isEqualTo(BatchStatus.COMPLETED); + assertThat(CheckpointJobFixture.await(repository, second.getId()).getStatus()) + .isEqualTo(BatchStatus.COMPLETED); + assertThat(effects.get()).isEqualTo(2); + assertThat(rejectedEntered.get()).isFalse(); + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchProcessDatabaseIT.java b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchProcessDatabaseIT.java new file mode 100644 index 000000000..8b1115b5d --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchProcessDatabaseIT.java @@ -0,0 +1,463 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.jdbc; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.fireflyframework.batch.fixture.BatchProcessMain; +import org.fireflyframework.batch.fixture.CheckpointJobFixture; +import org.fireflyframework.batch.fixture.PostgresBatchFixture; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.postgresql.ds.PGSimpleDataSource; +import org.springframework.jdbc.core.JdbcTemplate; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import javax.sql.DataSource; + +class JdbcBatchProcessDatabaseIT extends PostgresBatchFixture { + private Path temporary; + + @BeforeEach + void preserveOwnedProcessEvidence() throws Exception { + Path parent = Path.of("target/durable-process-evidence"); + Files.createDirectories(parent); + temporary = Files.createTempDirectory(parent, "owned-").toAbsolutePath(); + System.out.println("OWNED_PROCESS_EVIDENCE=" + temporary); + } + + private final List children = new ArrayList<>(); + + @AfterEach + void cleanupOwnedChildren() throws Exception { + var observed = new StringBuilder(); + for (Process child : children) { + if (child.isAlive()) child.destroyForcibly(); + assertThat(child.waitFor(10, TimeUnit.SECONDS)).isTrue(); + assertThat(child.isAlive()).isFalse(); + observed.append(child.pid()).append("\t").append(child.isAlive()).append("\n"); + } + Files.writeString(temporary.resolve("owned-child-cleanup.tsv"), observed); + } + + private DataSource prepared() throws Exception { + DataSource source = schema("complete"); + String selected; + try (var connection = source.getConnection()) { + selected = connection.getSchema(); + } + try (var connection = owner().getConnection(); + var statement = connection.createStatement()) { + connection.setSchema(selected); + statement.execute( + "CREATE TABLE owned_items(run_id varchar(100) NOT NULL,item_no integer NOT" + + " NULL,PRIMARY KEY(run_id,item_no))"); + statement.execute("GRANT SELECT,INSERT ON owned_items TO " + role); + } + return source; + } + + private Process child(DataSource source, String name, String mode, Long execution, Path release) + throws Exception { + return child(source, name, mode, execution, release, null); + } + + private Process child( + DataSource source, + String name, + String mode, + Long execution, + Path release, + Path startFile) + throws Exception { + Path bootstrap = temporary.resolve(name + "-bootstrap"); + for (Class> type : + List.of( + BatchProcessMain.class, + CheckpointJobFixture.class, + CheckpointJobFixture.CursorReader.class)) { + String resource = type.getName().replace('.', '/') + ".class"; + Path target = bootstrap.resolve(resource); + Files.createDirectories(target.getParent()); + try (var input = type.getClassLoader().getResourceAsStream(resource)) { + if (input == null) throw new IllegalStateException("Owned bootstrap class missing"); + Files.write(target, input.readAllBytes()); + } + } + Path artifact = Path.of(System.getProperty("durable.batch.jar")); + assertThat(artifact).isRegularFile(); + String runtime = Files.readString(Path.of("target/runtime-classpath.txt")).trim(); + assertThat(runtime) + .doesNotContain("spring-boot-batch", "testcontainers", "junit", "target/classes"); + String driver = + Path.of( + PGSimpleDataSource.class + .getProtectionDomain() + .getCodeSource() + .getLocation() + .toURI()) + .toString(); + String classpath = + bootstrap + + java.io.File.pathSeparator + + artifact + + java.io.File.pathSeparator + + runtime + + java.io.File.pathSeparator + + driver; + var pins = new StringBuilder(); + for (String entry : + classpath.split(java.util.regex.Pattern.quote(java.io.File.pathSeparator))) { + Path input = Path.of(entry); + if (Files.isRegularFile(input)) { + pins.append(input.toAbsolutePath()) + .append("\t") + .append( + java.util.HexFormat.of() + .formatHex( + java.security.MessageDigest.getInstance("SHA-256") + .digest(Files.readAllBytes(input)))) + .append("\n"); + } + } + Files.writeString(temporary.resolve(name + "-jar-pins.tsv"), pins); + var command = + new ArrayList<>( + List.of( + Path.of(System.getProperty("java.home"), "bin/java").toString(), + "-Xms16m", + "-Xmx192m", + "-cp", + classpath, + BatchProcessMain.class.getName(), + mode)); + if (execution != null) command.add(Long.toString(execution)); + var builder = + new ProcessBuilder(command) + .redirectErrorStream(true) + .redirectOutput(temporary.resolve(name + ".log").toFile()); + var environment = builder.environment(); + environment.put("OWNED_BATCH_URL", database.getJdbcUrl()); + environment.put("OWNED_BATCH_USER", role); + environment.put("OWNED_BATCH_PASSWORD", password); + try (var connection = source.getConnection()) { + environment.put("OWNED_BATCH_SCHEMA", connection.getSchema()); + } + environment.put("OWNED_BATCH_RUN", "stable-process-run"); + environment.put("OWNED_BATCH_TENANT", "owned-tenant"); + environment.remove("OWNED_BATCH_RELEASE_FILE"); + environment.remove("OWNED_BATCH_START_FILE"); + if (startFile != null) environment.put("OWNED_BATCH_START_FILE", startFile.toString()); + if (release != null) environment.put("OWNED_BATCH_RELEASE_FILE", release.toString()); + Process started = builder.start(); + children.add(started); + return started; + } + + private void held(DataSource source, Process process, String name) throws Exception { + long deadline = System.nanoTime() + Duration.ofSeconds(15).toNanos(); + Path log = temporary.resolve(name + ".log"); + while (System.nanoTime() < deadline) { + if (!process.isAlive()) throw new AssertionError(Files.readString(log)); + if (Files.readString(log).contains("OWNED_HOLD_BEFORE_FOUR=true")) { + assertThat(rows(source)).containsExactly(1, 2, 3); + return; + } + Thread.sleep(20); + } + throw new AssertionError( + "Owned child did not reach first committed checkpoint: " + Files.readString(log)); + } + + private List rows(DataSource source) { + return new JdbcTemplate(source) + .queryForList("SELECT item_no FROM owned_items ORDER BY item_no", Integer.class); + } + + private long lastExecution(DataSource source) { + return new JdbcTemplate(source) + .queryForObject( + "SELECT max(JOB_EXECUTION_ID) FROM BATCH_JOB_EXECUTION", Long.class); + } + + private void success(Process process, String name) throws Exception { + assertThat(process.waitFor(30, TimeUnit.SECONDS)).isTrue(); + String output = Files.readString(temporary.resolve(name + ".log")); + assertThat(process.exitValue()).withFailMessage(output).isZero(); + assertThat(output) + .contains( + "OWNED_MODULE_ORIGIN=" + + Path.of(System.getProperty("durable.batch.jar")).toUri().toURL()); + } + + private static boolean isNativeEmptyExecutionObservation( + String output, long instanceId, long version) { + var lines = output.lines().toList(); + return lines.contains( + "Exception in thread \"main\" java.lang.IllegalStateException: Cannot find" + + " any job execution for job instance: JobInstance: id=" + + instanceId + + ", version=" + + version + + ", Job=[owned-checkpoint-job]") + && lines.contains( + "\tat org.springframework.batch.core.launch.support.TaskExecutorJobLauncher.createJobExecution(TaskExecutorJobLauncher.java:130)"); + } + + private static boolean isNativeSerializationRejection(String output) { + var lines = output.lines().toList(); + return lines.contains( + "Exception in thread \"main\"" + + " org.springframework.dao.CannotAcquireLockException:" + + " PreparedStatementCallback; SQL [INSERT INTO" + + " BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)") + && lines.contains( + "Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize" + + " access due to read/write dependencies among transactions") + && lines.contains( + "\tat org.springframework.batch.core.repository.dao.jdbc.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:164)") + && lines.contains( + "\tat org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:144)"); + } + + @Test + void nativeRejectionRequiresExactDiagnosticDescriptors() { + String observed = + "OWNED_READY_TO_START=true\n" + + "Exception in thread \"main\" java.lang.IllegalStateException: Cannot find" + + " any job execution for job instance: JobInstance: id=7, version=0," + + " Job=[owned-checkpoint-job]\n" + + "\tat org.springframework.batch.core.launch.support.TaskExecutorJobLauncher.createJobExecution(TaskExecutorJobLauncher.java:130)\n"; + assertThat(isNativeEmptyExecutionObservation(observed, 7, 0)).isTrue(); + for (String other : + List.of( + observed.replace("id=7", "id=8"), + observed.replace("version=0", "version=1"), + observed.replace("owned-checkpoint-job", "another-job"), + observed.replace("IllegalStateException", "IllegalArgumentException"), + observed.replace(":130)", ":131)"), + observed.replace("createJobExecution(", "otherMethod("), + observed.replace("Exception in thread", "Prefix Exception in thread"), + "java.lang.IllegalStateException: unrelated initialization failure")) { + assertThat(isNativeEmptyExecutionObservation(other, 7, 0)).isFalse(); + } + String serialization = + "Exception in thread \"main\" org.springframework.dao.CannotAcquireLockException:" + + " PreparedStatementCallback; SQL [INSERT INTO" + + " BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n" + + "\tat org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:144)\n" + + "\tat org.springframework.batch.core.repository.dao.jdbc.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:164)\n" + + "Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize" + + " access due to read/write dependencies among transactions\n"; + assertThat(isNativeSerializationRejection(serialization)).isTrue(); + for (String other : + List.of( + serialization.replace("BATCH_JOB_INSTANCE(", "ANOTHER_TABLE("), + serialization.replace( + "CannotAcquireLockException", "IllegalStateException"), + serialization.replace( + "could not serialize access due to read/write dependencies among" + + " transactions", + "deadlock detected"), + serialization.replace( + "org.postgresql.util.PSQLException", "java.sql.SQLException"), + serialization.replace(":164)", ":165)"), + serialization.replace(":144)", ":145)"), + serialization.replace("Exception in thread", "Prefix Exception in thread"), + "org.springframework.dao.CannotAcquireLockException: unrelated lock" + + " failure")) { + assertThat(isNativeSerializationRejection(other)).isFalse(); + } + } + + @Test + void simultaneousIndependentProcessesAdmitOneLiveExecutionAndOneEffectStream() + throws Exception { + DataSource source = prepared(); + Path startFile = temporary.resolve("simultaneous-start"); + Path release = temporary.resolve("winner-release"); + Process first = child(source, "race-first", "start", null, release, startFile); + Process second = child(source, "race-second", "start", null, release, startFile); + long readyDeadline = System.nanoTime() + Duration.ofSeconds(15).toNanos(); + while (!Files.readString(temporary.resolve("race-first.log")) + .contains("OWNED_READY_TO_START=true") + || !Files.readString(temporary.resolve("race-second.log")) + .contains("OWNED_READY_TO_START=true")) { + if (!first.isAlive() || !second.isAlive() || System.nanoTime() > readyDeadline) { + throw new AssertionError( + Files.readString(temporary.resolve("race-first.log")) + + Files.readString(temporary.resolve("race-second.log"))); + } + Thread.sleep(20); + } + Files.writeString(startFile, "owned synchronized admission"); + long raceDeadline = System.nanoTime() + Duration.ofSeconds(15).toNanos(); + while (first.isAlive() == second.isAlive()) { + if (System.nanoTime() > raceDeadline) + throw new AssertionError("Native concurrent admission did not settle"); + Thread.sleep(20); + } + Process winner = first.isAlive() ? first : second; + Process loser = first.isAlive() ? second : first; + String winnerName = first.isAlive() ? "race-first" : "race-second"; + String loserName = first.isAlive() ? "race-second" : "race-first"; + held(source, winner, winnerName); + assertThat(loser.waitFor(10, TimeUnit.SECONDS)).isTrue(); + assertThat(loser.exitValue()).isNotZero(); + var jdbc = new JdbcTemplate(source); + assertThat(jdbc.queryForObject("SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isEqualTo(1); + var instance = + jdbc.queryForMap("SELECT JOB_INSTANCE_ID,VERSION,JOB_NAME FROM BATCH_JOB_INSTANCE"); + long instanceId = ((Number) instance.get("job_instance_id")).longValue(); + long instanceVersion = ((Number) instance.get("version")).longValue(); + assertThat(instance.get("job_name")).isEqualTo("owned-checkpoint-job"); + assertThat(jdbc.queryForObject("SELECT count(*) FROM BATCH_JOB_EXECUTION", Integer.class)) + .isEqualTo(1); + var execution = jdbc.queryForMap("SELECT JOB_INSTANCE_ID,STATUS FROM BATCH_JOB_EXECUTION"); + assertThat(((Number) execution.get("job_instance_id")).longValue()).isEqualTo(instanceId); + assertThat(execution.get("status")).isEqualTo("STARTED"); + String loserOutput = Files.readString(temporary.resolve(loserName + ".log")); + assertThat(loserOutput).doesNotContain("OWNED_EXECUTION_ID=", "OWNED_FINAL_STATUS="); + if (!isNativeEmptyExecutionObservation(loserOutput, instanceId, instanceVersion) + && !isNativeSerializationRejection(loserOutput)) { + assertThat(loserOutput) + .containsAnyOf( + "JobExecutionAlreadyRunningException", + "DuplicateKeyException", + "CannotSerializeTransactionException", + "JobInstanceAlreadyExistsException", + "java.lang.IllegalStateException: JobInstance must not already exist"); + } + Files.writeString(release, "owned admitted process may finish"); + success(winner, winnerName); + assertThat(rows(source)).containsExactly(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + assertThat( + jdbc.queryForObject( + "SELECT count(*) FROM BATCH_JOB_EXECUTION WHERE STATUS='COMPLETED'", + Integer.class)) + .isEqualTo(1); + assertThat( + jdbc.queryForObject( + "SELECT count(*) FROM BATCH_JOB_EXECUTION WHERE STATUS IN" + + " ('STARTING','STARTED','STOPPING')", + Integer.class)) + .isZero(); + Files.writeString( + temporary.resolve("concurrent-observation.txt"), + "winner=" + winnerName + "\nloser=" + loserName + "\nrows=" + rows(source)); + } + + @Test + void killedProcessRequiresObservedDeathThenNativeRecoverAndRestartFromPersistedCursor() + throws Exception { + DataSource source = prepared(); + Process original = + child(source, "before-kill", "start", null, temporary.resolve("never-release")); + held(source, original, "before-kill"); + long originalId = lastExecution(source); + original.destroyForcibly(); + assertThat(original.waitFor(10, TimeUnit.SECONDS)).isTrue(); + assertThat(original.isAlive()).isFalse(); + assertThat(rows(source)).containsExactly(1, 2, 3); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT STATUS FROM BATCH_JOB_EXECUTION WHERE" + + " JOB_EXECUTION_ID=?", + String.class, + originalId)) + .isEqualTo("STARTED"); + Files.writeString( + temporary.resolve("kill-observation.txt"), + "old-process-alive=" + + original.isAlive() + + "\nrows-before-recovery=" + + rows(source) + + "\noriginal-execution=" + + originalId); + Process replacement = child(source, "after-kill", "recover-restart", originalId, null); + success(replacement, "after-kill"); + assertThat(Files.readString(temporary.resolve("after-kill.log"))) + .contains("OWNED_RECOVERED_STORED_STATUS=FAILED", "OWNED_FINAL_STATUS=COMPLETED"); + assertThat(rows(source)).containsExactly(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + Files.writeString(temporary.resolve("kill-final-observation.txt"), "rows=" + rows(source)); + var jdbc = new JdbcTemplate(source); + assertThat(jdbc.queryForObject("SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isEqualTo(1); + assertThat(jdbc.queryForObject("SELECT count(*) FROM BATCH_JOB_EXECUTION", Integer.class)) + .isEqualTo(2); + assertThat( + jdbc.queryForObject( + "SELECT STATUS FROM BATCH_JOB_EXECUTION WHERE JOB_EXECUTION_ID=?", + String.class, + originalId)) + .isEqualTo("FAILED"); + } + + @Test + void separateProcessRequestsStopAndFreshProcessRestartsSameInstance() throws Exception { + DataSource source = prepared(); + Path release = temporary.resolve("release-after-stop"); + Process original = child(source, "before-stop", "start", null, release); + held(source, original, "before-stop"); + long originalId = lastExecution(source); + Process stopper = child(source, "stopper", "stop", originalId, null); + success(stopper, "stopper"); + assertThat(Files.readString(temporary.resolve("stopper.log"))) + .contains("OWNED_STOP_ACK=true"); + Files.writeString(release, "owned release after actual stop acknowledgment"); + success(original, "before-stop"); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT STATUS FROM BATCH_JOB_EXECUTION WHERE" + + " JOB_EXECUTION_ID=?", + String.class, + originalId)) + .isEqualTo("STOPPED"); + List stoppedRows = rows(source); + assertThat(stoppedRows.size()).isBetween(3, 6); + assertThat(stoppedRows) + .isEqualTo( + java.util.stream.IntStream.rangeClosed(1, stoppedRows.size()) + .boxed() + .toList()); + Files.writeString( + temporary.resolve("stop-observation.txt"), + "stopped-rows=" + stoppedRows + "\noriginal-execution=" + originalId); + Process replacement = child(source, "after-stop", "restart", originalId, null); + success(replacement, "after-stop"); + assertThat(rows(source)).containsExactly(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_INSTANCE", Integer.class)) + .isEqualTo(1); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM BATCH_JOB_EXECUTION", Integer.class)) + .isEqualTo(2); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchStepsTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchStepsTest.java new file mode 100644 index 000000000..354b043fd --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcBatchStepsTest.java @@ -0,0 +1,53 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.jdbc; + +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import org.junit.jupiter.api.Test; +import org.springframework.batch.core.repository.support.ResourcelessJobRepository; +import org.springframework.jdbc.datasource.DataSourceTransactionManager; +import org.springframework.jdbc.datasource.DriverManagerDataSource; + +class JdbcBatchStepsTest { + private JdbcBatchSteps steps() { + return new JdbcBatchSteps( + new ResourcelessJobRepository(), + new DataSourceTransactionManager(new DriverManagerDataSource())); + } + + @Test + void blankStepNamesAreRejectedBeforeReturningNativeBuilder() { + assertThatThrownBy(() -> steps().chunk(" ", 3)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> steps().tasklet("", (contribution, context) -> null)) + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + void nonpositiveChunkIsRejected() { + assertThatThrownBy(() -> steps().chunk("owned-step", 0)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> steps().chunk("owned-step", -1)) + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + void nullTaskletIsRejected() { + assertThatThrownBy(() -> steps().tasklet("owned-step", null)) + .isInstanceOf(IllegalArgumentException.class); + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcCheckpointDatabaseTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcCheckpointDatabaseTest.java new file mode 100644 index 000000000..270ab04b6 --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/JdbcCheckpointDatabaseTest.java @@ -0,0 +1,184 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.jdbc; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.fireflyframework.batch.fixture.CheckpointJobFixture; +import org.fireflyframework.batch.fixture.PostgresBatchFixture; +import org.junit.jupiter.api.Test; +import org.springframework.batch.core.BatchStatus; +import org.springframework.batch.core.job.JobExecution; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.jdbc.core.JdbcTemplate; + +import java.util.List; + +import javax.sql.DataSource; + +class JdbcCheckpointDatabaseTest extends PostgresBatchFixture { + private DataSource prepared() throws Exception { + DataSource source = schema("complete"); + String selected; + try (var connection = source.getConnection()) { + selected = connection.getSchema(); + } + try (var connection = owner().getConnection(); + var statement = connection.createStatement()) { + connection.setSchema(selected); + statement.execute( + "CREATE TABLE owned_items(run_id varchar(100) NOT NULL,item_no integer NOT" + + " NULL,PRIMARY KEY(run_id,item_no))"); + statement.execute("GRANT SELECT,INSERT ON owned_items TO " + role); + } + return source; + } + + private List rows(DataSource source) { + return new JdbcTemplate(source) + .queryForList("SELECT item_no FROM owned_items ORDER BY item_no", Integer.class); + } + + @Test + void failedChunkRollsBackBothItemsAndCheckpointThenRestartsInFreshContext() throws Exception { + DataSource source = prepared(); + var parameters = + new JobParametersBuilder() + .addString("tenant", "owned-tenant", true) + .addString("run", "stable-owner-run", true) + .toJobParameters(); + long failedId; + long instanceId; + try (var first = CheckpointJobFixture.open(source)) { + var repository = first.getBean(JobRepository.class); + var job = CheckpointJobFixture.job(first, source, "stable-owner-run", true, false); + JobExecution failed = + CheckpointJobFixture.await( + repository, + first.getBean(JobOperator.class).start(job, parameters).getId()); + failedId = failed.getId(); + instanceId = failed.getJobInstanceId(); + assertThat(failed.getStatus()).isEqualTo(BatchStatus.FAILED); + assertThat(rows(source)).containsExactly(1, 2, 3); + var step = repository.getLastStepExecution(failed.getJobInstance(), "owned-chunk"); + assertThat(step.getExecutionContext().getInt("owned.cursor")).isEqualTo(3); + assertThat(step.getRollbackCount()).isPositive(); + } + try (var second = CheckpointJobFixture.open(source)) { + var repository = second.getBean(JobRepository.class); + JobExecution original = repository.getJobExecution(failedId); + assertThat(original.getStatus()).isEqualTo(BatchStatus.FAILED); + assertThat( + repository + .getLastStepExecution(original.getJobInstance(), "owned-chunk") + .getExecutionContext() + .getInt("owned.cursor")) + .isEqualTo(3); + CheckpointJobFixture.job(second, source, "stable-owner-run", false, false); + JobExecution completed = + CheckpointJobFixture.await( + repository, + second.getBean(JobOperator.class).restart(original).getId()); + assertThat(completed.getId()).isNotEqualTo(failedId); + assertThat(completed.getJobInstanceId()).isEqualTo(instanceId); + assertThat(completed.getStatus()).isEqualTo(BatchStatus.COMPLETED); + assertThat(repository.getJobExecution(failedId).getStatus()) + .isEqualTo(BatchStatus.FAILED); + assertThat(repository.getJobExecutions(completed.getJobInstance())).hasSize(2); + assertThat( + repository + .getLastStepExecution(completed.getJobInstance(), "owned-chunk") + .getExecutionContext() + .getInt("owned.cursor")) + .isEqualTo(10); + assertThat(rows(source)).containsExactly(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + } + } + + @Test + void suppliedTaskletBuilderRollsBackSameResourceWritesOnFailure() throws Exception { + DataSource source = prepared(); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var helper = context.getBean(JdbcBatchSteps.class); + var step = + helper.tasklet( + "owned-tasklet", + (contribution, chunkContext) -> { + new JdbcTemplate(source) + .update( + "INSERT INTO owned_items(run_id,item_no)" + + " VALUES (?,?)", + "tasklet-run", + 1); + throw new IllegalStateException("Owned tasklet failure"); + }) + .build(); + var job = + new org.springframework.batch.core.job.builder.JobBuilder( + "owned-tasklet-job", repository) + .start(step) + .build(); + context.getBean( + org.springframework.batch.core.configuration.support.MapJobRegistry + .class) + .register(job); + var failed = + CheckpointJobFixture.await( + repository, + context.getBean(JobOperator.class) + .start( + job, + new JobParametersBuilder() + .addString("run", "tasklet-run", true) + .toJobParameters()) + .getId()); + assertThat(failed.getStatus()).isEqualTo(BatchStatus.FAILED); + assertThat(rows(source)).isEmpty(); + assertThat( + repository + .getLastStepExecution(failed.getJobInstance(), "owned-tasklet") + .getRollbackCount()) + .isPositive(); + } + } + + @Test + void nativeOmittedManagerControlLeavesPartialItemsDespiteFailedMetadata() throws Exception { + DataSource source = prepared(); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var job = + CheckpointJobFixture.job( + context, source, "omitted-manager-control", true, true); + var failed = + CheckpointJobFixture.await( + repository, + context.getBean(JobOperator.class) + .start( + job, + new JobParametersBuilder() + .addString( + "run", "omitted-manager-control", true) + .toJobParameters()) + .getId()); + assertThat(failed.getStatus()).isEqualTo(BatchStatus.FAILED); + assertThat(rows(source)).containsExactly(1, 2, 3, 4, 5); + } + } +} diff --git a/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/OwnedRunBoundaryDatabaseTest.java b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/OwnedRunBoundaryDatabaseTest.java new file mode 100644 index 000000000..9cb33d04e --- /dev/null +++ b/modules/batch/src/test/java/org/fireflyframework/batch/jdbc/OwnedRunBoundaryDatabaseTest.java @@ -0,0 +1,274 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.batch.jdbc; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import org.fireflyframework.batch.fixture.CheckpointJobFixture; +import org.fireflyframework.batch.fixture.OwnedRunFixture; +import org.fireflyframework.batch.fixture.PostgresBatchFixture; +import org.fireflyframework.security.api.domain.Decision; +import org.fireflyframework.security.api.domain.Obligation; +import org.fireflyframework.security.core.policy.EmbeddedPolicyDecisionAdapter; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; +import org.springframework.batch.core.BatchStatus; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.support.JdbcTransactionManager; + +import reactor.core.publisher.Mono; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicReference; + +import javax.sql.DataSource; + +class OwnedRunBoundaryDatabaseTest extends PostgresBatchFixture { + private DataSource prepared() throws Exception { + DataSource source = schema("complete"); + String selected; + try (var connection = source.getConnection()) { + selected = connection.getSchema(); + } + try (var connection = owner().getConnection(); + var statement = connection.createStatement()) { + connection.setSchema(selected); + statement.execute( + "CREATE TABLE owned_run_claim(tenant varchar(100),run_id varchar(100),canonical" + + " text NOT NULL,digest varchar(64) NOT NULL,state varchar(20) NOT" + + " NULL,PRIMARY KEY(tenant,run_id));CREATE TABLE owned_run_items(tenant" + + " varchar(100),run_id varchar(100),item_no integer,PRIMARY" + + " KEY(tenant,run_id,item_no),FOREIGN KEY(tenant,run_id) REFERENCES" + + " owned_run_claim(tenant,run_id));CREATE TABLE owned_run_receipts(tenant" + + " varchar(100),run_id varchar(100),item_no integer,PRIMARY" + + " KEY(tenant,run_id,item_no),FOREIGN KEY(tenant,run_id,item_no)" + + " REFERENCES owned_run_items(tenant,run_id,item_no));GRANT" + + " SELECT,INSERT,UPDATE ON owned_run_claim TO " + + role + + ";" + + "GRANT SELECT,INSERT ON owned_run_items,owned_run_receipts TO " + + role); + } + return source; + } + + private OwnedRunFixture.Intent intent(String tenant, String model) { + return new OwnedRunFixture.Intent( + tenant, + "stable-run", + "book=fictional-book;cutoff=2026-10-22T00:00:00Z;" + + "snapshot=fixture-snapshot-1;model=" + + model + + ";policy=fixture-policy-1;currency=fixture-currency-1"); + } + + private OwnedRunFixture recipe(DataSource source, AtomicReference > current) { + var policy = + new EmbeddedPolicyDecisionAdapter( + List.of((principal, action, resource, context) -> current.get()), true); + return new OwnedRunFixture( + source, + new JdbcTransactionManager(source), + policy, + org.fireflyframework.security.api.domain.SecurityPrincipal.builder() + .subject("owned-fixture-service") + .issuer("owned-fixture-deployment") + .tenantId("owned-a") + .scopes(java.util.Set.of("owned-run:execute")) + .build()); + } + + @Test + void immutableFullIntentClaimAcceptsExactReplayAndRejectsChangedIntent() throws Exception { + DataSource source = prepared(); + var recipe = recipe(source, new AtomicReference<>(Mono.just(Decision.permit()))); + var original = intent("owned-a", "fixture-model-1"); + assertThat(recipe.claim(original)).isEqualTo("CREATED"); + assertThat(recipe.claim(original)).isEqualTo("REPLAY"); + assertThatThrownBy(() -> recipe.claim(intent("owned-a", "changed-model"))) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("intent"); + var jdbc = new JdbcTemplate(source); + assertThat(jdbc.queryForObject("SELECT canonical FROM owned_run_claim", String.class)) + .isEqualTo(original.canonical()); + assertThat(jdbc.queryForObject("SELECT count(*) FROM owned_run_claim", Integer.class)) + .isEqualTo(1); + assertThat(jdbc.queryForObject("SELECT count(*) FROM owned_run_items", Integer.class)) + .isZero(); + } + + @Test + void differentJobIntentCannotUseAnExistingOwnerClaim() throws Exception { + DataSource source = prepared(); + var recipe = recipe(source, new AtomicReference<>(Mono.just(Decision.permit()))); + recipe.claim(intent("owned-a", "fixture-model-1")); + try (var context = CheckpointJobFixture.open(source)) { + assertThatThrownBy(() -> recipe.job(context, intent("owned-a", "changed-model"), false)) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("intent"); + } + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM owned_run_items", Integer.class)) + .isZero(); + } + + @Test + void callerRunTenantCannotSelectTheCurrentServiceTenant() throws Exception { + DataSource source = prepared(); + var recipe = recipe(source, new AtomicReference<>(Mono.just(Decision.permit()))); + assertThatThrownBy(() -> recipe.claim(intent("owned-b", "fixture-model-1"))) + .isInstanceOf(IllegalStateException.class); + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM owned_run_claim", Integer.class)) + .isZero(); + } + + @Test + void restartUsesCurrentServicePolicyInsteadOfRetainingEarlierPermit() throws Exception { + DataSource source = prepared(); + var current = new AtomicReference<>(Mono.just(Decision.permit())); + var original = intent("owned-a", "fixture-model-1"); + var recipe = recipe(source, current); + recipe.claim(original); + var parameters = + new JobParametersBuilder() + .addString("tenant", original.tenant(), true) + .addString("run", original.run(), true) + .addString("intent", original.digest(), true) + .toJobParameters(); + long failedId; + try (var first = CheckpointJobFixture.open(source)) { + var repository = first.getBean(JobRepository.class); + var job = recipe.job(first, original, true); + var failed = + CheckpointJobFixture.await( + repository, + first.getBean(JobOperator.class).start(job, parameters).getId()); + failedId = failed.getId(); + assertThat(failed.getStatus()).isEqualTo(BatchStatus.FAILED); + } + assertThat( + new JdbcTemplate(source) + .queryForList( + "SELECT item_no FROM owned_run_items ORDER BY item_no", + Integer.class)) + .containsExactly(1, 2, 3); + current.set(Mono.just(Decision.deny("current fixture policy denied"))); + var restartedRecipe = recipe(source, current); + assertThatThrownBy(() -> restartedRecipe.claim(original)) + .isInstanceOf(IllegalStateException.class); + try (var second = CheckpointJobFixture.open(source)) { + var repository = second.getBean(JobRepository.class); + restartedRecipe.job(second, original, false); + var failed = + CheckpointJobFixture.await( + repository, + second.getBean(JobOperator.class) + .restart(repository.getJobExecution(failedId)) + .getId()); + assertThat(failed.getStatus()).isEqualTo(BatchStatus.FAILED); + } + assertThat( + new JdbcTemplate(source) + .queryForList( + "SELECT item_no FROM owned_run_items ORDER BY item_no", + Integer.class)) + .containsExactly(1, 2, 3); + } + + private static java.util.stream.Stream > invalidCurrentDecisions() { + return java.util.stream.Stream.of( + Mono. empty(), + Mono. error(new IllegalStateException("owned policy failure")), + Mono.just( + Decision.permit(List.of(Obligation.of("fixture-unfulfilled", Map.of()))))); + } + + @ParameterizedTest + @MethodSource("invalidCurrentDecisions") + void emptyFailedAndUnfulfilledObligationDecisionsCannotProduceItems(Mono result) + throws Exception { + DataSource source = prepared(); + var current = new AtomicReference<>(Mono.just(Decision.permit())); + var recipe = recipe(source, current); + var intent = intent("owned-a", "fixture-model-1"); + recipe.claim(intent); + current.set(result); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var job = recipe.job(context, intent, false); + var failed = + CheckpointJobFixture.await( + repository, + context.getBean(JobOperator.class) + .start( + job, + new JobParametersBuilder() + .addString("tenant", intent.tenant(), true) + .addString("run", intent.run(), true) + .addString("intent", intent.digest(), true) + .toJobParameters()) + .getId()); + assertThat(failed.getStatus()).isEqualTo(BatchStatus.FAILED); + } + assertThat( + new JdbcTemplate(source) + .queryForObject( + "SELECT count(*) FROM owned_run_items", Integer.class)) + .isZero(); + } + + @Test + void nativeCompletedMetadataCannotFinalizeOwnerRunWithoutStoredAcknowledgments() + throws Exception { + DataSource source = prepared(); + var recipe = recipe(source, new AtomicReference<>(Mono.just(Decision.permit()))); + var intent = intent("owned-a", "fixture-model-1"); + recipe.claim(intent); + try (var context = CheckpointJobFixture.open(source)) { + var repository = context.getBean(JobRepository.class); + var job = recipe.job(context, intent, false); + var completed = + CheckpointJobFixture.await( + repository, + context.getBean(JobOperator.class) + .start( + job, + new JobParametersBuilder() + .addString("tenant", intent.tenant(), true) + .addString("run", intent.run(), true) + .addString("intent", intent.digest(), true) + .toJobParameters()) + .getId()); + assertThat(completed.getStatus()).isEqualTo(BatchStatus.COMPLETED); + } + assertThat(recipe.completeOnlyWithStoredAcknowledgments(intent)).isFalse(); + assertThat( + new JdbcTemplate(source) + .queryForObject("SELECT state FROM owned_run_claim", String.class)) + .isEqualTo("OPEN"); + } +} diff --git a/modules/cache-hazelcast/README.md b/modules/cache-hazelcast/README.md index 848efe60c..0c8bad17a 100644 --- a/modules/cache-hazelcast/README.md +++ b/modules/cache-hazelcast/README.md @@ -20,7 +20,7 @@ If the onboarding deployment already supplies a Hazelcast instance, this provide ``` diff --git a/modules/cache-hazelcast/pom.xml b/modules/cache-hazelcast/pom.xml index b0d4398c3..611f49df0 100644 --- a/modules/cache-hazelcast/pom.xml +++ b/modules/cache-hazelcast/pom.xml @@ -7,12 +7,12 @@ org.fireflyframework fireflyframework-cache-hazelcast -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml fireflyframework-cache-hazelcast -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Cache - Hazelcast diff --git a/modules/cache-jcache/README.md b/modules/cache-jcache/README.md index 6ae717abd..c229d8e1d 100644 --- a/modules/cache-jcache/README.md +++ b/modules/cache-jcache/README.md @@ -20,7 +20,7 @@ Use this adapter when the onboarding host supplies a JCache CacheManager. The ap``` diff --git a/modules/cache-jcache/pom.xml b/modules/cache-jcache/pom.xml index da3e07058..a82317cda 100644 --- a/modules/cache-jcache/pom.xml +++ b/modules/cache-jcache/pom.xml @@ -7,12 +7,12 @@ org.fireflyframework fireflyframework-cache-jcache -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml fireflyframework-cache-jcache -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Cache - JCache diff --git a/modules/cache-postgres/README.md b/modules/cache-postgres/README.md index c711d5fec..37cfb2991 100644 --- a/modules/cache-postgres/README.md +++ b/modules/cache-postgres/README.md @@ -20,7 +20,7 @@ A service that already operates PostgreSQL can use a separate cache table for re``` diff --git a/modules/cache-postgres/pom.xml b/modules/cache-postgres/pom.xml index 7fb3d0c24..b098eab24 100644 --- a/modules/cache-postgres/pom.xml +++ b/modules/cache-postgres/pom.xml @@ -7,12 +7,12 @@ org.fireflyframework fireflyframework-cache-postgres -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml fireflyframework-cache-postgres -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Cache - Postgres diff --git a/modules/cache-postgres/src/main/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapter.java b/modules/cache-postgres/src/main/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapter.java index 776c572ef..7b689e620 100644 --- a/modules/cache-postgres/src/main/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapter.java +++ b/modules/cache-postgres/src/main/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapter.java @@ -16,6 +16,8 @@ package org.fireflyframework.cache.adapter.postgres; +import org.fireflyframework.kernel.logging.LogFields; + import io.r2dbc.spi.Connection; import io.r2dbc.spi.ConnectionFactory; import io.r2dbc.spi.Readable; @@ -183,8 +185,8 @@ publicMono > get(K key, Class valueType) { totalLoadTime.addAndGet(System.nanoTime() - start); return Optional.ofNullable(value); } catch (Exception e) { - log.warn("Failed to deserialize cached value for key '{}' in cache '{}': {}", - key, cacheName, e.getMessage()); + log.warn("Failed to deserialize cached value for key present={} in cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); missCount.incrementAndGet(); return Optional. empty(); } @@ -192,7 +194,7 @@ public Mono > get(K key, Class valueType) { missCount.incrementAndGet(); return Optional. empty(); })).onErrorResume(e -> { - log.warn("Error reading key '{}' from cache '{}': {}", key, cacheName, e.getMessage()); + log.warn("Error reading key present={} from cache '{}': type={}", key != null, LogFields.text(cacheName), LogFields.errorType(e)); missCount.incrementAndGet(); return Mono.just(Optional. empty()); }); @@ -295,7 +297,7 @@ public Mono evict(K key) { return removed; }) .onErrorResume(e -> { - log.warn("Error evicting key '{}' from cache '{}': {}", key, cacheName, e.getMessage()); + log.warn("Error evicting key present={} from cache '{}': type={}", key != null, LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(false); }); } @@ -333,8 +335,8 @@ public Mono exists(K key) { .next()) .defaultIfEmpty(false) .onErrorResume(e -> { - log.warn("Error checking existence of key '{}' in cache '{}': {}", - key, cacheName, e.getMessage()); + log.warn("Error checking existence of key present={} in cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(false); }); } @@ -434,7 +436,7 @@ public Mono getHealth() { public void close() { // The ConnectionFactory/ConnectionPool is Spring-managed (disposed via the // bean destroy method); nothing to close here. - log.info("Closing PostgreSQL cache adapter '{}'", cacheName); + log.info("Closing PostgreSQL cache adapter '{}'", LogFields.text(cacheName)); } private Mono rowsUpdated(Statement statement) { diff --git a/modules/cache-postgres/src/test/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapterLoggingTest.java b/modules/cache-postgres/src/test/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapterLoggingTest.java new file mode 100644 index 000000000..0b32eef41 --- /dev/null +++ b/modules/cache-postgres/src/test/java/org/fireflyframework/cache/adapter/postgres/PostgresCacheAdapterLoggingTest.java @@ -0,0 +1,81 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.cache.adapter.postgres; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; +import reactor.core.publisher.Mono; +import java.util.Optional; +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +class PostgresCacheAdapterLoggingTest { + private static final String NAME = "cache\r\nregion\u000bvertical\fform\u0085next\u2028line\u2029paragraph"; + private final Logger logger = (Logger) LoggerFactory.getLogger(PostgresCacheAdapter.class); + private final ListAppender events = new ListAppender<>(); + private Level previousLevel; + private boolean previousAdditive; + + @BeforeEach void capture() { + previousLevel = logger.getLevel(); previousAdditive = logger.isAdditive(); + logger.setLevel(Level.DEBUG); logger.setAdditive(false); events.start(); logger.addAppender(events); + } + @AfterEach void release() { + logger.detachAppender(events); events.stop(); logger.setLevel(previousLevel); logger.setAdditive(previousAdditive); + } + private void safeEvent(Level expected, boolean error) { + assertFalse(events.list.isEmpty()); + var event = events.list.getLast(); String message = event.getFormattedMessage(); + assertEquals(expected, event.getLevel()); assertNull(event.getThrowableProxy()); + assertFalse(message.contains("PRIVATE_"), message); + assertFalse(message.matches("(?s).*[\r\n\u000b\f\u0085\u2028\u2029].*"), message); + if (error) assertTrue(message.contains("type=java.lang.IllegalStateException"), message); + } + private static final class Key { + int renders; + @Override public String toString() { renders++; return "PRIVATE_KEY\r\nforged\u2028entry"; } + } + + @Test void readEvictAndExistsErrorsKeepTheirFallbacksAndTypeOnlyWarnings() { + var factory = mock(io.r2dbc.spi.ConnectionFactory.class); var failure = new IllegalStateException("PRIVATE_MESSAGE\r\nforged"); + when(factory.create()).thenAnswer(invocation -> Mono.error(failure)); + var adapter = new PostgresCacheAdapter(NAME, factory, PostgresCacheConfig.builder().build(), mock(org.fireflyframework.cache.serialization.CacheSerializer.class)); + var key = new Key(); assertEquals(Optional.empty(), adapter.get(key).block()); safeEvent(Level.WARN, true); assertEquals(1,key.renders); + assertEquals(false, adapter.evict(key).block()); safeEvent(Level.WARN, true); assertEquals(2,key.renders); + assertEquals(false, adapter.exists(key).block()); safeEvent(Level.WARN, true); assertEquals(3,key.renders); + adapter.close(); safeEvent(Level.INFO, false); + } + @Test @SuppressWarnings({"rawtypes","unchecked"}) void deserializerFailureRetainsEmptyReadWithoutRenderingKeyAgain() { + var factory = mock(io.r2dbc.spi.ConnectionFactory.class); var connection = mock(io.r2dbc.spi.Connection.class); + var statement = mock(io.r2dbc.spi.Statement.class, RETURNS_SELF); var result = mock(io.r2dbc.spi.Result.class); var row = mock(io.r2dbc.spi.Row.class); + when(factory.create()).thenAnswer(invocation -> Mono.just(connection)); when(connection.close()).thenReturn(Mono.empty()); + when(connection.createStatement(anyString())).thenReturn(statement); when(statement.execute()).thenAnswer(invocation -> reactor.core.publisher.Flux.just(result)); + when(row.get("value")).thenReturn("serialized".getBytes(java.nio.charset.StandardCharsets.UTF_8)); + when(result.map(any(java.util.function.BiFunction.class))).thenAnswer(invocation -> reactor.core.publisher.Flux.just(((java.util.function.BiFunction)invocation.getArgument(0)).apply(row,mock(io.r2dbc.spi.RowMetadata.class)))); + var serializer = mock(org.fireflyframework.cache.serialization.CacheSerializer.class); + when(serializer.deserialize(any(),any())).thenThrow(new IllegalStateException("PRIVATE_MESSAGE\r\nforged")); + var adapter = new PostgresCacheAdapter(NAME,factory,PostgresCacheConfig.builder().build(),serializer); var key = new Key(); + assertEquals(Optional.empty(),adapter.get(key,String.class).block()); safeEvent(Level.WARN,true); assertEquals(1,key.renders); + verify(connection).close(); + } +} diff --git a/modules/cache-redis/README.md b/modules/cache-redis/README.md index 004f412c1..89e1d95ca 100644 --- a/modules/cache-redis/README.md +++ b/modules/cache-redis/README.md @@ -20,7 +20,7 @@ Use a Redis-backed cache when several onboarding-service instances need the same ``` diff --git a/modules/cache-redis/pom.xml b/modules/cache-redis/pom.xml index 5cdc91985..89c0bcf3a 100644 --- a/modules/cache-redis/pom.xml +++ b/modules/cache-redis/pom.xml @@ -7,12 +7,12 @@ org.fireflyframework fireflyframework-cache-redis -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml fireflyframework-cache-redis -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Cache - Redis diff --git a/modules/cache-redis/src/main/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapter.java b/modules/cache-redis/src/main/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapter.java index 76b36be1c..b7adf2e7b 100644 --- a/modules/cache-redis/src/main/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapter.java +++ b/modules/cache-redis/src/main/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapter.java @@ -16,6 +16,8 @@ package org.fireflyframework.cache.adapter.redis; +import org.fireflyframework.kernel.logging.LogFields; + import org.fireflyframework.cache.core.CacheAdapter; import org.fireflyframework.cache.core.CacheHealth; import org.fireflyframework.cache.core.CacheStats; @@ -107,18 +109,18 @@ publicMono > get(K key) { V deserializedValue = serializer.deserialize(value, (Class ) Object.class); hitCount.incrementAndGet(); totalLoadTime.addAndGet(System.nanoTime() - startTime); - log.debug("Cache hit for key '{}' in cache '{}'", key, cacheName); + log.debug("Cache hit for key present={} in cache '{}'", key != null, LogFields.text(cacheName)); return Optional.of(deserializedValue); } catch (Exception e) { - log.warn("Failed to deserialize value for key '{}' in cache '{}': {}", - key, cacheName, e.getMessage()); + log.warn("Failed to deserialize value for key present={} in cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); missCount.incrementAndGet(); return Optional. empty(); } }) .switchIfEmpty(Mono.fromCallable(() -> { missCount.incrementAndGet(); - log.debug("Cache miss for key '{}' in cache '{}'", key, cacheName); + log.debug("Cache miss for key present={} in cache '{}'", key != null, LogFields.text(cacheName)); return Optional. empty(); })); }).flatMap(mono -> mono); @@ -138,18 +140,18 @@ public Mono > get(K key, Class valueType) { V deserializedValue = serializer.deserialize(value, valueType); hitCount.incrementAndGet(); totalLoadTime.addAndGet(System.nanoTime() - startTime); - log.debug("Cache hit for key '{}' in cache '{}'", key, cacheName); + log.debug("Cache hit for key present={} in cache '{}'", key != null, LogFields.text(cacheName)); return Optional.of(deserializedValue); } catch (Exception e) { - log.warn("Failed to deserialize value for key '{}' in cache '{}': {}", - key, cacheName, e.getMessage()); + log.warn("Failed to deserialize value for key present={} in cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); missCount.incrementAndGet(); return Optional. empty(); } }) .switchIfEmpty(Mono.fromCallable(() -> { missCount.incrementAndGet(); - log.debug("Cache miss for key '{}' in cache '{}'", key, cacheName); + log.debug("Cache miss for key present={} in cache '{}'", key != null, LogFields.text(cacheName)); return Optional. empty(); })); }).flatMap(mono -> mono); @@ -175,11 +177,11 @@ public Mono put(K key, V value) { return operation.doOnSuccess(unused -> { putCount.incrementAndGet(); - log.debug("Put value in cache '{}' for key '{}'", cacheName, key); + log.debug("Put value in cache '{}' for key present={}", LogFields.text(cacheName), key != null); }); } catch (Exception e) { - log.error("Error putting value in cache '{}' for key '{}': {}", - cacheName, key, e.getMessage(), e); + log.error("Error putting value in cache '{}' for key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value in cache", e); } }).flatMap(mono -> mono); @@ -197,11 +199,11 @@ public Mono put(K key, V value, Duration ttl) { .then() .doOnSuccess(unused -> { putCount.incrementAndGet(); - log.debug("Put value in cache '{}' for key '{}' with TTL {}", cacheName, key, ttl); + log.debug("Put value in cache '{}' for key present={} with TTL {}", LogFields.text(cacheName), key != null, ttl); }); } catch (Exception e) { - log.error("Error putting value in cache '{}' for key '{}' with TTL: {}", - cacheName, key, e.getMessage(), e); + log.error("Error putting value in cache '{}' for key present={} with TTL: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value in cache with TTL", e); } }).flatMap(mono -> mono); @@ -226,14 +228,14 @@ public Mono putIfAbsent(K key, V value) { return operation.doOnNext(success -> { if (success) { putCount.incrementAndGet(); - log.debug("Put new value in cache '{}' for key '{}'", cacheName, key); + log.debug("Put new value in cache '{}' for key present={}", LogFields.text(cacheName), key != null); } else { - log.debug("Key '{}' already exists in cache '{}'", key, cacheName); + log.debug("Key present={} already exists in cache '{}'", key != null, LogFields.text(cacheName)); } }); } catch (Exception e) { - log.error("Error in putIfAbsent for cache '{}' and key '{}': {}", - cacheName, key, e.getMessage(), e); + log.error("Error in putIfAbsent for cache '{}' and key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value if absent", e); } }).flatMap(mono -> mono); @@ -251,15 +253,15 @@ public Mono putIfAbsent(K key, V value, Duration ttl) { .doOnNext(success -> { if (success) { putCount.incrementAndGet(); - log.debug("Put new value in cache '{}' for key '{}' with TTL {}", - cacheName, key, ttl); + log.debug("Put new value in cache '{}' for key present={} with TTL {}", + LogFields.text(cacheName), key != null, ttl); } else { - log.debug("Key '{}' already exists in cache '{}'", key, cacheName); + log.debug("Key present={} already exists in cache '{}'", key != null, LogFields.text(cacheName)); } }); } catch (Exception e) { - log.error("Error in putIfAbsent with TTL for cache '{}' and key '{}': {}", - cacheName, key, e.getMessage(), e); + log.error("Error in putIfAbsent with TTL for cache '{}' and key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value if absent with TTL", e); } }).flatMap(mono -> mono); @@ -273,12 +275,12 @@ public Mono evict(K key) { if (existed) { evictionCount.incrementAndGet(); } - log.debug("Evicted key '{}' from cache '{}': {}", key, cacheName, existed); + log.debug("Evicted key present={} from cache '{}': {}", key != null, LogFields.text(cacheName), existed); return existed; }) .onErrorResume(e -> { - log.error("Error evicting key '{}' from cache '{}': {}", - key, cacheName, e.getMessage(), e); + log.error("Error evicting key present={} from cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(false); }); } @@ -294,11 +296,11 @@ public Mono clear() { }) .doOnNext(deletedCount -> { evictionCount.addAndGet(deletedCount); - log.debug("Cleared cache '{}', deleted {} keys", cacheName, deletedCount); + log.debug("Cleared cache '{}', deleted {} keys", LogFields.text(cacheName), deletedCount); }) .then() .onErrorResume(e -> { - log.error("Error clearing cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error clearing cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); return Mono.empty(); }); } @@ -307,8 +309,8 @@ public Mono clear() { public Mono exists(K key) { return redisTemplate.hasKey(buildKey(key)) .onErrorResume(e -> { - log.warn("Error checking existence of key '{}' in cache '{}': {}", - key, cacheName, e.getMessage()); + log.warn("Error checking existence of key present={} in cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(false); }); } @@ -322,7 +324,7 @@ public Mono > keys() { .map(key -> (K) key) .collect(Collectors.toSet())) .onErrorResume(e -> { - log.error("Error getting keys from cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error getting keys from cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(Set.of()); }); } @@ -332,7 +334,7 @@ public Mono size() { return getKeys() .map(keys -> (long) keys.size()) .onErrorResume(e -> { - log.error("Error getting size of cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error getting size of cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(0L); }); } @@ -435,8 +437,8 @@ public Mono getHealth() { }) .onErrorResume(infoError -> { // If we can't get info, still return healthy but without details - log.warn("Could not get Redis info for cache '{}': {}", - cacheName, infoError.getMessage()); + log.warn("Could not get Redis info for cache '{}': type={}", + LogFields.text(cacheName), LogFields.errorType(infoError)); return Mono.just(CacheHealth.builder() .status("UP") .cacheType(CacheType.REDIS) @@ -456,7 +458,7 @@ public Mono getHealth() { }); }) .onErrorResume(e -> { - log.error("Health check failed for cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Health check failed for cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); return Mono.just(CacheHealth.unhealthy(CacheType.REDIS, cacheName, "Health check failed: " + e.getMessage(), e)); }); @@ -466,10 +468,10 @@ public Mono getHealth() { @Override public void close() { try { - log.info("Closing Redis cache adapter '{}'", cacheName); + log.info("Closing Redis cache adapter '{}'", LogFields.text(cacheName)); // Connection factory is typically managed by Spring, so we don't close it here } catch (Exception e) { - log.error("Error closing cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error closing cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); } } diff --git a/modules/cache-redis/src/test/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapterLoggingTest.java b/modules/cache-redis/src/test/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapterLoggingTest.java new file mode 100644 index 000000000..b9c1c9beb --- /dev/null +++ b/modules/cache-redis/src/test/java/org/fireflyframework/cache/adapter/redis/RedisCacheAdapterLoggingTest.java @@ -0,0 +1,98 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.cache.adapter.redis; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; +import reactor.core.publisher.Mono; +import java.util.Optional; +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +class RedisCacheAdapterLoggingTest { + private static final String NAME = "cache\r\nregion\u000bvertical\fform\u0085next\u2028line\u2029paragraph"; + private final Logger logger = (Logger) LoggerFactory.getLogger(RedisCacheAdapter.class); + private final ListAppender events = new ListAppender<>(); + private Level previousLevel; + private boolean previousAdditive; + + @BeforeEach void capture() { + previousLevel = logger.getLevel(); previousAdditive = logger.isAdditive(); + logger.setLevel(Level.DEBUG); logger.setAdditive(false); events.start(); logger.addAppender(events); + } + @AfterEach void release() { + logger.detachAppender(events); events.stop(); logger.setLevel(previousLevel); logger.setAdditive(previousAdditive); + } + private void safeEvent(Level expected, boolean error) { + assertFalse(events.list.isEmpty()); + var event = events.list.getLast(); String message = event.getFormattedMessage(); + assertEquals(expected, event.getLevel()); assertNull(event.getThrowableProxy()); + assertFalse(message.contains("PRIVATE_"), message); + assertFalse(message.matches("(?s).*[\r\n\u000b\f\u0085\u2028\u2029].*"), message); + if (error) assertTrue(message.contains("type=java.lang.IllegalStateException"), message); + } + private static final class Key { + int renders; + @Override public String toString() { renders++; return "PRIVATE_KEY\r\nforged\u2028entry"; } + } + + private record Fixture(RedisCacheAdapter adapter, org.springframework.data.redis.core.ReactiveValueOperations values, org.fireflyframework.cache.serialization.CacheSerializer serializer) { } + @SuppressWarnings("unchecked") private Fixture fixture() { + var template = mock(org.springframework.data.redis.core.ReactiveRedisTemplate.class); + var values = mock(org.springframework.data.redis.core.ReactiveValueOperations.class); + var serializer = mock(org.fireflyframework.cache.serialization.CacheSerializer.class); + when(template.opsForValue()).thenReturn(values); + return new Fixture(new RedisCacheAdapter(NAME, template, mock(org.springframework.data.redis.connection.ReactiveRedisConnectionFactory.class), RedisCacheConfig.builder().build(), serializer), values, serializer); + } + @Test void bothReadOverloadsKeepHitsWithoutExtraKeyRendering() { + for (boolean typed : new boolean[]{false,true}) { + events.list.clear(); var f = fixture(); var key = new Key(); + when(f.values().get(anyString())).thenReturn(Mono.just("serialized")); + when(f.serializer().deserialize(any(), any())).thenReturn("value"); + assertEquals(Optional.of("value"), typed ? f.adapter().get(key, String.class).block() : f.adapter().get(key).block()); + safeEvent(Level.DEBUG, false); assertEquals(1, key.renders); + assertTrue(events.list.getLast().getFormattedMessage().contains("cache\\r\\nregion")); + } + } + @Test void bothReadOverloadsKeepMissesWithoutExtraKeyRendering() { + for (boolean typed : new boolean[]{false,true}) { + events.list.clear(); var f = fixture(); var key = new Key(); when(f.values().get(anyString())).thenReturn(Mono.empty()); + assertEquals(Optional.empty(), typed ? f.adapter().get(key, String.class).block() : f.adapter().get(key).block()); + safeEvent(Level.DEBUG, false); assertEquals(1, key.renders); + } + } + @Test void bothDeserializerFailuresKeepEmptyFallbackAndTypeOnlyWarnings() { + for (boolean typed : new boolean[]{false,true}) { + events.list.clear(); var f = fixture(); var key = new Key(); when(f.values().get(anyString())).thenReturn(Mono.just("serialized")); + when(f.serializer().deserialize(any(), any())).thenThrow(new IllegalStateException("PRIVATE_MESSAGE\r\nforged")); + assertEquals(Optional.empty(), typed ? f.adapter().get(key, String.class).block() : f.adapter().get(key).block()); + safeEvent(Level.WARN, true); assertEquals(1, key.renders); + } + } + @Test void serializationFailureRetainsWrappedCauseWithoutRenderingKey() { + var f = fixture(); var key = new Key(); var failure = new IllegalStateException("PRIVATE_MESSAGE\u2028forged"); + when(f.serializer().serialize(any())).thenThrow(failure); + assertSame(failure, assertThrows(RuntimeException.class, () -> f.adapter().put(key, "value").block()).getCause()); + safeEvent(Level.ERROR, true); assertEquals(0, key.renders); + } +} diff --git a/modules/cache/README.md b/modules/cache/README.md index 9a8978397..4cfc2fb9a 100644 --- a/modules/cache/README.md +++ b/modules/cache/README.md @@ -20,7 +20,7 @@ A reviewer portal may read the same application status repeatedly. Cache those r ``` diff --git a/modules/cache/docs/ARCHITECTURE.md b/modules/cache/docs/ARCHITECTURE.md index 5c2ab2720..58f3fb969 100644 --- a/modules/cache/docs/ARCHITECTURE.md +++ b/modules/cache/docs/ARCHITECTURE.md @@ -137,7 +137,7 @@ For Caffeine only (in-memory cache): org.fireflyframework fireflyframework-cache -26.10.21-SNAPSHOT +26.10.22 ``` @@ -146,7 +146,7 @@ For Caffeine + Redis (distributed cache): org.fireflyframework fireflyframework-cache -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework fireflyframework-cache -26.10.21-SNAPSHOT +26.10.22 org.springframework.boot diff --git a/modules/cache/docs/AUTO_CONFIGURATION.md b/modules/cache/docs/AUTO_CONFIGURATION.md index dd75d5ae6..4591112f4 100644 --- a/modules/cache/docs/AUTO_CONFIGURATION.md +++ b/modules/cache/docs/AUTO_CONFIGURATION.md @@ -290,7 +290,7 @@ Just add the dependency - no configuration needed:``` diff --git a/modules/cache/docs/OPTIONAL_DEPENDENCIES.md b/modules/cache/docs/OPTIONAL_DEPENDENCIES.md index c1c18dccf..2e78d28ec 100644 --- a/modules/cache/docs/OPTIONAL_DEPENDENCIES.md +++ b/modules/cache/docs/OPTIONAL_DEPENDENCIES.md @@ -48,7 +48,7 @@ Maven Dependencies: org.fireflyframework fireflyframework-cache -26.10.21-SNAPSHOT +26.10.22 ``` @@ -76,7 +76,7 @@ Maven Dependencies: org.fireflyframework fireflyframework-cache -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/cache/pom.xml b/modules/cache/pom.xml index 7d8bfc0ea..923447ac2 100644 --- a/modules/cache/pom.xml +++ b/modules/cache/pom.xml @@ -7,7 +7,7 @@ org.fireflyframework fireflyframework-cache -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/cache/src/main/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapter.java b/modules/cache/src/main/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapter.java index 55b33d5e6..d9f5b07bc 100644 --- a/modules/cache/src/main/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapter.java +++ b/modules/cache/src/main/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapter.java @@ -16,6 +16,8 @@ package org.fireflyframework.cache.adapter.caffeine; +import org.fireflyframework.kernel.logging.LogFields; + import org.fireflyframework.cache.core.CacheAdapter; import org.fireflyframework.cache.core.CacheHealth; import org.fireflyframework.cache.core.CacheStats; @@ -129,8 +131,8 @@ public org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml Mono > get(K key) { V value = (V) cache.getIfPresent(prefixedKey); return Optional.ofNullable(value); } catch (Exception e) { - log.warn("Error getting value from cache '{}' for key '{}': {}", - cacheName, key, e.getMessage()); + log.warn("Error getting value from cache '{}' for key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); return Optional. empty(); } }); @@ -152,10 +154,10 @@ public Mono put(K key, V value) { cache.put(prefixedKey, value); // Remove any custom expiration time expirationTimes.remove(prefixedKey); - log.debug("Put value in cache '{}' for key '{}'", cacheName, key); + log.debug("Put value in cache '{}' for key present={}", LogFields.text(cacheName), key != null); } catch (Exception e) { - log.error("Error putting value in cache '{}' for key '{}': {}", - cacheName, key, e.getMessage(), e); + log.error("Error putting value in cache '{}' for key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value in cache", e); } }); @@ -169,10 +171,10 @@ public Mono put(K key, V value, Duration ttl) { cache.put(prefixedKey, value); // Track custom expiration time expirationTimes.put(prefixedKey, Instant.now().plus(ttl)); - log.debug("Put value in cache '{}' for key '{}' with TTL {}", cacheName, key, ttl); + log.debug("Put value in cache '{}' for key present={} with TTL {}", LogFields.text(cacheName), key != null, ttl); } catch (Exception e) { - log.error("Error putting value in cache '{}' for key '{}' with TTL: {}", - cacheName, key, e.getMessage(), e); + log.error("Error putting value in cache '{}' for key present={} with TTL: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value in cache with TTL", e); } }); @@ -196,15 +198,15 @@ public Mono putIfAbsent(K key, V value) { if (existingValue == null) { cache.put(prefixedKey, value); expirationTimes.remove(prefixedKey); // No custom TTL - log.debug("Put new value in cache '{}' for key '{}'", cacheName, key); + log.debug("Put new value in cache '{}' for key present={}", LogFields.text(cacheName), key != null); return true; } else { - log.debug("Key '{}' already exists in cache '{}'", key, cacheName); + log.debug("Key present={} already exists in cache '{}'", key != null, LogFields.text(cacheName)); return false; } } catch (Exception e) { - log.error("Error in putIfAbsent for cache '{}' and key '{}': {}", - cacheName, key, e.getMessage(), e); + log.error("Error in putIfAbsent for cache '{}' and key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value if absent", e); } }); @@ -228,16 +230,16 @@ public Mono putIfAbsent(K key, V value, Duration ttl) { if (existingValue == null) { cache.put(prefixedKey, value); expirationTimes.put(prefixedKey, Instant.now().plus(ttl)); - log.debug("Put new value in cache '{}' for key '{}' with TTL {}", - cacheName, key, ttl); + log.debug("Put new value in cache '{}' for key present={} with TTL {}", + LogFields.text(cacheName), key != null, ttl); return true; } else { - log.debug("Key '{}' already exists in cache '{}'", key, cacheName); + log.debug("Key present={} already exists in cache '{}'", key != null, LogFields.text(cacheName)); return false; } } catch (Exception e) { - log.error("Error in putIfAbsent with TTL for cache '{}' and key '{}': {}", - cacheName, key, e.getMessage(), e); + log.error("Error in putIfAbsent with TTL for cache '{}' and key present={}: type={}", + LogFields.text(cacheName), key != null, LogFields.errorType(e)); throw new RuntimeException("Failed to put value if absent with TTL", e); } }); @@ -251,11 +253,11 @@ public Mono evict(K key) { boolean existed = cache.getIfPresent(prefixedKey) != null || expirationTimes.containsKey(prefixedKey); cache.invalidate(prefixedKey); expirationTimes.remove(prefixedKey); - log.debug("Evicted key '{}' from cache '{}': {}", key, cacheName, existed); + log.debug("Evicted key present={} from cache '{}': {}", key != null, LogFields.text(cacheName), existed); return existed; } catch (Exception e) { - log.error("Error evicting key '{}' from cache '{}': {}", - key, cacheName, e.getMessage(), e); + log.error("Error evicting key present={} from cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); throw new RuntimeException("Failed to evict key from cache", e); } }); @@ -267,9 +269,9 @@ public Mono clear() { try { cache.invalidateAll(); expirationTimes.clear(); - log.debug("Cleared cache '{}'", cacheName); + log.debug("Cleared cache '{}'", LogFields.text(cacheName)); } catch (Exception e) { - log.error("Error clearing cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error clearing cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); throw new RuntimeException("Failed to clear cache", e); } }); @@ -291,8 +293,8 @@ public Mono exists(K key) { return cache.getIfPresent(prefixedKey) != null; } catch (Exception e) { - log.warn("Error checking existence of key '{}' in cache '{}': {}", - key, cacheName, e.getMessage()); + log.warn("Error checking existence of key present={} in cache '{}': type={}", + key != null, LogFields.text(cacheName), LogFields.errorType(e)); return false; } }); @@ -313,7 +315,7 @@ public Mono > keys() { return unprefixedKeys; } catch (Exception e) { - log.error("Error getting keys from cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error getting keys from cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); throw new RuntimeException("Failed to get cache keys", e); } }); @@ -327,7 +329,7 @@ public Mono size() { cleanupExpiredKeys(); return cache.estimatedSize(); } catch (Exception e) { - log.error("Error getting size of cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error getting size of cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); throw new RuntimeException("Failed to get cache size", e); } }); @@ -353,7 +355,7 @@ public Mono getStats() { .cacheName(cacheName) .build(); } catch (Exception e) { - log.error("Error getting stats for cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error getting stats for cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); return CacheStats.empty(CacheType.CAFFEINE, cacheName); } }); @@ -395,7 +397,7 @@ public Mono getHealth() { "Health check failed: test value mismatch", null); } } catch (Exception e) { - log.error("Health check failed for cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Health check failed for cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); return CacheHealth.unhealthy(CacheType.CAFFEINE, cacheName, "Health check failed: " + e.getMessage(), e); } @@ -408,9 +410,9 @@ public void close() { cache.invalidateAll(); cache.cleanUp(); expirationTimes.clear(); - log.info("Closed Caffeine cache adapter '{}'", cacheName); + log.info("Closed Caffeine cache adapter '{}'", LogFields.text(cacheName)); } catch (Exception e) { - log.error("Error closing cache '{}': {}", cacheName, e.getMessage(), e); + log.error("Error closing cache '{}': type={}", LogFields.text(cacheName), LogFields.errorType(e)); } } diff --git a/modules/cache/src/main/java/org/fireflyframework/cache/manager/FireflyCacheManager.java b/modules/cache/src/main/java/org/fireflyframework/cache/manager/FireflyCacheManager.java index 8233562ee..3557b2302 100644 --- a/modules/cache/src/main/java/org/fireflyframework/cache/manager/FireflyCacheManager.java +++ b/modules/cache/src/main/java/org/fireflyframework/cache/manager/FireflyCacheManager.java @@ -16,6 +16,8 @@ package org.fireflyframework.cache.manager; +import org.fireflyframework.kernel.logging.LogFields; + import org.fireflyframework.cache.core.CacheAdapter; import org.fireflyframework.cache.core.CacheHealth; import org.fireflyframework.cache.core.CacheStats; @@ -79,7 +81,7 @@ public FireflyCacheManager(CacheAdapter primaryCache, CacheAdapter fallbackCache public Mono > get(K key) { return getActiveCache(). get(key) .onErrorResume(e -> { - log.error("Error during get operation for key {}: {}", key, e.getMessage()); + log.error("Error during get operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.just(Optional.empty()); }); } @@ -88,7 +90,7 @@ public Mono > get(K key) { public Mono > get(K key, Class valueType) { return getActiveCache(). get(key, valueType) .onErrorResume(e -> { - log.error("Error during get operation for key {}: {}", key, e.getMessage()); + log.error("Error during get operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.just(Optional.empty()); }); } @@ -97,7 +99,7 @@ public Mono > get(K key, Class valueType) { public Mono put(K key, V value) { return getActiveCache(). put(key, value) .onErrorResume(e -> { - log.error("Error during put operation for key {}: {}", key, e.getMessage()); + log.error("Error during put operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.empty(); }); } @@ -106,7 +108,7 @@ public Mono put(K key, V value) { public Mono put(K key, V value, Duration ttl) { return getActiveCache(). put(key, value, ttl) .onErrorResume(e -> { - log.error("Error during put operation for key {}: {}", key, e.getMessage()); + log.error("Error during put operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.empty(); }); } @@ -115,7 +117,7 @@ public Mono put(K key, V value, Duration ttl) { public Mono putIfAbsent(K key, V value) { return getActiveCache(). putIfAbsent(key, value) .onErrorResume(e -> { - log.error("Error during putIfAbsent operation for key {}: {}", key, e.getMessage()); + log.error("Error during putIfAbsent operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.just(false); }); } @@ -124,7 +126,7 @@ public Mono putIfAbsent(K key, V value) { public Mono putIfAbsent(K key, V value, Duration ttl) { return getActiveCache(). putIfAbsent(key, value, ttl) .onErrorResume(e -> { - log.error("Error during putIfAbsent operation for key {}: {}", key, e.getMessage()); + log.error("Error during putIfAbsent operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.just(false); }); } @@ -133,7 +135,7 @@ public Mono putIfAbsent(K key, V value, Duration ttl) { public Mono evict(K key) { return getActiveCache(). evict(key) .onErrorResume(e -> { - log.error("Error during evict operation for key {}: {}", key, e.getMessage()); + log.error("Error during evict operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.just(false); }); } @@ -142,7 +144,7 @@ public Mono evict(K key) { public Mono clear() { return getActiveCache().clear() .onErrorResume(e -> { - log.error("Error clearing cache: {}", e.getMessage(), e); + log.error("Error clearing cache: type={}", LogFields.errorType(e)); return Mono.empty(); }); } @@ -151,7 +153,7 @@ public Mono clear() { public Mono exists(K key) { return getActiveCache(). exists(key) .onErrorResume(e -> { - log.error("Error during exists operation for key {}: {}", key, e.getMessage()); + log.error("Error during exists operation for key present={}: type={}", key != null, LogFields.errorType(e)); return Mono.just(false); }); } @@ -160,7 +162,7 @@ public Mono exists(K key) { public Mono > keys() { return getActiveCache(). keys() .onErrorResume(e -> { - log.error("Error getting keys: {}", e.getMessage(), e); + log.error("Error getting keys: type={}", LogFields.errorType(e)); return Mono.just(Set.of()); }); } @@ -169,7 +171,7 @@ public Mono > keys() { public Mono size() { return getActiveCache().size() .onErrorResume(e -> { - log.error("Error getting size: {}", e.getMessage(), e); + log.error("Error getting size: type={}", LogFields.errorType(e)); return Mono.just(0L); }); } @@ -212,7 +214,7 @@ public void close() { primaryCache.close(); log.debug("Closed primary cache"); } catch (Exception e) { - log.error("Error closing primary cache: {}", e.getMessage(), e); + log.error("Error closing primary cache: type={}", LogFields.errorType(e)); } if (fallbackCache != null) { @@ -220,7 +222,7 @@ public void close() { fallbackCache.close(); log.debug("Closed fallback cache"); } catch (Exception e) { - log.error("Error closing fallback cache: {}", e.getMessage(), e); + log.error("Error closing fallback cache: type={}", LogFields.errorType(e)); } } diff --git a/modules/cache/src/test/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapterLoggingTest.java b/modules/cache/src/test/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapterLoggingTest.java new file mode 100644 index 000000000..d9ba17717 --- /dev/null +++ b/modules/cache/src/test/java/org/fireflyframework/cache/adapter/caffeine/CaffeineCacheAdapterLoggingTest.java @@ -0,0 +1,82 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.cache.adapter.caffeine; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; +import reactor.core.publisher.Mono; +import java.util.Optional; +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +class CaffeineCacheAdapterLoggingTest { + private static final String NAME = "cache\r\nregion\u000bvertical\fform\u0085next\u2028line\u2029paragraph"; + private final Logger logger = (Logger) LoggerFactory.getLogger(CaffeineCacheAdapter.class); + private final ListAppender events = new ListAppender<>(); + private Level previousLevel; + private boolean previousAdditive; + + @BeforeEach void capture() { + previousLevel = logger.getLevel(); previousAdditive = logger.isAdditive(); + logger.setLevel(Level.DEBUG); logger.setAdditive(false); events.start(); logger.addAppender(events); + } + @AfterEach void release() { + logger.detachAppender(events); events.stop(); logger.setLevel(previousLevel); logger.setAdditive(previousAdditive); + } + private void safeEvent(Level expected, boolean error) { + assertFalse(events.list.isEmpty()); + var event = events.list.getLast(); String message = event.getFormattedMessage(); + assertEquals(expected, event.getLevel()); assertNull(event.getThrowableProxy()); + assertFalse(message.contains("PRIVATE_"), message); + assertFalse(message.matches("(?s).*[\r\n\u000b\f\u0085\u2028\u2029].*"), message); + if (error) assertTrue(message.contains("type=java.lang.IllegalStateException"), message); + } + private static final class Key { + int renders; + @Override public String toString() { renders++; return "PRIVATE_KEY\r\nforged\u2028entry"; } + } + + private CaffeineCacheAdapter adapter() { return new CaffeineCacheAdapter(NAME, CaffeineCacheConfig.builder().maximumSize(10L).build()); } + @Test void successfulOperationsLogKeyPresenceAndEscapeOnlyTheCacheName() { + var cache = adapter(); var key = new Key(); cache.put(key, "value").block(); + safeEvent(Level.DEBUG, false); assertEquals(1, key.renders); + assertTrue(events.list.getLast().getFormattedMessage().contains("cache\\r\\nregion")); + assertEquals(Optional.of("value"), cache.get(key).block()); + cache.close(); safeEvent(Level.INFO, false); + } + @Test void keyFailureKeepsEmptyReadAndOneBusinessRenderingWithoutErrorPayload() { + var renders = new java.util.concurrent.atomic.AtomicInteger(); + Object key = new Object() { @Override public String toString() { if (renders.incrementAndGet() == 1) throw new IllegalStateException("PRIVATE_MESSAGE\r\nforged"); return "PRIVATE_KEY"; } }; + assertEquals(Optional.empty(), adapter().get(key).block()); safeEvent(Level.WARN, true); assertEquals(1, renders.get()); + } + @Test void putFailurePreservesOriginalWrappedCauseWithoutLoggingIt() { + var failure = new IllegalStateException("PRIVATE_MESSAGE\u2028forged"); + Object key = new Object() { @Override public String toString() { throw failure; } }; + var error = assertThrows(RuntimeException.class, () -> adapter().put(key, "value").block()); + assertSame(failure, error.getCause()); safeEvent(Level.ERROR, true); + } + @Test void disabledLoggingAddsNoGenericKeyRendering() { + logger.setLevel(Level.OFF); var key = new Key(); adapter().put(key, "value").block(); + assertEquals(1, key.renders); assertTrue(events.list.isEmpty()); + } +} diff --git a/modules/cache/src/test/java/org/fireflyframework/cache/manager/FireflyCacheManagerLoggingTest.java b/modules/cache/src/test/java/org/fireflyframework/cache/manager/FireflyCacheManagerLoggingTest.java new file mode 100644 index 000000000..60b5c03e6 --- /dev/null +++ b/modules/cache/src/test/java/org/fireflyframework/cache/manager/FireflyCacheManagerLoggingTest.java @@ -0,0 +1,96 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.fireflyframework.cache.manager; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; +import reactor.core.publisher.Mono; +import java.util.Optional; +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +class FireflyCacheManagerLoggingTest { + private static final String NAME = "cache\r\nregion\u000bvertical\fform\u0085next\u2028line\u2029paragraph"; + private final Logger logger = (Logger) LoggerFactory.getLogger(FireflyCacheManager.class); + private final ListAppender events = new ListAppender<>(); + private Level previousLevel; + private boolean previousAdditive; + + @BeforeEach void capture() { + previousLevel = logger.getLevel(); previousAdditive = logger.isAdditive(); + logger.setLevel(Level.DEBUG); logger.setAdditive(false); events.start(); logger.addAppender(events); + } + @AfterEach void release() { + logger.detachAppender(events); events.stop(); logger.setLevel(previousLevel); logger.setAdditive(previousAdditive); + } + private void safeEvent(Level expected, boolean error) { + assertFalse(events.list.isEmpty()); + var event = events.list.getLast(); String message = event.getFormattedMessage(); + assertEquals(expected, event.getLevel()); assertNull(event.getThrowableProxy()); + assertFalse(message.contains("PRIVATE_"), message); + assertFalse(message.matches("(?s).*[\r\n\u000b\f\u0085\u2028\u2029].*"), message); + if (error) assertTrue(message.contains("type=java.lang.IllegalStateException"), message); + } + private static final class Key { + int renders; + @Override public String toString() { renders++; return "PRIVATE_KEY\r\nforged\u2028entry"; } + } + + private org.fireflyframework.cache.core.CacheAdapter primary() { + var cache = mock(org.fireflyframework.cache.core.CacheAdapter.class); + when(cache.isAvailable()).thenReturn(true); return cache; + } + @Test void genericGetErrorsKeepEmptyFallbackWithoutRenderingKeysOrPayload() { + var primary = primary(); var key = new Key(); + when(primary.get(key)).thenReturn(Mono.error(new IllegalStateException("PRIVATE_MESSAGE\r\nforged"))); + assertEquals(Optional.empty(), new FireflyCacheManager(primary).get(key).block()); + safeEvent(Level.ERROR, true); assertEquals(0, key.renders); + } + @Test void typedGetAndOtherFallbacksRetainResultsAndErrorLevels() { + var primary = primary(); var key = new Key(); var failure = new IllegalStateException("PRIVATE_MESSAGE\u2028forged"); + when(primary.get(key, String.class)).thenReturn(Mono.error(failure)); + when(primary.put(key, "value")).thenReturn(Mono.error(failure)); + when(primary.evict(key)).thenReturn(Mono.error(failure)); + when(primary.exists(key)).thenReturn(Mono.error(failure)); + when(primary.clear()).thenReturn(Mono.error(failure)); + when(primary.keys()).thenReturn(Mono.error(failure)); + when(primary.size()).thenReturn(Mono.error(failure)); + var manager = new FireflyCacheManager(primary); + assertEquals(Optional.empty(), manager.get(key, String.class).block()); safeEvent(Level.ERROR, true); + assertNull(manager.put(key, "value").block()); safeEvent(Level.ERROR, true); + assertEquals(false, manager.evict(key).block()); safeEvent(Level.ERROR, true); + assertEquals(false, manager.exists(key).block()); safeEvent(Level.ERROR, true); + assertNull(manager.clear().block()); safeEvent(Level.ERROR, true); + assertEquals(java.util.Set.of(), manager.keys().block()); safeEvent(Level.ERROR, true); + assertEquals(0L, manager.size().block()); safeEvent(Level.ERROR, true); + assertEquals(0, key.renders); + } + @Test void disabledLogsDoNotReadErrorPayloadOrRenderTheGenericKey() { + logger.setLevel(Level.OFF); var primary = primary(); var key = new Key(); + var reads = new java.util.concurrent.atomic.AtomicInteger(); + var error = new IllegalStateException() { @Override public String getMessage() { reads.incrementAndGet(); return "PRIVATE_MESSAGE"; } }; + when(primary.get(key)).thenReturn(Mono.error(error)); + assertEquals(Optional.empty(), new FireflyCacheManager(primary).get(key).block()); + assertEquals(0, key.renders); assertEquals(0, reads.get()); assertTrue(events.list.isEmpty()); + } +} diff --git a/modules/callbacks/README.md b/modules/callbacks/README.md index e261b6f02..6b8f408b3 100644 --- a/modules/callbacks/README.md +++ b/modules/callbacks/README.md @@ -32,7 +32,7 @@ This artifact has `pom` packaging and groups child projects. Select a concrete c ``` diff --git a/modules/callbacks/fireflyframework-callbacks-core/README.md b/modules/callbacks/fireflyframework-callbacks-core/README.md index d70c001e5..e8b872759 100644 --- a/modules/callbacks/fireflyframework-callbacks-core/README.md +++ b/modules/callbacks/fireflyframework-callbacks-core/README.md @@ -20,7 +20,7 @@ Route a configured application.submitted or application.reviewed event to author org.fireflyframework fireflyframework-callbacks-interfaces -26.10.21-SNAPSHOT +26.10.22 ``` diff --git a/modules/callbacks/fireflyframework-callbacks-core/pom.xml b/modules/callbacks/fireflyframework-callbacks-core/pom.xml index 520d4baa2..8d0a42fbb 100644 --- a/modules/callbacks/fireflyframework-callbacks-core/pom.xml +++ b/modules/callbacks/fireflyframework-callbacks-core/pom.xml @@ -6,7 +6,7 @@ org.fireflyframework fireflyframework-callbacks-core -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/callbacks/fireflyframework-callbacks-interfaces/README.md b/modules/callbacks/fireflyframework-callbacks-interfaces/README.md index fa306b2b4..5268b66ff 100644 --- a/modules/callbacks/fireflyframework-callbacks-interfaces/README.md +++ b/modules/callbacks/fireflyframework-callbacks-interfaces/README.md @@ -20,7 +20,7 @@ Use these DTOs to configure a partner callback for application-status events. Cr org.fireflyframework fireflyframework-callbacks -26.10.21-SNAPSHOT +26.10.22 ../pom.xml ``` diff --git a/modules/callbacks/fireflyframework-callbacks-interfaces/pom.xml b/modules/callbacks/fireflyframework-callbacks-interfaces/pom.xml index 15daddeae..55d33fe54 100644 --- a/modules/callbacks/fireflyframework-callbacks-interfaces/pom.xml +++ b/modules/callbacks/fireflyframework-callbacks-interfaces/pom.xml @@ -6,7 +6,7 @@ org.fireflyframework fireflyframework-callbacks-interfaces -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/callbacks/fireflyframework-callbacks-models/README.md b/modules/callbacks/fireflyframework-callbacks-models/README.md index 097e0846a..72e98ca70 100644 --- a/modules/callbacks/fireflyframework-callbacks-models/README.md +++ b/modules/callbacks/fireflyframework-callbacks-models/README.md @@ -20,7 +20,7 @@ The callback service stores destinations, subscriptions, and executions separate org.fireflyframework fireflyframework-callbacks -26.10.21-SNAPSHOT +26.10.22 ../pom.xml ``` diff --git a/modules/callbacks/fireflyframework-callbacks-models/pom.xml b/modules/callbacks/fireflyframework-callbacks-models/pom.xml index e3fac4dbe..f31278b12 100644 --- a/modules/callbacks/fireflyframework-callbacks-models/pom.xml +++ b/modules/callbacks/fireflyframework-callbacks-models/pom.xml @@ -6,7 +6,7 @@ org.fireflyframework fireflyframework-callbacks-models -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/callbacks/fireflyframework-callbacks-sdk/README.md b/modules/callbacks/fireflyframework-callbacks-sdk/README.md index 15211de9d..0793893bc 100644 --- a/modules/callbacks/fireflyframework-callbacks-sdk/README.md +++ b/modules/callbacks/fireflyframework-callbacks-sdk/README.md @@ -20,7 +20,7 @@ A partner-management client can use this generated SDK to register or inspect ca org.fireflyframework fireflyframework-callbacks -26.10.21-SNAPSHOT +26.10.22 ../pom.xml ``` diff --git a/modules/callbacks/fireflyframework-callbacks-sdk/pom.xml b/modules/callbacks/fireflyframework-callbacks-sdk/pom.xml index 3b08a72f8..6fb5fca51 100644 --- a/modules/callbacks/fireflyframework-callbacks-sdk/pom.xml +++ b/modules/callbacks/fireflyframework-callbacks-sdk/pom.xml @@ -6,7 +6,7 @@ org.fireflyframework fireflyframework-callbacks-sdk -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/callbacks/fireflyframework-callbacks-web/README.md b/modules/callbacks/fireflyframework-callbacks-web/README.md index 9701b2f9b..8789526b0 100644 --- a/modules/callbacks/fireflyframework-callbacks-web/README.md +++ b/modules/callbacks/fireflyframework-callbacks-web/README.md @@ -20,7 +20,7 @@ Operate this service if onboarding partners manage callbacks through HTTP. Confi org.fireflyframework fireflyframework-callbacks -26.10.21-SNAPSHOT +26.10.22 ../pom.xml ``` diff --git a/modules/callbacks/fireflyframework-callbacks-web/pom.xml b/modules/callbacks/fireflyframework-callbacks-web/pom.xml index a90b6d07c..990fec648 100644 --- a/modules/callbacks/fireflyframework-callbacks-web/pom.xml +++ b/modules/callbacks/fireflyframework-callbacks-web/pom.xml @@ -6,7 +6,7 @@ org.fireflyframework fireflyframework-callbacks-web -26.10.21-SNAPSHOT +26.10.22 diff --git a/modules/callbacks/pom.xml b/modules/callbacks/pom.xml index 440d81e2f..2c238877c 100644 --- a/modules/callbacks/pom.xml +++ b/modules/callbacks/pom.xml @@ -7,13 +7,13 @@ org.fireflyframework fireflyframework-callbacks -26.10.21-SNAPSHOT +26.10.22 ../pom.xml org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml org.fireflyframework fireflyframework-callbacks -26.10.21-SNAPSHOT +26.10.22 pom Firefly Framework - Callbacks Library diff --git a/modules/client/README.md b/modules/client/README.md index 82c034952..e14108738 100644 --- a/modules/client/README.md +++ b/modules/client/README.md @@ -20,10 +20,18 @@ The service may call a document-verification partner during a later onboarding s``` +## Security helper diagnostics + +Secret storage, missing-secret, and removal events use fixed messages without caller-supplied secret names. Successful JWT validation does not log the verified subject. A non-null subject must remain text; other shapes receive a constant `JwtValidationException` diagnostic. Missing/null subjects and the existing required-claim presence rule retain their behavior. Encryption checks the combined IV and ciphertext length before allocating its output buffer; this check does not impose a new plaintext or configured-IV size budget. The cipher format and validation APIs are unchanged. + +The master key accepts 32 UTF-8 bytes or the standard Base64 encoding of 32 key bytes. Existing 32-byte UTF-8 keys retain their raw interpretation. `generateMasterKey()` returns a Base64 key that can be supplied to the builder or `rotateKey(String)`; `create()` uses a generated key directly. The AES-GCM IV/ciphertext format is preserved. Keep the configured key in an application-managed secret source when ciphertext must outlive a manager instance. + +This encoding correction is part of the current source checkpoint. Consumer adoption requires a coordinated follow-up release; previously published artifacts are not replaced under the same version. + ## Quick start The injection examples use Lombok `@RequiredArgsConstructor`. Add its provided dependency and annotation processor as described in [Constructor injection](../../docs/site/maven.md#constructor-injection). diff --git a/modules/client/examples/complete-example/pom.xml b/modules/client/examples/complete-example/pom.xml index 79eb056e2..b1d6268d9 100644 --- a/modules/client/examples/complete-example/pom.xml +++ b/modules/client/examples/complete-example/pom.xml @@ -7,7 +7,7 @@ org.fireflyframework fireflyframework-client -26.10.21-SNAPSHOT +26.10.22 org.fireflyframework.example complete-example -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Common Client - Complete Example @@ -21,7 +21,7 @@- 26.10.21-SNAPSHOT +26.10.22 25 25 25 diff --git a/modules/client/pom.xml b/modules/client/pom.xml index 8178e5a9a..2e1690eb7 100644 --- a/modules/client/pom.xml +++ b/modules/client/pom.xml @@ -7,12 +7,12 @@org.fireflyframework fireflyframework-parent -26.10.21-SNAPSHOT +26.10.22 ../../build/parent/pom.xml fireflyframework-client -26.10.21-SNAPSHOT +26.10.22 jar Firefly Framework - Common Client Library diff --git a/modules/client/src/main/java/org/fireflyframework/client/multipart/MultipartUploadHelper.java b/modules/client/src/main/java/org/fireflyframework/client/multipart/MultipartUploadHelper.java index 80d23288c..15ffed176 100644 --- a/modules/client/src/main/java/org/fireflyframework/client/multipart/MultipartUploadHelper.java +++ b/modules/client/src/main/java/org/fireflyframework/client/multipart/MultipartUploadHelper.java @@ -481,7 +481,7 @@ publicMono uploadFileCompressed(String endpoint, File file, String field * Compresses a file using GZIP. */ private File compressFile(File file) throws IOException { - File compressedFile = File.createTempFile("upload_", ".gz"); + File compressedFile = Files.createTempFile("upload_", ".gz").toFile(); try (FileInputStream fis = new FileInputStream(file); FileOutputStream fos = new FileOutputStream(compressedFile); diff --git a/modules/client/src/main/java/org/fireflyframework/client/security/JwtValidator.java b/modules/client/src/main/java/org/fireflyframework/client/security/JwtValidator.java index 7ee81e3de..164c5bd56 100644 --- a/modules/client/src/main/java/org/fireflyframework/client/security/JwtValidator.java +++ b/modules/client/src/main/java/org/fireflyframework/client/security/JwtValidator.java @@ -135,7 +135,7 @@ public JwtClaims validate(String token) throws JwtValidationException { // Validate claims validateClaims(claims); - log.debug("JWT validated successfully for subject: {}", claims.getSubject()); + log.debug("JWT validated successfully"); return claims; } @@ -229,6 +229,11 @@ private void validateClaims(JwtClaims claims) throws JwtValidationException { throw new JwtValidationException("Missing required claim: " + requiredClaim); } } + + Object subject = claims.getClaim("sub"); + if (subject != null && !(subject instanceof String)) { + throw new JwtValidationException("Invalid subject claim"); + } } /** diff --git a/modules/client/src/main/java/org/fireflyframework/client/security/SecretsEncryptionManager.java b/modules/client/src/main/java/org/fireflyframework/client/security/SecretsEncryptionManager.java index 5b3473596..74bfa27b2 100644 --- a/modules/client/src/main/java/org/fireflyframework/client/security/SecretsEncryptionManager.java +++ b/modules/client/src/main/java/org/fireflyframework/client/security/SecretsEncryptionManager.java @@ -48,7 +48,7 @@ public class SecretsEncryptionManager { /** - * Master encryption key (32 bytes for AES-256). + * Master encryption key: 32 UTF-8 bytes or their Base64 encoding for AES-256. */ private final String masterKey; @@ -114,7 +114,7 @@ public String encrypt(String plaintext) { byte[] ciphertext = cipher.doFinal(plaintext.getBytes(StandardCharsets.UTF_8)); // Combine IV and ciphertext - ByteBuffer byteBuffer = ByteBuffer.allocate(iv.length + ciphertext.length); + ByteBuffer byteBuffer = ByteBuffer.allocate(checkedCombinedLength(iv.length, ciphertext.length)); byteBuffer.put(iv); byteBuffer.put(ciphertext); @@ -180,7 +180,7 @@ public void storeSecret(String key, String value) { } String encrypted = encrypt(value); encryptedSecrets.put(key, encrypted); - log.debug("Stored encrypted secret: {}", key); + log.debug("Stored encrypted secret"); } /** @@ -192,7 +192,7 @@ public void storeSecret(String key, String value) { public String getSecret(String key) { String encrypted = encryptedSecrets.get(key); if (encrypted == null) { - log.warn("Secret not found: {}", key); + log.warn("Secret not found"); return null; } return decrypt(encrypted); @@ -207,7 +207,7 @@ public String getSecret(String key) { public boolean removeSecret(String key) { boolean removed = encryptedSecrets.remove(key) != null; if (removed) { - log.debug("Removed secret: {}", key); + log.debug("Removed secret"); } return removed; } @@ -267,6 +267,13 @@ public SecretsEncryptionManager rotateKey(String newMasterKey) { return newManager; } + static int checkedCombinedLength(int ivLength, int ciphertextLength) { + if (ivLength < 0 || ciphertextLength < 0) { + throw new IllegalArgumentException("Lengths cannot be negative"); + } + return Math.addExact(ivLength, ciphertextLength); + } + /** * Gets the secret key from the master key. * @@ -277,8 +284,15 @@ private SecretKey getSecretKey() { throw new IllegalStateException("Master key is not configured"); } - // Ensure master key is exactly 32 bytes for AES-256 + // Prefer legacy UTF-8 keys so existing configurations keep their meaning. byte[] keyBytes = masterKey.getBytes(StandardCharsets.UTF_8); + if (keyBytes.length != 32) { + try { + keyBytes = Base64.getDecoder().decode(masterKey); + } catch (IllegalArgumentException e) { + throw new IllegalStateException("Master key must be exactly 32 bytes for AES-256"); + } + } if (keyBytes.length != 32) { throw new IllegalStateException("Master key must be exactly 32 bytes for AES-256"); } @@ -308,13 +322,8 @@ public static String generateMasterKey() { * @return configured SecretsEncryptionManager */ public static SecretsEncryptionManager create() { - String masterKey = generateMasterKey(); - // Decode to get exactly 32 bytes - byte[] keyBytes = Base64.getDecoder().decode(masterKey); - String key32Bytes = new String(keyBytes, StandardCharsets.ISO_8859_1); - return SecretsEncryptionManager.builder() - .masterKey(key32Bytes) + .masterKey(generateMasterKey()) .build(); } } diff --git a/modules/client/src/test/java/org/fireflyframework/client/helpers/MultipartUploadHelperTest.java b/modules/client/src/test/java/org/fireflyframework/client/helpers/MultipartUploadHelperTest.java index 8a924067f..3ba79a5f1 100644 --- a/modules/client/src/test/java/org/fireflyframework/client/helpers/MultipartUploadHelperTest.java +++ b/modules/client/src/test/java/org/fireflyframework/client/helpers/MultipartUploadHelperTest.java @@ -25,6 +25,30 @@ @DisplayName("Multipart Upload Helper Tests") class MultipartUploadHelperTest { + @Test + void compressedTemporaryFileIsOwnerOnlyAndContainsTheOriginalBytes(@TempDir Path directory) + throws Exception { + Path input = java.nio.file.Files.writeString(directory.resolve("input.txt"), "owned upload content"); + var helper = new MultipartUploadHelper("http://localhost:8080"); + var compress = MultipartUploadHelper.class.getDeclaredMethod("compressFile", File.class); + compress.setAccessible(true); + File compressed = (File) compress.invoke(helper, input.toFile()); + try { + if (compressed.toPath().getFileSystem().supportedFileAttributeViews().contains("posix")) { + assertThat(java.nio.file.Files.getPosixFilePermissions(compressed.toPath())) + .containsExactlyInAnyOrder( + java.nio.file.attribute.PosixFilePermission.OWNER_READ, + java.nio.file.attribute.PosixFilePermission.OWNER_WRITE); + } + try (var gzip = new java.util.zip.GZIPInputStream(java.nio.file.Files.newInputStream(compressed.toPath()))) { + assertThat(new String(gzip.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8)) + .isEqualTo("owned upload content"); + } + } finally { + java.nio.file.Files.deleteIfExists(compressed.toPath()); + } + } + @Test @DisplayName("Should create upload helper with valid base URL") void shouldCreateUploadHelperWithValidBaseUrl() { diff --git a/modules/client/src/test/java/org/fireflyframework/client/security/JwtValidatorLoggingTest.java b/modules/client/src/test/java/org/fireflyframework/client/security/JwtValidatorLoggingTest.java new file mode 100644 index 000000000..85270b9ec --- /dev/null +++ b/modules/client/src/test/java/org/fireflyframework/client/security/JwtValidatorLoggingTest.java @@ -0,0 +1,208 @@ +/* + * Copyright 2024-2026 Firefly Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.fireflyframework.client.security; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; + +import com.fasterxml.jackson.databind.ObjectMapper; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; +import org.slf4j.LoggerFactory; + +import java.nio.charset.StandardCharsets; +import java.time.Instant; +import java.util.Arrays; +import java.util.Base64; +import java.util.HexFormat; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import javax.crypto.KeyGenerator; +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; + +class JwtValidatorLoggingTest { + private Logger logger; + private Level previousLevel; + private ListAppender appender; + private String signingKey; + + @BeforeEach + void captureLogs() throws Exception { + logger = (Logger) LoggerFactory.getLogger(JwtValidator.class); + previousLevel = logger.getLevel(); + appender = new ListAppender<>(); + appender.start(); + logger.addAppender(appender); + logger.setLevel(Level.DEBUG); + var generator = KeyGenerator.getInstance("HmacSHA256"); + generator.init(256); + signingKey = HexFormat.of().formatHex(generator.generateKey().getEncoded()); + } + + @AfterEach + void restoreLogs() { + logger.detachAppender(appender); + logger.setLevel(previousLevel); + appender.stop(); + } + + @Test + void validatesTheRealSignatureAndReturnsClaimsWithoutLoggingSubjectOrToken() throws Exception { + String subject = "synthetic-sensitive-subject-Ω"; + String token = + signedToken(subject, signingKey, Instant.now().plusSeconds(300).getEpochSecond()); + var claims = validator().validate(token); + assertThat(claims.getSubject()).isEqualTo(subject); + assertThat(claims.getIssuer()).isEqualTo("https://synthetic-issuer.example"); + assertThat(claims.getAudience()).isEqualTo("synthetic-audience"); + assertThat(appender.list).isNotEmpty(); + for (var event : appender.list) { + assertThat(event.getFormattedMessage()).doesNotContain(subject, token, signingKey); + if (event.getArgumentArray() != null) { + assertThat(Arrays.toString(event.getArgumentArray())) + .doesNotContain(subject, token, signingKey); + } + } + } + + @Test + void rejectsAnotherKeysSignatureWithoutEmittingASuccessLog() throws Exception { + String token = + signedToken( + "synthetic-untrusted-subject", + signingKey + "different", + Instant.now().plusSeconds(300).getEpochSecond()); + assertThatThrownBy(() -> validator().validate(token)) + .isInstanceOf(JwtValidator.JwtValidationException.class) + .hasMessage("Invalid JWT signature"); + assertThat(appender.list).isEmpty(); + } + + @Test + void rejectsExpiredSignedClaimsWithoutEmittingASuccessLog() throws Exception { + String token = + signedToken( + "synthetic-expired-subject", + signingKey, + Instant.now().minusSeconds(300).getEpochSecond()); + assertThatThrownBy(() -> validator().validate(token)) + .isInstanceOf(JwtValidator.JwtValidationException.class) + .hasMessage("JWT has expired"); + assertThat(appender.list).isEmpty(); + } + + @ParameterizedTest + @MethodSource("nontextSubjects") + void rejectsSignedNontextSubjectsWithAConstantDiagnostic(Object subject) throws Exception { + Map claims = baseClaims(); + claims.put("sub", subject); + String token = signedClaims(claims, signingKey); + assertThatThrownBy(() -> validator().validate(token)) + .isInstanceOf(JwtValidator.JwtValidationException.class) + .hasMessage("Invalid subject claim"); + assertThat(appender.list).isEmpty(); + } + + static Stream