DRAFT: Change Data Capture documentation

content/en/docs/refguide/modeling/integration/_index.md

@@ -9,7 +9,7 @@ description: "Introduces the services that Mendix uses for application integrati
 
 ## Introduction
 
-Integration with other applications (other Mendix apps or applications built with different technology) can be done using REST, OData, GraphQL, SOAP/Web Services, or Business Events. Mendix can also import and export data from XML and JSON and supports integration with machine learning models.
+Integration with other applications (other Mendix apps or applications built with different technology) can be done using REST, OData, GraphQL, SOAP/Web Services, Business Events, or Change Data Capture. Mendix can also import and export data from XML and JSON and supports integration with machine learning models.
 
 Data integration capabilities in Studio Pro is part of [Mendix Connect](#integration-mx-connect).
 
@@ -21,6 +21,7 @@ Data integration capabilities in Studio Pro is part of [Mendix Connect](#integra
 * [REST services](/refguide/integration/rest-services/)
 * [Web services](/refguide/integration/web-services/)
 * [Business Event services](/refguide/business-event-services/)
+* [Change Data Capture services](/refguide/change-data-capture/)
 * [Published GraphQL services](/refguide/published-odata-services/) (experimental)
 
 In addition to integration capabilities in Studio Pro, Mendix Connect functionalities include:

content/en/docs/refguide/modeling/integration/change-data-capture/_index.md

@@ -0,0 +1,98 @@
+---
+title: "Change Data Capture"
+url: /refguide/change-data-capture/
+weight: 45
+description: "Describes Change Data Capture (CDC) services in Studio Pro, which publish domain model entity changes as Kafka streams for data warehouse and analytics pipelines."
+---
+
+## Introduction
+
+Change Data Capture (CDC) lets you stream domain model data out of a Mendix app in near real time. When objects in a tracked entity are created, updated, or deleted, the Mendix runtime captures those changes and publishes them as events to a Kafka topic. Downstream systems — such as data warehouses or analytics pipelines — can then consume the stream without polling the Mendix database.
+
+CDC is intended for developers who need to move Mendix domain model data to external data stores, such as a data warehouse, data lake, or blob storage.
+
+## How It Works
+
+A CDC service document in Studio Pro defines which entities the runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs — create, update, or delete — the runtime publishes an event to the corresponding topic.
+
+Read events are triggered as snapshots during startup or when a stream changes in a way that constitutes a breaking change.  Snapshots populate the new topic with the current data of the tracked entity. See [Revisions](/refguide/published-cdc-services/#revisions).
+
+The Kafka broker is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For details on configuring BYOK, see the [Mendix Event Broker](/appstore/services/event-broker/) page.
+
+To move the streamed data to a destination such as Azure Blob Storage or AWS S3, you configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) separately in the Event Broker Manager after deployment.
+
+## Prerequisites
+
+{{% alert color="warning" %}}
+Change Data Capture is not available for Free Apps. A licensed Mendix Cloud environment is required.
+{{% /alert %}}
+
+* A licensed Mendix Cloud environment.
+* A [Mendix Event Broker](/appstore/services/event-broker/) license, or a BYOK Kafka cluster configured as described in [Mendix Event Broker](/appstore/services/event-broker/).
+* An [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) configured in the Event Broker Manager if you want to route CDC events to external storage.
+
+## Setting Up Change Data Capture
+
+1. In Studio Pro, right-click a module in the App Explorer and choose **Add other** > **Change data capture service**.
+2. Configure the entities to track and their exposed names. See [Published CDC Services](/refguide/published-cdc-services/) for details.
+3. Deploy the app. The runtime creates Kafka topics for each tracked entity automatically.
+4. In the [Event Broker Manager](https://broker.mendix.com/), configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) to route CDC events to your destination.
+
+## Runtime Configuration {#runtime-configuration}
+
+CDC requires runtime settings to connect to a Kafka broker. In Studio Pro, open **App** > **Settings**, go to the **Configurations** tab, select or create a configuration, and open the **Custom** tab to add the settings below. For deployed environments, set these via your environment's custom runtime settings.
+
+{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/cdc-custom-configuration.png" alt="Edit Configuration dialog in Studio Pro showing the Custom tab with Kafka.BootstrapServers set to an IP address and port" >}}
+
+### Running Locally {#local-configuration}
+
+When running the app locally, only the bootstrap server address is required:
+
+| Name | Description | Default Value |
+| --- | --- | --- |
+| `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | |
+
+### Bring Your Own Kafka (BYOK) {#byok-configuration}
+
+When connecting to a BYOK Kafka cluster, provide the bootstrap server address and credentials for authentication. The supported authentication method is SASL/SCRAM-SHA-512.
+
+| Name | Description | Default Value | Required |
+| --- | --- | --- | --- |
+| `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | | true |
+| `Kafka.Username` | The username for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. | | true |
+| `Kafka.Password` | The password for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. | | true |
+| `EventBroker.Space` | The Event Broker space in which the app is placed. Defines which other applications can exchange events with this app. | `local` | |
+| `Datastorage.StartSnapshotMinInterval` | The minimum time, in milliseconds, that must have elapsed since the last snapshot before a new snapshot is triggered on startup. If startup occurs within this window, the snapshot is skipped. | `300000` | |
+| `EventBroker.CdcForceSnapshotEntities` | A comma-separated list of entity names (for example, `Module.EntityA,Module.EntityB`) that are always snapshotted on startup, even when no topic change has occurred that would otherwise trigger a snapshot. | | |
+| `EventBroker.CdcActiveProducerElectionTopic` | The Kafka topic used for active-producer election among clustered app instances. When set together with `EventBroker.CdcActiveProducerElectionGroupId` and `EventBroker.CdcProducerTransactionId`, instances coordinate to elect a single active CDC producer. If any of the three settings is missing, the app falls back to single-instance mode. | | |
+| `EventBroker.CdcActiveProducerElectionGroupId` | The Kafka consumer group ID used for active-producer election. Must be set together with `EventBroker.CdcActiveProducerElectionTopic` and `EventBroker.CdcProducerTransactionId` to enable active-producer election mode. | | |
+| `EventBroker.CdcProducerTransactionId` | The transactional ID for the Kafka CDC producer, enabling exactly-once delivery semantics. Must be set together with `EventBroker.CdcActiveProducerElectionTopic` and `EventBroker.CdcActiveProducerElectionGroupId` to enable active-producer election mode. | | |
+
+
+### Active-Producer Election {#active-producer-election}
+
+In a horizontally scaled deployment, multiple app instances would otherwise all publish CDC events to the same Kafka topics, producing duplicates. Active-producer election ensures that only one instance publishes at a time.
+
+When all three election settings are configured, each instance joins the same Kafka consumer group and subscribes to the election topic. Kafka assigns partition 0 of that topic to exactly one member of the group — that instance becomes the active producer. All other instances stand by without publishing.
+
+If the active instance stops sending heartbeats (for example, because it crashed or was scaled down), Kafka triggers a rebalance and assigns the partition to another instance, which then starts producing. The departing instance drains any in-flight events before relinquishing the partition to prevent gaps.
+
+The `EventBroker.CdcProducerTransactionId` setting enables Kafka transactions on the active producer, which provides exactly-once delivery guarantees during normal operation and clean handover during failover.
+
+If any of the three settings is missing, the runtime logs a warning and falls back to single-instance mode, where all instances publish independently.
+
+### Message Format {#message-format}
+
+CDC events follow the [Debezium](https://debezium.io/) envelope format. Each message contains an `op` field indicating the operation type (`c` for create, `u` for update, `d` for delete, `r` for read/snapshot), a `before` and `after` payload with the entity attribute values, and a `source` block with metadata such as the timestamp and Mendix version.
+
+Because the format is Debezium-compatible, BYOK consumers that already support Debezium — such as Kafka Connect sink connectors for databases or data warehouses — can consume CDC events without any message transformation.
+
+To parse the messages, the schema for each tracked entity is available as an AsyncAPI document. You can download it from the [Event Broker Manager](https://broker.mendix.com/) on the service details page for the CDC service.
+
+For details on setting up a BYOK cluster with the Mendix Event Broker, see [Mendix Event Broker](/appstore/services/event-broker/).
+
+## Read More
+
+* [Published CDC Services](/refguide/published-cdc-services/)
+* [Mendix Event Broker](/appstore/services/event-broker/)
+* [Event Broker Bridges](/appstore/services/event-broker/#manage-mx-broker-bridge)

content/en/docs/refguide/modeling/integration/change-data-capture/published-cdc-services.md

@@ -0,0 +1,99 @@
+---
+title: "Published CDC Services"
+url: /refguide/published-cdc-services/
+weight: 10
+description: "Describes how to configure a Published CDC Service document in Studio Pro to stream entity changes to Kafka topics."
+---
+
+## Introduction
+
+A Published CDC Service document defines the entities whose object changes the Mendix runtime tracks and publishes as Kafka events. Each tracked entity produces a stream of create, update, and delete events on its own Kafka topic.
+
+{{% alert color="warning" %}}
+Change Data Capture is a beta feature in Mendix 11.12. Its behavior and configuration options may change in future releases.
+{{% /alert %}}
+
+## Creating a Published CDC Service {#create}
+
+To create a published CDC service, right-click a module in the App Explorer and choose **Add other** > **Change data capture service**. Studio Pro adds the document to that module for logical grouping, but the service operates at app level.
+
+You can have multiple CDC service documents in an app — for example, to group entities by domain area or team ownership.
+
+## General {#general}
+
+### Service Name {#service-name}
+
+The service name uniquely identifies the CDC service within the app.  The App Name is used as part of the topic to ensure uniqueness.
+
+### Description {#description}
+
+An optional description for the CDC service.
+
+## Entities to Track {#entities}
+
+The **Entities to track** table lists the entities whose object changes are published to Kafka.
+
+{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/published-cdc-service.png" alt="Published CDC Service document showing the Entities to track table with columns for Exposed name, Modification, Revision, and Topic" >}}
+
+Use the toolbar to manage tracked entities:
+
+* **+ Add** — add an entity from the domain model
+* **Remove** — stop tracking a selected entity
+* **Accept changes** — lock in the current revision numbers after reviewing modifications (see [Revisions](#revisions))
+
+Each row in the table has the following columns:
+
+| Column | Description |
+| --- | --- |
+| **Entities** | The domain model entity being tracked. Expand the row to view and select individual attributes and associations. |
+| **Exposed name** | The name used for this entity in the Kafka topic and event payload. Defaults to the entity name. |
+| **Modification** | The pending change state: **Added**, **Changed**, or **Removed**. Blank if the entity is unchanged since the last accepted revision. |
+| **Revision** | The schema version of the entity's event payload. See [Revisions](#revisions). |
+| **Topic** | The Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name.  See [Bring Your Own Kafka (BYOK)](/refguide/change-data-capture/#byok-configuration) |
+
+### Adding an Entity {#add-entity}
+
+1. Click **+ Add** in the toolbar.
+2. Select a persistable entity from the domain model.
+3. Optionally edit the **Exposed name**.
+4. Expand the row to deselect any attributes or associations you do not want included in the event payload.
+
+### Attribute and Association Selection {#attributes}
+
+Expand an entity row to see each attribute and association with a checkbox. Uncheck an item to exclude it from the event payload. The **Exposed name** column lets you rename individual attributes in the payload independently of their domain model names.
+
+Associations appear as nested data within the parent entity's event payload. They do not produce separate Kafka topics and show no **Revision** or **Topic** values of their own.
+
+## Revisions {#revisions}
+
+Each tracked entity has a **Revision** number that identifies the schema version of its event payload. Downstream consumers use the revision to detect and respond to schema changes. The revision is also embedded in the Kafka topic name, so each schema version has its own isolated topic.
+
+Studio Pro manages revisions automatically. When you modify the tracked configuration of an entity, Studio Pro marks it as **Changed** and calculates the new revision based on whether the change is breaking or non-breaking:
+
+* **Minor revision** (for example, `1.0` → `1.1`) — a non-breaking change such as adding a new attribute. Existing consumers can continue reading the topic without modification.
+* **Major revision** (for example, `1.0` → `2.0`) — a breaking change such as removing an attribute, renaming an entity's exposed name, or removing an entity from tracking. Consumers must be updated to use the new topic.
+
+{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/published-cdc-service-changes.png" alt="Published CDC Service document showing entities with Changed and Removed modification states and updated revision numbers" >}}
+
+### Accepting Changes {#accepting-changes}
+
+Pending modifications are not finalized until you click **Accept changes** in the toolbar. Until you do, Studio Pro shows a consistency error on the document. You must resolve this error by accepting the changes before you can deploy the app.
+
+Accepting changes confirms the new revision numbers and clears the modification states, leaving the document in a clean state ready for deployment.
+
+{{% alert color="warning" %}}
+A major revision creates a new Kafka topic. Consumers subscribed to the previous topic version will no longer receive events after deployment. Ensure downstream systems are updated before deploying a major revision change.
+{{% /alert %}}
+
+## Runtime Behavior {#runtime}
+
+* The CDC service runs in the system context — no user security applies.
+* Events are published for every committed object change: create, update, and delete.
+* Kafka topics are created automatically on deployment for each tracked entity.
+* Events use [CloudEvents](https://cloudevents.io/) payload format, consistent with other Mendix Event Broker services.
+
+## Read More
+
+* [Change Data Capture](/refguide/change-data-capture/)
+* [Mendix Event Broker](/appstore/services/event-broker/)
+* [Event Broker Bridges](/appstore/services/event-broker/#manage-mx-broker-bridge)