Conversation
…e streams protocol
Kafka 4.4 adds static membership (group.instance.id) to the Streams
Rebalance Protocol, but the 4.4 docs announce it without the broker
requirement. The InstanceId field exists in StreamsGroupHeartbeatRequest
v0, so a 4.4 Kafka Streams client sends it to any broker; brokers older
than 4.4 reject it with INVALID_REQUEST ("Static membership is not yet
supported"), which the client treats as a fatal error and shuts down.
This matters in particular when brokers are downgraded from 4.4 to 4.3
or 4.2 while static members are running.
Add the broker requirement, the failure mode, and the recovery step
(remove group.instance.id and restart the application) to the Streams
upgrade guide, the Streams Rebalance Protocol developer guide, the
group.instance.id section of the Streams config reference, and the 4.4
notable changes in the core upgrade notes.
mjsax
reviewed
Sep 17, 2026
Co-authored-by: Matthias J. Sax <mjsax@apache.org>
Co-authored-by: Matthias J. Sax <mjsax@apache.org>
Co-authored-by: Matthias J. Sax <mjsax@apache.org>
Co-authored-by: Matthias J. Sax <mjsax@apache.org>
Contributor
Author
|
@mjsax applied suggested changes |
chia7712
reviewed
Sep 17, 2026
| * **Offline Migration**: After shutting down all members and waiting for their `session.timeout.ms` to expire (or forcing an explicit group leave), a classic group can be converted to a streams group and a streams group can be converted to a classic group. The only broker-side group data that will be preserved are the committed offsets. Internal topics (changelog and repartition topics) will continue to exist as regular Kafka topics. | ||
|
|
||
| * **Static Membership**: Streams applications can configure [`group.instance.id`](../config-streams#group-instance-id) when using `group.protocol=streams`. However, for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. | ||
| * **Static Membership**: Streams applications can configure [`group.instance.id`](../config-streams#group-instance-id) when using `group.protocol=streams`. This requires clients and brokers running Kafka 4.4 or newer: brokers on older versions reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running; if brokers have already been downgraded, remove `group.instance.id` from the application configuration and restart the application. Also note that for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. |
Member
There was a problem hiding this comment.
Do not downgrade brokers below 4.4 while static members are running
This is a bit misleading to me since it reads like "I will be fine if I stop the application". Maybe we could tweak the two statements, for example:
Before downgrading brokers below 4.4, or if brokers have already been downgraded, remove group.instance.id from all Streams applications using group.protocol=streams and restart them.
Contributor
Author
There was a problem hiding this comment.
Thanks, replaced statement in streams-rebalance-protocol.md and upgrade-guide.md with it
…cations before a broker downgrade
chia7712
reviewed
Sep 18, 2026
| * Share groups now support dead-letter queue functionality as outlined in [KIP-1191](https://cwiki.apache.org/confluence/x/fApJFg). Any records which are released (beyond max delivery count) or rejected by the share consumer become eligible for DLQ. Share group DLQ gets enabled when the Kafka feature `share.version` is upgraded to 2. The user can configure a DLQ topic on a share group by setting the dynamic config `errors.deadletterqueue.topic.name` (default `""`) to the name of the DLQ topic. The cluster can be configured to auto create the DLQ topics by setting the dynamic cluster config `errors.deadletterqueue.auto.create.topics.enable` to `true` (default `false`). If auto create is not enabled, the user must create the DLQ topic like a standard Kafka topic and set the dynamic config `errors.deadletterqueue.group.enable` to `true` on the DLQ topic. The DLQ topic name must be prefixed by the value set in the dynamic cluster config `errors.deadletterqueue.topic.name.prefix` (default `dlq.`). The records sent to the DLQ topic by default only contain source record metadata like group, topic name, partition id, offset and delivery count. If original record data is also required, the user must set the dynamic config `errors.deadletterqueue.copy.record.enable` to `true` on the share group. | ||
| * Kafka Connect distributed workers now support the `internal.topics.automatic.creation.enable` configuration (default: `true`). When set to `false`, Connect will not automatically create internal topics (offset, config, status, and connector-specific offset topics) and will instead fail at startup if any of these topics are missing. A new `connect-internal-topics.sh` tool is also available for manually creating these topics. For further details, please refer to [KIP-1209](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209:+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect). | ||
| * Streams groups now support broker-side custom task assignors, registered via the new broker configuration `group.streams.assignors` and selected per group with the new group configuration `streams.assignor.name`. For further details, please refer to [KIP-1357](https://cwiki.apache.org/confluence/x/NoSnGQ). | ||
| * Kafka Streams now supports static membership (`group.instance.id`) with the Streams Rebalance Protocol (`group.protocol=streams`). This requires brokers and clients running 4.4 or newer: brokers older than 4.4 reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running. For further details, please refer to the [Kafka Streams upgrade guide](/{version}/streams/upgrade-guide/). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update doc for KIP 1071's static membership in 4.4+ broker
Reviewers: Matthias J. Sax matthias@confluent.io, Chia-Ping Tsai chia7712@gmail.com