Skip to content

Drop unexpected Pacemaker frames instead of terminating the daemon; mark Pacemaker as deprecated - #9084

Open
rzo1 wants to merge 3 commits into
masterfrom
pacemaker-hardening
Open

Drop unexpected Pacemaker frames instead of terminating the daemon; mark Pacemaker as deprecated#9084
rzo1 wants to merge 3 commits into
masterfrom
pacemaker-hardening

Conversation

@rzo1

@rzo1 rzo1 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Robustness: drop unexpected Pacemaker frames

A CONTROL_MESSAGE frame is decoded by the Pacemaker ThriftDecoder into a ControlMessage. With pacemaker.auth.method: NONE, or after the SASL handler forwarded it upstream, it reached PacemakerServer.received(), which cast it to HBMessage. The ClassCastException ended in StormServerHandler.exceptionCaught, which treats anything but an IOException as fatal and exits the JVM. Malformed requests took the same path.

This mirrors what was recently done for the worker path (#9076):

  • The server-side ThriftDecoder only accepts SASL_TOKEN_MESSAGE_REQUEST (the control message a client sends to start the DIGEST handshake). Other control frames, and control frames with a missing/short/unknown payload, are discarded and the connection closed. SASL_MESSAGE_TOKEN and HBMessage frames decode as before, so DIGEST and KERBEROS handshakes are unchanged. On the client side a malformed control frame surfaces as an IOException, so PacemakerClientHandler reconnects as before.
  • PacemakerServer.received() checks the message type; anything other than an HBMessage is logged and the connection closed.
  • The Pacemaker pipeline uses a new PacemakerServerHandler that closes only the failing connection on an Exception; Errors still go to StormServerHandler. StormServerHandler and the worker messaging pipeline are unchanged.

Tests: new ThriftDecoderTest and PacemakerServerTest.

Mark Pacemaker as deprecated

Since STORM-2693 workers heartbeat to their supervisor, which reports them to Nimbus, so Pacemaker is only kept for backward compatibility (docs/ClusterMetrics.md already says so). This marks the Pacemaker server/client classes, the pacemaker state storage and the pacemaker.* config keys as @Deprecated, and adds a notice to docs/Pacemaker.md and the storm pacemaker help.

Richard Zowalla added 2 commits September 11, 2026 19:12
The Pacemaker server decodes a CONTROL_MESSAGE frame into a ControlMessage.
With no SASL handler installed (pacemaker.auth.method NONE), or once the SASL
handler has forwarded it upstream, the object reached PacemakerServer.received(),
which cast it to HBMessage. The resulting ClassCastException reached
StormServerHandler.exceptionCaught, which treats anything but an IOException as
fatal and exits the JVM. A malformed request (undecodable thrift payload, a
control frame without payload or with an unknown code, a request without data)
took the same path.

The server-side ThriftDecoder now accepts only the control message a Pacemaker
client sends, SASL_TOKEN_MESSAGE_REQUEST, which starts the DIGEST handshake.
Any other control frame, and control frames whose payload is missing, too short
or carries an unknown code, is discarded and the connection closed, in the
style of the worker MessageDecoder. SASL_MESSAGE_TOKEN and HBMessage frames are
decoded as before, so DIGEST and KERBEROS handshakes are unchanged. On the
client side a malformed control frame is reported as an IOException, so that
PacemakerClientHandler reconnects as it did before.

PacemakerServer.received() checks the message type before using it: anything
that is not an HBMessage is logged with its type and remote address and the
connection is closed.

The Pacemaker pipeline uses a new PacemakerServerHandler, a StormServerHandler
whose exceptionCaught closes the failing connection for any Exception and
keeps serving the other clients. Errors are still handed to
StormServerHandler. StormServerHandler itself, and with it the worker
messaging pipeline, is unchanged.

PacemakerServer gets a package-private close() used by the new tests.
Pacemaker was built to take worker heartbeat writes off ZooKeeper. Since
STORM-2693 workers heartbeat to their supervisor, which reports them to
Nimbus, so the daemon is only kept for backward compatibility; the metrics
documentation already describes it as deprecated.

Mark the Pacemaker server and client classes, the pacemaker state storage
and the pacemaker.* configuration keys as @deprecated, and add a notice to
docs/Pacemaker.md and to the storm pacemaker command help.
@reiabreu reiabreu added this to the 3.2.0 milestone Sep 12, 2026

@reiabreu reiabreu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Generated with the help of an LLM.)

Nice hardening of the Pacemaker frame handling — the layered decoder / received() type check / PacemakerServerHandler approach looks sound and is well tested, and the DIGEST/KERBEROS handshakes stay intact. A few minor notes:

  1. Test gap. Worth adding a SASL_MESSAGE_TOKEN test with a declared payload length exceeding its actual bytes, asserting the connection closes and the daemon survives — the control-frame malformed cases are well covered, but the SASL branch only has a well-formed case.

  2. docs/Pacemaker.md. "existing deployments should plan to move back to it" reads as migrating to Pacemaker (nearest antecedent). Suggest naming the target explicitly, e.g. "migrate away from Pacemaker to the default ZooKeeper state store (org.apache.storm.cluster.ZKStateStorageFactory)."

  3. PacemakerServer.close(). Appears to have no production caller (only test teardown) — if it's purely test support, a one-line comment saying so would clear up the intent.

Bound the declared payload length in SaslMessageToken.read and drop
malformed SASL_MESSAGE_TOKEN frames in the Pacemaker ThriftDecoder
(close the connection on the server, surface an IOException on the
client), mirroring the existing control-frame handling so a malformed
SASL frame can no longer take a large allocation path in the decoder.

Also: add regression tests for the SASL token path, reword the
Pacemaker deprecation note in docs, and document PacemakerServer.close()
as test-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants