Drop unexpected Pacemaker frames instead of terminating the daemon; mark Pacemaker as deprecated - #9084
Drop unexpected Pacemaker frames instead of terminating the daemon; mark Pacemaker as deprecated#9084rzo1 wants to merge 3 commits into
Conversation
The Pacemaker server decodes a CONTROL_MESSAGE frame into a ControlMessage. With no SASL handler installed (pacemaker.auth.method NONE), or once the SASL handler has forwarded it upstream, the object reached PacemakerServer.received(), which cast it to HBMessage. The resulting ClassCastException reached StormServerHandler.exceptionCaught, which treats anything but an IOException as fatal and exits the JVM. A malformed request (undecodable thrift payload, a control frame without payload or with an unknown code, a request without data) took the same path. The server-side ThriftDecoder now accepts only the control message a Pacemaker client sends, SASL_TOKEN_MESSAGE_REQUEST, which starts the DIGEST handshake. Any other control frame, and control frames whose payload is missing, too short or carries an unknown code, is discarded and the connection closed, in the style of the worker MessageDecoder. SASL_MESSAGE_TOKEN and HBMessage frames are decoded as before, so DIGEST and KERBEROS handshakes are unchanged. On the client side a malformed control frame is reported as an IOException, so that PacemakerClientHandler reconnects as it did before. PacemakerServer.received() checks the message type before using it: anything that is not an HBMessage is logged with its type and remote address and the connection is closed. The Pacemaker pipeline uses a new PacemakerServerHandler, a StormServerHandler whose exceptionCaught closes the failing connection for any Exception and keeps serving the other clients. Errors are still handed to StormServerHandler. StormServerHandler itself, and with it the worker messaging pipeline, is unchanged. PacemakerServer gets a package-private close() used by the new tests.
Pacemaker was built to take worker heartbeat writes off ZooKeeper. Since STORM-2693 workers heartbeat to their supervisor, which reports them to Nimbus, so the daemon is only kept for backward compatibility; the metrics documentation already describes it as deprecated. Mark the Pacemaker server and client classes, the pacemaker state storage and the pacemaker.* configuration keys as @deprecated, and add a notice to docs/Pacemaker.md and to the storm pacemaker command help.
reiabreu
left a comment
There was a problem hiding this comment.
(Generated with the help of an LLM.)
Nice hardening of the Pacemaker frame handling — the layered decoder / received() type check / PacemakerServerHandler approach looks sound and is well tested, and the DIGEST/KERBEROS handshakes stay intact. A few minor notes:
-
Test gap. Worth adding a
SASL_MESSAGE_TOKENtest with a declared payload length exceeding its actual bytes, asserting the connection closes and the daemon survives — the control-frame malformed cases are well covered, but the SASL branch only has a well-formed case. -
docs/Pacemaker.md. "existing deployments should plan to move back to it" reads as migrating to Pacemaker (nearest antecedent). Suggest naming the target explicitly, e.g. "migrate away from Pacemaker to the default ZooKeeper state store (org.apache.storm.cluster.ZKStateStorageFactory)." -
PacemakerServer.close(). Appears to have no production caller (only test teardown) — if it's purely test support, a one-line comment saying so would clear up the intent.
Bound the declared payload length in SaslMessageToken.read and drop malformed SASL_MESSAGE_TOKEN frames in the Pacemaker ThriftDecoder (close the connection on the server, surface an IOException on the client), mirroring the existing control-frame handling so a malformed SASL frame can no longer take a large allocation path in the decoder. Also: add regression tests for the SASL token path, reword the Pacemaker deprecation note in docs, and document PacemakerServer.close() as test-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Robustness: drop unexpected Pacemaker frames
A
CONTROL_MESSAGEframe is decoded by the PacemakerThriftDecoderinto aControlMessage. Withpacemaker.auth.method: NONE, or after the SASL handler forwarded it upstream, it reachedPacemakerServer.received(), which cast it toHBMessage. TheClassCastExceptionended inStormServerHandler.exceptionCaught, which treats anything but anIOExceptionas fatal and exits the JVM. Malformed requests took the same path.This mirrors what was recently done for the worker path (#9076):
ThriftDecoderonly acceptsSASL_TOKEN_MESSAGE_REQUEST(the control message a client sends to start the DIGEST handshake). Other control frames, and control frames with a missing/short/unknown payload, are discarded and the connection closed.SASL_MESSAGE_TOKENandHBMessageframes decode as before, so DIGEST and KERBEROS handshakes are unchanged. On the client side a malformed control frame surfaces as anIOException, soPacemakerClientHandlerreconnects as before.PacemakerServer.received()checks the message type; anything other than anHBMessageis logged and the connection closed.PacemakerServerHandlerthat closes only the failing connection on anException;Errors still go toStormServerHandler.StormServerHandlerand the worker messaging pipeline are unchanged.Tests: new
ThriftDecoderTestandPacemakerServerTest.Mark Pacemaker as deprecated
Since STORM-2693 workers heartbeat to their supervisor, which reports them to Nimbus, so Pacemaker is only kept for backward compatibility (
docs/ClusterMetrics.mdalready says so). This marks the Pacemaker server/client classes, the pacemaker state storage and thepacemaker.*config keys as@Deprecated, and adds a notice todocs/Pacemaker.mdand thestorm pacemakerhelp.