HBASE-29640 Add metric for oldest procedure age

[Sigma-Ma]

JIRA: https://issues.apache.org/jira/browse/HBASE-29640

What changes were proposed in this pull request?

This pull request adds an oldestProcedureAge gauge to the existing Master,sub=Procedure metrics source. The gauge reports the age in milliseconds of the oldest unfinished master procedure, calculated from its submitted time.

The metric returns zero when there are no active procedures or when the master procedure executor is not yet available. Finished procedures are ignored, and negative values caused by clock adjustments are clamped to zero.

Why are the changes needed?

Operators currently have no direct metric for identifying how long the oldest master procedure has been active. Exposing this value makes it easier to detect stuck or unexpectedly long-running procedures and to configure monitoring and alerting around procedure execution.

How was this patch tested?

A new TestOldestProcedureMetrics test verifies:

• Selection of the oldest unfinished procedure.
• Exclusion of finished procedures.
• Export of the oldestProcedureAge gauge.
• The zero value when there are no active procedures.
• The zero value when the procedure executor is unavailable.
• Protection against negative values after clock adjustments.

mvn -pl hbase-server -am \
  -Dtest=TestOldestProcedureMetrics \
  -Dsurefire.failIfNoSpecifiedTests=false test

[Umeshkumar9414]

OldestAge from all the procedures doesn't give much info IMHO. As there are procedure like LockProcedure that might run for a longer time compared to other procedures (In other words avarage time taken differs based on the type of procedure.). It would be great if we can fetch or expose metric/log for oldestAge prcedure of each type.

[Sigma-Ma]

Good point, thanks. A LockProcedure can naturally live much longer, so the global max may hide a stuck procedure of another type.

I updated the patch to calculate the oldest age per procedure class and expose gauges such as oldestProcedureAge_LockProcedure and oldestProcedureAge_ServerCrashProcedure. I kept oldestProcedureAge as the overall max for the original use case.

The test now covers two procedure types, multiple procedures of the same type, and finished procedures.

[mnpoonia]

This is a much needed metrics. Thanks @Sigma-Ma