[yeya24]

What this PR does:

The Prometheus notifier the ruler embeds does not retry. sendOne calls the Do
option once, and the caller only logs the error and adds to the error counter
before moving on to the next batch, so a notification the Alertmanager fails to
accept is dropped: its alert is not delivered until the rule group is evaluated
again.

Two of those failures recover on a second attempt, and both are routine when the
ruler keeps connections alive to the Alertmanager:

• The Alertmanager closes a connection it considers idle just as the ruler reuses
  it for a POST, and the ruler logs error occurred while sending alerts ... EOF.
  The notifier client's IdleConnTimeout is independent of the server's idle
  timeout, so the race repeats for as long as both are running. This is Ruler reports an error EOF when sending alert to alertmanager #4958,
  also reported in cortex ruler send alert to alertmanager ERROR EOF #3009 and asked about in Guidance on Ruler -> AlertManager intermittent reachability #4355.
• A Cortex Alertmanager that received SIGTERM replies 503 for as long as its ring
  final_sleep lasts, because MultitenantAlertmanager.ServeHTTP returns early
  while the service is not Running (pkg/alertmanager/multitenant.go) and does
  not forward the alerts to another replica. A keep-alive connection pinned to
  that instance keeps getting 503 until it is replaced, even though healthy
  replicas are behind the same address.

This PR retries such a notification instead of dropping it:

• A transport error, except one caused by the context being cancelled or its
  deadline being exceeded, and a 5xx response are retried. A 4xx response is not:
  the Alertmanager rejects the request however many times it is sent.
• The idle connections of the tenant's client are closed before a retry, so the
  attempt opens a new connection. That is also how it reaches another instance
  when the Alertmanagers are behind a load balancer.
• The request body is rebuilt from GetBody on every attempt. POST /api/v2/alerts
  is idempotent, the Alertmanager merges the alerts by fingerprint.
• The retries are bounded by the context of the notification, which carries
  -ruler.notification-timeout, and they delay the other notifications of the same
  tenant, which is why the flag description asks for a small number.

Configuration

Disabled by default, so nothing changes unless it is configured:

Flag	Default
-ruler.notification.backoff-retries	0, the retries are disabled
-ruler.notification.backoff-min-period	100ms
-ruler.notification.backoff-max-period	1s

A new per-user counter, cortex_ruler_notification_retries_total, counts the
retried notifications.

One thing worth a look in review: the config reuses backoff.Config, but
max_retries = 0 disables the retries here instead of making them infinite, so
that the default does not change the behaviour of the ruler. transport.Retry
reads a retry count the same way. That is why the three flags are registered with
their own descriptions instead of calling backoff.Config.RegisterFlagsWithPrefix,
which would default the count to 10 and document it as the generic backoff count.
Happy to move to a separate -ruler.notification-retry-enabled flag and leave
backoff.Config with its usual semantics if that reads better.

Which issue(s) this PR fixes:
Related to #4958

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
• docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags