Skip to content

#87 - Alarming construct to streamline alarming setup - #88

Open
dario-fazio wants to merge 16 commits into
mainfrom
#87-cdk-alarming
Open

dario-fazio wants to merge 16 commits into
mainfrom
#87-cdk-alarming

Conversation

@dario-fazio

Copy link
Copy Markdown
Contributor

No description provided.

@dario-fazio dario-fazio linked an issue Sep 18, 2026 that may be closed by this pull request
@dario-fazio
dario-fazio marked this pull request as ready for review September 29, 2026 14:22
Copilot AI balanced review requested due to automatic review settings September 29, 2026 14:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Invocation permissions, filtering, pagination, Slack error handling, and batch delivery contain reliability defects that can silently prevent notifications.

Review effort: Balanced
Findings: 1 High severity · 4 Medium severity

Open (5)
What changed in this PR

Introduces a reusable CDK alarming construct with bundled Lambda functions that forward CloudWatch alarms and error logs to Slack.

Changes:

  • Adds alarm, log-subscription, fallback-email, and Slack notification infrastructure.
  • Adds bundled Slack-forwarding handlers with tests and supporting utilities.
  • Updates build, dependency, lint, and TypeScript configuration.
File Description
tsconfig.spec.json Adds workspace import aliases.
tsconfig.lint.json Adds lint-time import aliases.
packages/​cdk-utilities/​src/​scripts/​esbuild.script.ts Bundles notification Lambdas.
packages/​cdk-utilities/​src/​lib/​utils/​slack-messages.utils.ts Adds Slack text helpers.
packages/​cdk-utilities/​src/​lib/​utils/​cloudwatch.utils.ts Adds log parsing and URL helpers.
packages/​cdk-utilities/​src/​lib/​models/​slack-notification.model.ts Defines Slack attachment payloads.
packages/​cdk-utilities/​src/​lib/​models/​custom-cloudwatch-alarm-message.model.ts Defines parsed alarm messages.
packages/​cdk-utilities/​src/​lib/​lambda-function-name.enum.ts Registers bundled handlers.
packages/​cdk-utilities/​src/​lib/​functions/​publish-error-logs-to-slack-fn.ts Forwards error logs to Slack.
packages/​cdk-utilities/​src/​lib/​functions/​publish-error-logs-to-slack-fn.spec.ts Tests error-log forwarding.
packages/​cdk-utilities/​src/​lib/​functions/​publish-alarm-to-slack-fn.ts Forwards alarms to Slack.
packages/​cdk-utilities/​src/​lib/​functions/​publish-alarm-to-slack-fn.spec.ts Tests alarm forwarding.
packages/​cdk-utilities/​src/​lib/​alarming.construct.ts Implements the alarming construct.
packages/​cdk-utilities/​src/​index.ts Exports the construct.
packages/​cdk-utilities/​package.json Updates build and dependencies.
packages/​cdk-utilities/​eslint.config.js Allows console use in scripts.
package-lock.json Locks added dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/cdk-utilities/src/lib/alarming.construct.ts Outdated
addPermissions: props.addPermissions ?? !this.grantCloudWatchLogsInvokeBroadly,
}),
filterPattern: {
logPatternString: props.filterPattern ?? '{ $.level = "ERROR" }',
attachment.author_name = errorLogs.errorType
attachment.title = ':bookmark_tabs: CloudWatch Log events:'
attachment.mrkdwn_in = ['text']
attachment.text = errorLogs.text.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;')
Comment thread packages/cdk-utilities/src/lib/functions/publish-error-logs-to-slack-fn.ts Outdated
Comment on lines +230 to +232
await Promise.all(
messages.map((message) => postSlackMessage(message.blocks, message.attachments, slackWebhookEndpoint)),
)
Copilot AI balanced review requested due to automatic review settings October 1, 2026 14:27
 - @shiftcode/cdk-utilities@1.1.0-pr87.3

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Webhook failures, raw log parsing, unbounded delivery, and cross-stack references can cause lost notifications or deployment failures.

Review effort: Balanced
Findings: 4 High severity · 4 Medium severity

Open (8)
Previously missed (4)

In code that hasn't changed since last review

Medium severity Format timestamps instead of displaying epoch values

packages/​cdk-utilities/​src/​lib/​functions/​publish-alarm-to-slack-fn.ts:162

Date#setTime returns the numeric epoch in milliseconds, so each Slack log entry displays a value such as 1700000000001 instead of a readable event time. Format the date value before embedding it.

Medium severity Preserve raw messages when JSON parsing fails

packages/​cdk-utilities/​src/​lib/​functions/​publish-error-logs-to-slack-fn.ts:122

Both branches call JSON.parse without the fallback promised above. Any non-JSON message matched by a custom subscription filter aborts the entire CloudWatch Logs batch, so the documented “any other format still works” behavior does not work; preserve the raw text when parsing fails.

Medium severity Classify standard Lambda module-load failures correctly

packages/​cdk-utilities/​src/​lib/​utils/​cloudwatch.utils.ts:22

The standard Lambda module-load failures contain Runtime.ImportModuleError and/or Cannot find module; they do not contain “missing on module”. As a result, these failures fall through to the general classifier and are mislabeled instead of being reported as configuration errors.

Low severity Use the noun “implementation” in the comment

packages/​cdk-utilities/​src/​scripts/​esbuild.script.ts:27

Use “implementation” here; “implement” is a verb and makes the explanatory comment ungrammatical.

Comment on lines +194 to +196
const topic = Topic.fromTopicAttributes(scope, `${props.id}Topic`, {
topicArn: this.alarmTopic.topicArn,
})
Comment thread packages/cdk-utilities/src/lib/alarming.construct.ts Outdated
Copilot AI balanced review requested due to automatic review settings October 1, 2026 14:32
@michaellieberherr

Copy link
Copy Markdown
Member

@dario-fazio : I reviewed the esbuild setup and added a small optimisation to reduce the Lambda bundle size. I haven’t checked if the potential issues flagged by Copilot are valid. Otherwise it looks good to me.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Notification delivery, default filtering, batching, and cross-stack behavior contain unresolved correctness and reliability problems.

Review effort: Balanced
Findings: 4 High severity · 4 Medium severity

Open (8)
Previously missed (3)

In code that hasn't changed since last review

Medium severity Date#setTime return value produces an unreadable timestamp

packages/​cdk-utilities/​src/​lib/​functions/​publish-alarm-to-slack-fn.ts:162

Date#setTime returns the epoch value as a number, so Slack displays values such as 1700000000001 instead of a readable event time. Format the constructed date rather than interpolating the setter's return value.

This issue also appears on line 188 of the same file.

Medium severity Malformed event JSON aborts the entire log batch

packages/​cdk-utilities/​src/​lib/​functions/​publish-error-logs-to-slack-fn.ts:122

A non-JSON event makes JSON.parse throw and fails the entire CloudWatch Logs batch, despite this handler and its documentation supporting plaintext/custom filter patterns. Fall back to the raw log message when parsing fails so one event cannot block every notification in the batch.

Medium severity Missing-dependency errors are not classified as configuration errors

packages/​cdk-utilities/​src/​lib/​utils/​cloudwatch.utils.ts:22

This phrase does not match Lambda's missing-dependency errors (Runtime.ImportModuleError: Cannot find module/package ...), so those events are classified only as generic errors rather than configuration errors. Match the actual runtime messages.

- escaping for slack (alarm message)
- handle non-successful fetch responses
Copilot AI balanced review requested due to automatic review settings October 5, 2026 08:24
 - @shiftcode/cdk-utilities@1.1.0-pr87.4

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Several event-handling and infrastructure defects can lose alerts or break deployments.

Review effort: Balanced
Findings: 4 High severity · 3 Medium severity

Open (7)
Resolved since last review (2)
Previously missed (3)

In code that hasn't changed since last review

Medium severity Avoid token-based Lambda ARN cross-stack coupling

packages/​cdk-utilities/​src/​lib/​alarming.construct.ts:213

this.publishErrorLogsToSlackFunction.functionArn remains a token owned by the construct stack; importing that token into scope still creates the export/import coupling this code says it avoids. Use an ARN built from a retained stable function name and environment, or keep the direct reference and document the dependency.

Medium severity Format event timestamps instead of returning epoch numbers

packages/​cdk-utilities/​src/​lib/​functions/​publish-alarm-to-slack-fn.ts:163

Date#setTime returns the epoch millisecond number, so Slack receives values such as 1700000000000 rather than a readable event time. Format the constructed date instead.

This issue also appears in the following locations of the same file:

  • line 189
  • line 194
Medium severity Handle CloudWatch control messages and empty logEvents

packages/​cdk-utilities/​src/​lib/​functions/​publish-error-logs-to-slack-fn.ts:115

CloudWatch Logs can send CONTROL_MESSAGE payloads without data events to verify the destination. Calling .map unconditionally makes that invocation fail and retry; return successfully for control messages or an absent/empty logEvents array.

This issue also appears in the following locations of the same file:

  • line 131
  • line 141

Comment thread packages/cdk-utilities/package.json Outdated
Copilot AI balanced review requested due to automatic review settings October 5, 2026 08:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Notification delivery, log handling, CDK compatibility, and cross-stack decoupling contain confirmed correctness and reliability issues.

Review effort: Balanced
Findings: 6 High severity · 3 Medium severity

Open (9)
Previously missed (3)

In code that hasn't changed since last review

Medium severity Avoid cross-stack token dependency for function ARN

packages/​cdk-utilities/​src/​lib/​alarming.construct.ts:213

This import still receives functionArn as a token owned by the construct's stack, so a subscription created in another stack retains a CloudFormation export/import dependency despite the comment. Import from a stable physical ARN derived or configured in the consuming stack instead.

Medium severity Format event timestamps as readable date strings

packages/​cdk-utilities/​src/​lib/​functions/​publish-alarm-to-slack-fn.ts:163

Date#setTime returns the numeric epoch value, so Slack displays values such as 1700000000001 rather than a readable event time. Format the timestamp as a date string.

Medium severity Match actual Lambda module import failure text

packages/​cdk-utilities/​src/​lib/​utils/​cloudwatch.utils.ts:22

This pattern does not match the standard Lambda import failure text (Cannot find module ...), so those failures fall through to the generic type instead of being labeled as configuration errors. Match the actual runtime phrase here.

Comment on lines +294 to +296
runtime: Runtime.NODEJS_24_X,
// the bundled lambda code uses the params-and-secrets layer to fetch the Slack webhook URL from SSM
paramsAndSecrets: ParamsAndSecretsLayerVersion.fromVersion(ParamsAndSecretsVersions.V1_0_103, {
…sh for length constraints

- raise cdk version to match used API
Copilot AI balanced review requested due to automatic review settings October 5, 2026 08:51
 - @shiftcode/cdk-utilities@1.1.0-pr87.5

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

: ''
const text = errorLogs
.map((fle) => {
const eventTime = `${new Date().setTime(fle.timestamp || 0)}`
Comment on lines +139 to +144
const editorString = [
'fields @timestamp, level, message, data, logger',
`| filter @requestId = '${parsedLambdaLog?.requestId}'`,
'sort @timestamp desc',
'limit 100',
].join('\n')
slackErrorTypeMessage: ':thermometer: Out of memory error',
},
missingModule: {
isTypeRegex: /\b(missing on module)\b/i,
Copilot AI balanced review requested due to automatic review settings October 5, 2026 08:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Alert filtering, delivery, parsing, concurrency, and IAM issues can lose notifications or expose unrelated logs.

Review effort: Balanced
Findings: 4 High severity · 6 Medium severity

Open (10)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Error regex excludes structured logger errors

packages/​cdk-utilities/​src/​lib/​utils/​cloudwatch.utils.ts:28

General errors emitted by @shiftcode/logger are JSON such as {"level":"ERROR",...}, which this regex does not match. Consequently, alarm enrichment filters out the repository's normal structured error logs and reports “no error logs found.” Include the structured level field in the general-error matcher.

Comment on lines +329 to +333
new PolicyStatement({
effect: Effect.ALLOW,
actions: ['logs:FilterLogEvents'],
resources: ['*'], // TODO be more restrictive here
}),
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CDK Construct for Alarming

4 participants