New serverless pattern - lambda-cloudwatch-otlp-metrics-sam

lambda-cloudwatch-otlp-metrics-sam/README.md

@@ -0,0 +1,121 @@
+# Send custom Lambda metrics to Amazon CloudWatch over OTLP
+
+This pattern records custom application metrics in an AWS Lambda function using the OpenTelemetry metrics API and sends them to Amazon CloudWatch over OTLP. The AWS Distro for OpenTelemetry (ADOT) Lambda layer runs a collector next to the function that signs each request with SigV4 and forwards it to the CloudWatch OTLP endpoint. The function makes no PutMetricData calls, writes no embedded metric format logs, and performs no request signing of its own.
+
+Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/lambda-cloudwatch-otlp-metrics-sam
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## How it works
+
+```
+   AWS Lambda                                  Amazon CloudWatch
+   +----------------------------+              +--------------------------+
+   |  handler.py                |   OTLP       |  OTLP metrics endpoint   |
+   |  OpenTelemetry metrics API |  over HTTP   |  monitoring.<region>     |
+   |            |               |  SigV4       |  .amazonaws.com          |
+   |            v               | -----------> |                          |
+   |  ADOT layer collector      |              |  queried with PromQL     |
+   +----------------------------+              +--------------------------+
+```
+
+- The function uses only the OpenTelemetry metrics API to record a counter and a histogram.
+- The ADOT Lambda layer starts a reduced OpenTelemetry collector that receives those metrics locally over OTLP.
+- The collector signs each request with SigV4 (signing name `monitoring`) and forwards it to the CloudWatch OTLP endpoint.
+- The only IAM permission the function needs is `cloudwatch:PutMetricData`.
+
+## Why OTLP rather than PutMetricData or embedded metric format
+
+- No per series charge. Embedded metric format bills log ingestion plus a monthly charge for every unique metric and dimension combination, which grows with cardinality. OTLP is billed on volume.
+- Up to 150 labels per metric, against the 30 dimensions allowed by PutMetricData, so you can attach much richer context.
+- No synchronous AWS API call inside the invocation, because the collector handles delivery.
+
+## Requirements
+
+- An AWS account with permissions to create Lambda functions and IAM roles.
+- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) v2.
+- [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html).
+- The CloudWatch OTLP endpoint enabled for your account and Region, as shown below.
+
+## Enable the CloudWatch OTLP endpoint
+
+The endpoint is turned off by default. Enable it once per account and Region, otherwise every export fails and no metrics appear.
+
+```bash
+REGION=us-east-1
+aws cloudwatch start-otel-enrichment --region $REGION
+aws observabilityadmin start-telemetry-enrichment --region $REGION
+
+# Both should report Running
+aws cloudwatch get-otel-enrichment --region $REGION
+aws observabilityadmin get-telemetry-enrichment-status --region $REGION
+```
+
+## Deployment
+
+```bash
+sam build
+sam deploy --guided
+#   - Stack Name  : otlp-metrics
+#   - AWS Region  : us-east-1
+#   - AdotLayerArn: keep the default for us-east-1, or change the Region in the ARN
+```
+
+The `AdotLayerArn` parameter is Region specific. The default points at the ADOT Python layer in us-east-1, so replace the Region in the ARN when deploying elsewhere.
+
+## Testing
+
+### 1. Invoke the function
+
+```bash
+FN=<the MetricsFunctionName output>
+echo {\"order\":{\"channel\":\"mobile\",\"country\":\"IN\",\"value\":129.50}} > event.json
+aws lambda invoke --function-name $FN --cli-binary-format raw-in-base64-out --payload file://event.json out.json
+cat out.json
+```
+
+Invoke it a few times so there is more than one data point.
+
+### 2. Confirm the metrics arrived
+
+These metrics are queried with PromQL. They do not appear in the classic CloudWatch metrics list, so `aws cloudwatch list-metrics` returns nothing for them. In the CloudWatch console open Metrics and switch the query editor to PromQL, then run:
+
+```promql
+{__name__="orders.processed"}
+```
+
+The metric names keep their dots, so they must be selected with `__name__` rather than written directly. You should see a value equal to the number of invocations, carrying the `order.channel` and `order.country` labels from the function plus resource labels such as `@resource.faas.name` and `@resource.service.name` added automatically by the layer.
+
+The same query is available over HTTP at `https://monitoring.<region>.amazonaws.com/api/v1/query`, signed with SigV4. Listing available names is a quick check:
+
+```
+GET https://monitoring.<region>.amazonaws.com/api/v1/label/__name__/values
+```
+
+Querying requires `cloudwatch:GetMetricData` and `cloudwatch:ListMetrics` on the caller, which is separate from the permission the function needs to publish.
+
+## Notes
+
+- The collector configuration deliberately declares no processors. The collector build inside the ADOT Lambda layer is compiled without them, so naming one such as `batch` stops the collector from starting and nothing is exported.
+- Use `metrics_endpoint` in the exporter, not `endpoint`. The exporter appends the signal path to `endpoint`, so setting `endpoint` to the full metrics URL produces `/v1/metrics/v1/metrics` and every export fails with HTTP 404.
+- The function flushes the meter provider before returning. Lambda freezes the execution environment as soon as the handler returns, so waiting for the next periodic export would lose the data.
+- The exporter is named `otlp_http`. Older collector builds use the `otlphttp` alias, which now logs a deprecation warning.
+
+## Cleanup
+
+```bash
+sam delete
+```
+
+Optionally turn the endpoint back off:
+
+```bash
+aws cloudwatch stop-otel-enrichment --region $REGION
+aws observabilityadmin stop-telemetry-enrichment --region $REGION
+```
+
+----
+
+Author: Manish S
+
+Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

lambda-cloudwatch-otlp-metrics-sam/example-pattern.json

@@ -0,0 +1,49 @@
+{
+    "title": "Send custom Lambda metrics to Amazon CloudWatch over OTLP",
+    "description": "Record custom application metrics in an AWS Lambda function using the OpenTelemetry metrics API and forward them to Amazon CloudWatch over OTLP. The AWS Distro for OpenTelemetry Lambda layer signs each request with SigV4, so the function makes no PutMetricData calls and writes no embedded metric format logs.",
+    "language": "Python",
+    "level": "300",
+    "framework": "AWS SAM",
+    "patternArch": {
+        "icon1": { "x": 30, "y": 50, "service": "lambda", "label": "AWS Lambda with ADOT layer" },
+        "icon2": { "x": 70, "y": 50, "service": "cloudwatch", "label": "Amazon CloudWatch OTLP endpoint" },
+        "line1": { "from": "icon1", "to": "icon2", "label": "OTLP over HTTP, SigV4 signed" }
+    },
+    "introBox": {
+        "headline": "How it works",
+        "text": [
+            "Teams that need custom business metrics from AWS Lambda have had two options, and both cost something. Calling PutMetricData adds a synchronous AWS API call to the invocation and caps you at 30 dimensions per metric. Writing embedded metric format logs avoids the API call but bills you for log ingestion plus a monthly charge for every unique metric and dimension combination, which gets expensive as cardinality grows. Amazon CloudWatch now accepts OpenTelemetry metrics directly over OTLP, and this pattern shows how to send them from a Lambda function without either of those trade-offs.",
+            "The template deploys a Python function with the AWS Distro for OpenTelemetry Lambda layer attached and a small collector configuration bundled with the function code. The layer starts a reduced OpenTelemetry collector alongside the function. The function records metrics using only the OpenTelemetry metrics API, the collector receives them over local OTLP, signs each request with SigV4, and forwards them to the CloudWatch OTLP endpoint. The function code contains no AWS SDK calls for metrics and performs no request signing, and the only permission it needs is cloudwatch PutMetricData.",
+            "Two details make the OTLP path attractive beyond convenience. There is no per series charge, so wide, high cardinality metrics are billed on volume rather than on the number of unique dimension combinations. The endpoint also accepts up to 150 labels per metric against the 30 dimensions allowed by PutMetricData, so you can attach far richer context to each data point.",
+            "The pattern captures the practical details that are easy to get wrong. The collector build inside the ADOT Lambda layer is compiled without processors, so naming one such as batch stops the collector from starting and nothing is exported at all. Lambda freezes the execution environment the moment the handler returns, so the function flushes the meter provider before returning rather than waiting for the next periodic export. Metrics sent this way are queried with PromQL instead of appearing in the classic CloudWatch metrics console. The OTLP endpoint is also turned off by default and has to be enabled once per account and Region, which the README covers.",
+            "Good fits include business and product metrics such as orders, signups, or revenue, per tenant metrics in multi tenant applications where cardinality is high, and any team already standardized on OpenTelemetry that wants CloudWatch as the backend without running a separate collector gateway."
+        ]
+    },
+    "gitHub": {
+        "template": {
+            "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-cloudwatch-otlp-metrics-sam",
+            "templateURL": "serverless-patterns/lambda-cloudwatch-otlp-metrics-sam",
+            "projectFolder": "lambda-cloudwatch-otlp-metrics-sam",
+            "templateFile": "template.yaml"
+        }
+    },
+    "resources": {
+        "headline": "Additional resources",
+        "bullets": [
+            { "text": "Send OpenTelemetry metrics to Amazon CloudWatch", "link": "https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OTLP-metrics.html" },
+            { "text": "AWS Distro for OpenTelemetry Lambda support", "link": "https://aws-otel.github.io/docs/getting-started/lambda" },
+            { "text": "Query CloudWatch metrics with PromQL", "link": "https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-PromQL.html" },
+            { "text": "OpenTelemetry metrics API for Python", "link": "https://opentelemetry.io/docs/languages/python/instrumentation/" }
+        ]
+    },
+    "deploy": { "text": [ "sam build", "sam deploy --guided" ] },
+    "testing": {
+        "headline": "Testing",
+        "text": [ "Enable the CloudWatch OTLP endpoint for your account and Region, invoke the function a few times, then query the metric with PromQL to confirm the data points arrived. See the README for detailed instructions." ]
+    },
+    "cleanup": {
+        "headline": "Cleanup",
+        "text": [ "1. Delete the stack: <code>sam delete</code>." ]
+    },
+    "authors": [ { "name": "Manish S", "image": "", "bio": "AWS Support Engineer, trying to build things", "linkedin": "https://www.linkedin.com/in/manish-s-84199221b", "twitter": "" } ]
+}

lambda-cloudwatch-otlp-metrics-sam/src/collector.yaml

@@ -0,0 +1,40 @@
+# Configuration for the reduced ADOT collector that ships inside the Lambda layer.
+#
+# Two things here are easy to get wrong:
+#
+# 1. There are deliberately no processors. The collector build in the ADOT Lambda
+#    layer is compiled without processors, so naming one (for example batch) stops
+#    the collector extension from starting and nothing is exported at all.
+#
+# 2. Use metrics_endpoint, not endpoint. The exporter appends the signal path to
+#    endpoint, so setting endpoint to the full metrics URL produces a doubled path
+#    such as /v1/metrics/v1/metrics and every export fails with HTTP 404.
+#    metrics_endpoint is taken as the exact URL.
+
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: "localhost:4318"
+
+extensions:
+  # Signs every outbound request with SigV4. The CloudWatch OTLP endpoint is an
+  # authenticated AWS API and its signing name is monitoring.
+  sigv4auth:
+    region: "${env:AWS_REGION}"
+    service: "monitoring"
+
+exporters:
+  # Named otlp_http here. Older collector builds use the otlphttp alias instead.
+  otlp_http:
+    metrics_endpoint: "https://monitoring.${env:AWS_REGION}.amazonaws.com/v1/metrics"
+    auth:
+      authenticator: sigv4auth
+    compression: gzip
+
+service:
+  extensions: [sigv4auth]
+  pipelines:
+    metrics:
+      receivers: [otlp]
+      exporters: [otlp_http]

lambda-cloudwatch-otlp-metrics-sam/src/handler.py

@@ -0,0 +1,51 @@
+"""Record custom application metrics and send them to CloudWatch over OTLP.
+
+This function only uses the OpenTelemetry metrics API. The AWS Distro for
+OpenTelemetry Lambda layer starts a collector that signs each request with SigV4
+and forwards it to the CloudWatch OTLP endpoint, so the application code never
+calls PutMetricData, never writes embedded metric format logs, and does no AWS
+request signing of its own.
+"""
+from opentelemetry import metrics
+
+meter = metrics.get_meter("orders")
+
+orders_processed = meter.create_counter(
+    name="orders.processed",
+    unit="1",
+    description="Number of orders processed.",
+)
+
+order_value = meter.create_histogram(
+    name="orders.value",
+    unit="USD",
+    description="Distribution of order values.",
+)
+
+
+def handler(event, context):
+    order = (event or {}).get("order") or {}
+
+    # Dimensions are plain OpenTelemetry attributes. The OTLP endpoint accepts up to
+    # 150 labels per metric, well above the 30 dimensions allowed by PutMetricData.
+    attributes = {
+        "order.channel": order.get("channel", "web"),
+        "order.country": order.get("country", "IN"),
+    }
+
+    value = float(order.get("value", 49.99))
+
+    orders_processed.add(1, attributes)
+    order_value.record(value, attributes)
+
+    # Lambda freezes the execution environment as soon as the handler returns, so
+    # flush now rather than waiting for the next periodic export, which would
+    # otherwise be lost.
+    provider = metrics.get_meter_provider()
+    if hasattr(provider, "force_flush"):
+        provider.force_flush()
+
+    return {
+        "recorded": {"orders.processed": 1, "orders.value": value},
+        "attributes": attributes,
+    }

lambda-cloudwatch-otlp-metrics-sam/src/requirements.txt

@@ -0,0 +1,2 @@
+# The AWS Distro for OpenTelemetry Lambda layer provides the OpenTelemetry API and
+# SDK at runtime, so no packages need to be bundled with the function.

lambda-cloudwatch-otlp-metrics-sam/template.yaml

@@ -0,0 +1,78 @@
+AWSTemplateFormatVersion: "2010-09-09"
+Transform: AWS::Serverless-2016-10-31
+Description: >
+  Send custom application metrics from AWS Lambda to Amazon CloudWatch over OTLP.
+  The AWS Distro for OpenTelemetry Lambda layer runs a collector that signs each
+  request with SigV4 and forwards OTLP metrics to the CloudWatch OTLP endpoint, so
+  the function code never calls PutMetricData and writes no embedded metric logs.
+  (lambda-cloudwatch-otlp-metrics-sam)
+
+Parameters:
+  AdotLayerArn:
+    Type: String
+    Default: arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-32-0:7
+    Description: >
+      ARN of the AWS Distro for OpenTelemetry Python Lambda layer. This ARN is Region
+      specific, so change the Region in the value when you deploy outside us-east-1.
+  ServiceName:
+    Type: String
+    Default: orders-service
+    Description: Value reported as the service.name resource attribute on every metric.
+
+Resources:
+  MetricsFunctionRole:
+    Type: AWS::IAM::Role
+    Properties:
+      AssumeRolePolicyDocument:
+        Version: "2012-10-17"
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service: lambda.amazonaws.com
+            Action: sts:AssumeRole
+      ManagedPolicyArns:
+        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
+      Policies:
+        - PolicyName: PublishOtlpMetrics
+          PolicyDocument:
+            Version: "2012-10-17"
+            Statement:
+              # cloudwatch:PutMetricData is the only permission the OTLP metrics
+              # endpoint requires. It does not support resource level permissions.
+              - Sid: PublishMetrics
+                Effect: Allow
+                Action: cloudwatch:PutMetricData
+                Resource: "*"
+
+  MetricsFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      FunctionName: !Sub "${AWS::StackName}-otlp-metrics"
+      CodeUri: src/
+      Handler: handler.handler
+      Runtime: python3.13
+      Architectures:
+        - x86_64
+      Timeout: 30
+      MemorySize: 256
+      Role: !GetAtt MetricsFunctionRole.Arn
+      Layers:
+        - !Ref AdotLayerArn
+      Environment:
+        Variables:
+          # Starts the ADOT auto instrumentation wrapper and the bundled collector.
+          AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
+          OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml
+          OTEL_METRICS_EXPORTER: otlp
+          OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
+          OTEL_TRACES_EXPORTER: none
+          OTEL_LOGS_EXPORTER: none
+          OTEL_SERVICE_NAME: !Ref ServiceName
+
+Outputs:
+  MetricsFunctionName:
+    Description: Name of the Lambda function that records the custom metrics.
+    Value: !Ref MetricsFunction
+  ServiceNameValue:
+    Description: service.name attribute reported with every metric.
+    Value: !Ref ServiceName