Skip to content

Update and Improve the CDN Publishing Workflow #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cabutlermit wants to merge 8 commits into from
Closed

Update and Improve the CDN Publishing Workflow #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4b1c723
Update and Improve CDN Publish
cabutlermit May 22, 2026
f011a62
Enhance the CDN Publishing Workflow
cabutlermit May 26, 2026
98f7d25
Ensure Backwards Compatibility
cabutlermit May 29, 2026
2a4e2ba
Validate and Construct sync Parameters
cabutlermit May 29, 2026
530239b
Address GitHub Copilot Review
cabutlermit Jun 12, 2026
b757be6
Additional Fixes per GitHub Copilot Review
cabutlermit Jun 18, 2026
4565c53
More Fixes for GitHub Copilot Review
cabutlermit Jun 29, 2026
e97398d
WIP: More fixes for Copilot
cabutlermit Jun 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
322 changes: 249 additions & 73 deletions .github/workflows/cdn-shared-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,47 @@ on:
workflow_call:
inputs:
AWS_REGION:
required: true
description: "Region for AWS resources."
required: false
type: string
GHA_ROLE:
required: true
default: us-east-1
Comment thread
qltysh[bot] marked this conversation as resolved.
DOMAIN:
description: "Indicates the standard CDN or a custom domain for the URL to the CDN (only standard or custom accepted)."
required: false
type: string
default: standard
ENVIRONMENT:
description: "The AWS environment where the resources will be deployed."
required: true
type: string
S3URI:
GHA_ROLE:
description: "The IAM Role linked to the OIDC connection."
required: true
type: string
DOMAIN:
S3URI:
description: "Legacy (deprecated) full S3 URI for the sync target in AWS."
required: false
type: string
default: standard
SOURCE_PATH:
description: "The path in the caller repository containing the files to sync to the S3 bucket."
required: false
type: string
default: .
SYNC_PARAMS:
description: "Additional parameters for the aws s3 sync command, specific to the caller repository."
required: false
type: string
TARGET_PATH:
description: "The prefix in the S3 bucket to which the repository files should be synced (must start with slash)."
required: false
type: string
default: /


permissions:
# These are the minimum permissions to allow for OIDC connection to AWS
id-token: write
contents: read

# Set defaults
defaults:
Expand All @@ -31,97 +54,250 @@ defaults:

jobs:
publish:
name: Publish content to CDN
# Start with validating the inputs from the caller workflow and prepping
# environment variables for the synchronization job.
name: Publish
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read

steps:
- uses: actions/checkout@v4

- name: DEV Configure AWS credentials
# Only run this step if the environment is "dev"
if: ${{ inputs.ENVIRONMENT == 'dev' }}
uses: aws-actions/configure-aws-credentials@v4
- uses: actions/checkout@v6
Comment thread
qltysh[bot] marked this conversation as resolved.
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCT_DEV }}:role/${{ inputs.GHA_ROLE }}
aws-region: ${{ inputs.AWS_REGION }}
persist-credentials: false

- name: STAGE Configure AWS credentials
# Only run this step if the environment is "stage"
if: ${{ inputs.ENVIRONMENT == 'stage' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCT_STAGE }}:role/${{ inputs.GHA_ROLE }}
aws-region: ${{ inputs.AWS_REGION }}
- name: Validate
# Verify that the DOMAIN & ENVIRONMENT inputs are using the correct
# values. Verify that the SOURCE_PATH and TARGET_PATH inputs are
# formatted correctly and ensure this supports legacy caller workflows.
# Validate the SYNC_PARAMS input to only allow `--exclude` and
# `--include` parameters. Verify that the GHA_ROLE is properly formatted
# for an IAM Role name (using a more tightly restricted regex pattern
# than AWS that matches our naming scheme: letters, numbers, hyphens,
# underscores).
Comment thread
cabutlermit marked this conversation as resolved.
id: validate
env:
DOMAIN: ${{ inputs.DOMAIN }}
ENVIRONMENT: ${{ inputs.ENVIRONMENT }}
GHA_ROLE: ${{ inputs.GHA_ROLE }}
S3URI: ${{ inputs.S3URI }}
SOURCE_PATH: ${{ inputs.SOURCE_PATH }}
SYNC_PARAMS: ${{ inputs.SYNC_PARAMS }}
TARGET_PATH: ${{ inputs.TARGET_PATH }}
WORKSPACE_PATH: ${{ github.workspace }}
run: |
# Only "standard" or "custom" are allowed values for DOMAIN
case "$DOMAIN" in
standard|custom)
echo "Valid DOMAIN=$DOMAIN input, proceed."
;;
*)
echo "Invalid DOMAIN=$DOMAIN input, exiting."
exit 1
;;
esac

- name: PROD Configure AWS credentials
# Only run this step if the environment is "prod"
if: ${{ inputs.ENVIRONMENT == 'prod' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCT_PROD }}:role/${{ inputs.GHA_ROLE }}
aws-region: ${{ inputs.AWS_REGION }}
# Only "dev", "stage", or "prod" are allowed values for ENVIRONMENT
case "$ENVIRONMENT" in
dev|stage|prod)
echo "Valid ENVIRONMENT=$ENVIRONMENT input, proceed."
;;
*)
echo "Invalid ENVIRONMENT=$ENVIRONMENT input, exiting."
exit 1
;;
esac

- name: Sync custom domain CDN S3 content
# Only run this step if this is custom domain content (e.g., a folder at the root of bucket)
if: ${{ inputs.DOMAIN == 'custom' }}
run: |
if [ '${{ inputs.SYNC_PARAMS }}' != '' ]; then
aws s3 sync . ${{ inputs.S3URI }} --delete --exclude ".github/*" --exclude ".git/*" --exclude ".gitignore" ${{ inputs.SYNC_PARAMS }}
# Ensure SOURCE_PATH is a relative path starting with "." and exists in the repository
if [[ "${SOURCE_PATH:0:1}" != "." ]]; then
echo "Invalid. The SOURCE_PATH must begin with a '.'"
exit 1
fi
REPO_PATH=$(realpath -e "${WORKSPACE_PATH}" 2>/dev/null)

@JPrevost JPrevost Jul 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: I'm not sure I see the value in swallowing error messages here. Was it really noisy?

ABSOLUTE_PATH=$(realpath -e "${WORKSPACE_PATH}/${SOURCE_PATH}" 2>/dev/null)
if [[ $? -ne 0 ]]; then
echo "Invalid. The SOURCE_PATH does not exist."
exit 1
elif [[ ! "${ABSOLUTE_PATH}" =~ ^"${REPO_PATH}"(/|$) ]]; then
echo "Invalid. The SOURCE_PATH is not within the repository!"
exit 1
else
aws s3 sync . ${{ inputs.S3URI }} --delete --exclude ".github/*" --exclude ".git/*" --exclude ".gitignore"
echo "Valid SOURCE_PATH=${SOURCE_PATH}, proceed."
fi
echo "Content is synchronized to ${{ inputs.S3URI }}" >> $GITHUB_STEP_SUMMARY

- name: Sync standard CDN S3 content
# Only run this step if this is standard content (e.g., a subfolder of the cdn/ folder)
if: ${{ inputs.DOMAIN == 'standard' }}
run: |
if [ '${{ inputs.SYNC_PARAMS }}' != '' ]; then
aws s3 sync ./$(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}') ${{ inputs.S3URI }} --delete --exclude ".github/*" --exclude ".git/*" --exclude ".gitignore" ${{ inputs.SYNC_PARAMS }}
# Check for "legacy" caller workflow that passes S3URI and set booleans
# and paths for later steps.
if [[ "$S3URI" == "" ]]; then
if [[ "${TARGET_PATH:0:1}" != "/" ]]; then
echo "Invalid TARGET_PATH=${TARGET_PATH} does not start with '/', exiting."
exit 1
elif [[ "${TARGET_PATH}" == "/" && "$DOMAIN" == "custom" ]]; then
echo "Invalid TARGET_PATH=${TARGET_PATH} cannot be just '/' for custom domain, exiting."
exit 1
else
echo "Valid TARGET_PATH=${TARGET_PATH}, proceed."
fi
else
aws s3 sync ./$(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}') ${{ inputs.S3URI }} --delete --exclude ".github/*" --exclude ".git/*" --exclude ".gitignore"
echo "Legacy caller workflow passed an S3URI value."
if [[ ! "S3URI" =~ ^s3://[a-z0-9._-]+(/[^/]+)+/$ ]]; then
echo "Invalid S3URI format"
exit 1
else
echo "Properly formatted S3URI=${S3URI}, proceeding."
fi
if [[ "$DOMAIN" == "standard" ]]; then
echo "LEGACY_TARGET_PATH=/$(echo "$S3URI" | awk -F/ '{print $5}')" >> $GITHUB_ENV
echo "LEGACY_SOURCE_PATH=$(echo "$S3URI" | awk -F/ '{print $5}')" >> $GITHUB_ENV
else
echo "LEGACY_TARGET_PATH=/$(echo "$S3URI" | awk -F/ '{print $4}')" >> $GITHUB_ENV
fi
echo "LEGACY=true" >> $GITHUB_ENV
fi
echo "Content is synchronized to ${{ inputs.S3URI }}" >> $GITHUB_STEP_SUMMARY

- name: Invalidate cache
run: |
if [ '${{ inputs.DOMAIN }}' == 'standard' ]; then
aws cloudfront create-invalidation --distribution-id $(aws ssm get-parameter --name "/tfvars/libraries-website/standard-cdn-id" --query 'Parameter.Value' --output text) --paths "/*"
echo "The cache for the $(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}') folder has been cleared." >> $GITHUB_STEP_SUMMARY
# Ensure that SYNC_PARAMS input only has `--include` and `--exclude` options
if [[ -n "$SYNC_PARAMS" ]]; then
TEMP_PARAMS="${SYNC_PARAMS//--include/}"
TEMP_PARAMS="${TEMP_PARAMS//--exclude/}"
# If there's still a -- in there, it's an invalid flag
if [[ $TEMP_PARAMS =~ -- || $TEMP_PARAMS =~ $'\n' || $TEMP_PARAMS =~ $'\r' ]]; then
echo "Invalid SYNC_PARAMS: only --include and --exclude parameters are allowed, exiting."
exit 1
fi
echo "Valid SYNC_PARAMS=${SYNC_PARAMS}, proceed."
else
echo "No additional SYNC_PARAMS, proceed."
fi

# Validate GHA_ROLE is a proper IAM Role name
if [[ "$GHA_ROLE" =~ ^[a-zA-Z0-9_-]{1,64}$ ]]; then
echo "Valid GHA_ROLE, proceed"
else
aws cloudfront create-invalidation --distribution-id $(aws ssm get-parameter --name "/tfvars/libraries-website/custom-cdn-id" --query 'Parameter.Value' --output text) --paths "/$(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}')/*"
echo "The cache for the $(echo ${{ inputs.S3URI }} | awk -F/ '{print $4}') site has been cleared." >> $GITHUB_STEP_SUMMARY
echo "Invalid GHA_ROLE, exiting."
exit 1
fi

- name: Generate DEV Summary
# Only run this step if the environment is "dev"
if: ${{ inputs.ENVIRONMENT == 'dev' }}
- name: Set Environment
# Prepare environment variables for the synchronization job.
id: env
env:
AWS_DEV_ACCT: ${{ secrets.AWS_ACCT_DEV }}
AWS_STAGE_ACCT: ${{ secrets.AWS_ACCT_STAGE }}
AWS_PROD_ACCT: ${{ secrets.AWS_ACCT_PROD }}
ENVIRONMENT: ${{ inputs.ENVIRONMENT }}
GHA_ROLE: ${{ inputs.GHA_ROLE }}
run: |
case "$ENVIRONMENT" in
Comment thread
cabutlermit marked this conversation as resolved.
dev)
echo "AWS_ROLE=arn:aws:iam::$AWS_DEV_ACCT:role/$GHA_ROLE" >> $GITHUB_ENV
echo "CDN_DOMAIN=dev1.mitlibrary.net" >> $GITHUB_ENV
echo "AWS_ROLE and CDN_DOMAIN set for synchronization job to Dev1"
;;
stage)
echo "AWS_ROLE=arn:aws:iam::$AWS_STAGE_ACCT:role/$GHA_ROLE" >> $GITHUB_ENV
echo "CDN_DOMAIN=stage.mitlibrary.net" >> $GITHUB_ENV
echo "AWS_ROLE and CDN_DOMAIN set for synchronization job to Stage-Workloads"
;;
prod)
echo "AWS_ROLE=arn:aws:iam::$AWS_PROD_ACCT:role/$GHA_ROLE" >> $GITHUB_ENV
echo "CDN_DOMAIN=libraries.mit.edu" >> $GITHUB_ENV
echo "AWS_ROLE and CDN_DOMAIN set for synchronization job to Prod-Workloads"
;;
esac

- name: Configure AWS Credentials
id: aws_credentials
uses: aws-actions/configure-aws-credentials@v6
Comment thread
cabutlermit marked this conversation as resolved.
with:
aws-region: ${{ inputs.AWS_REGION }}
role-to-assume: ${{ env.AWS_ROLE }}

- name: Set S3 Target URI
# Set the correct S3 URI for the synchronization job, differentiating
# between the general CDN prefix and the custom domain CDN prefix
id: s3_target
env:
AWS_REGION: ${{ inputs.AWS_REGION }}
DOMAIN: ${{ inputs.DOMAIN }}
TARGET_PATH: ${{ env.LEGACY && env.LEGACY_TARGET_PATH || inputs.TARGET_PATH }}
run: |
if [ '${{ inputs.DOMAIN }}' == 'standard' ]; then
echo "The updates to https://cdn.dev1.mitlibrary.net/$(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}') are now available" >> $GITHUB_STEP_SUMMARY
# Using values from SSM Parameter Store, construct the properly formatted
# S3 URI value for the synchronization step
BUCKET=$(aws ssm get-parameter \
--region ${AWS_REGION} \
--name "/tfvars/libraries-website/cdn-origin-bucket-name" \
--query 'Parameter.Value' \
--output text)
if [[ "$DOMAIN" == "standard" ]]; then
echo "DISTRIBUTION_ID=$(aws ssm get-parameter \
--region ${AWS_REGION} \
--name '/tfvars/libraries-website/standard-cdn-id' \
--query 'Parameter.Value' \
--output text)" >> $GITHUB_ENV
echo "S3URI=s3://${BUCKET}/cdn${TARGET_PATH%/}" >> $GITHUB_ENV
else
echo "The updates to https://$(echo ${{ inputs.S3URI }} | awk -F/ '{print $4}').dev1.mitlibrary.net site are now available" >> $GITHUB_STEP_SUMMARY
echo "DISTRIBUTION_ID=$(aws ssm get-parameter \
--region ${AWS_REGION} \
--name '/tfvars/libraries-website/custom-cdn-id' \
--query 'Parameter.Value' \
--output text)" >> $GITHUB_ENV
echo "S3URI=s3://${BUCKET}${TARGET_PATH%/}" >> $GITHUB_ENV
fi

- name: Generate STAGE Summary
# Only run this step if the environment is "stage"
if: ${{ inputs.ENVIRONMENT == 'stage' }}
- name: Sync To CDN S3 Bucket
# This uses a bash array to build the command in order to avoid the
# command injection risk.
env:
S3URI: ${{ env.S3URI }}
SOURCE_PATH: ${{ env.LEGACY && env.LEGACY_SOURCE_PATH || inputs.SOURCE_PATH }}
SYNC_PARAMS: ${{ inputs.SYNC_PARAMS }}
run: |
if [ '${{ inputs.DOMAIN }}' == 'standard' ]; then
echo "The updates to https://cdn.stage.mitlibrary.net/$(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}') are now available" >> $GITHUB_STEP_SUMMARY
echo "### Content synchronization to $S3URI." >> $GITHUB_STEP_SUMMARY
if [[ "$S3URI" == *"cdn/"* ]]; then
echo "Standard CDN content is synchronizing"
else
echo "The updates to https://$(echo ${{ inputs.S3URI }} | awk -F/ '{print $4}').stage.mitlibrary.net site are now available" >> $GITHUB_STEP_SUMMARY
echo "Custom CDN content is synchronizing"
fi
cd "$GITHUB_WORKSPACE"

# Build an array with the default command line parameters for the sync command
sync_cmd=(
"${SOURCE_PATH}"
"${S3URI}"
--delete
--exclude ".github/*"
--exclude ".git/*"
--exclude ".gitignore"
)

# Parse the previously validated SYNC_PARAMS input into the array, without any
# glob expansion.
if [[ -n "$SYNC_PARAMS" ]]; then
readarray -d '' -t extra_params \
< <(printf '%s' "$SYNC_PARAMS" | xargs printf '%s\0')
sync_cmd+=("${extra_params[@]}")
fi

- name: Generate PROD Summary
# Only run this step if the environment is "prod"
if: ${{ inputs.ENVIRONMENT == 'prod' }}
aws s3 sync "${sync_cmd[@]}"
echo "Content is synchronized to $S3URI." >> $GITHUB_STEP_SUMMARY

- name: Invalidate cache
env:
CDN_DOMAIN: ${{ env.CDN_DOMAIN }}
DISTRIBUTION_ID: ${{ env.DISTRIBUTION_ID }}
DOMAIN: ${{ inputs.DOMAIN }}
TARGET_PATH: ${{ env.LEGACY && env.LEGACY_TARGET_PATH || inputs.TARGET_PATH }}
run: |
if [ '${{ inputs.DOMAIN }}' == 'standard' ]; then
echo "The updates to https://cdn.libraries.mit.edu/$(echo ${{ inputs.S3URI }} | awk -F/ '{print $5}') are now available" >> $GITHUB_STEP_SUMMARY
echo "### CDN cache invalidation" >> $GITHUB_STEP_SUMMARY
echo "Start CDN Cache invalidation."
INVALIDATION_ID=$(aws cloudfront create-invalidation \
--distribution-id "$DISTRIBUTION_ID" \
--paths "${TARGET_PATH%/}/*" \
--query 'Invalidation.Id' \
--output text)
aws cloudfront wait invalidation-completed \
--distribution-id "$DISTRIBUTION_ID" \
--id "$INVALIDATION_ID"
echo "The cache has been cleared." >> $GITHUB_STEP_SUMMARY
if [[ "$DOMAIN" == "standard" ]]; then
echo "The updates to https://cdn.${CDN_DOMAIN}${TARGET_PATH#/} are now available." >> $GITHUB_STEP_SUMMARY
else
echo "The updates to https://$(echo ${{ inputs.S3URI }} | awk -F/ '{print $4}').libraries.mit.edu site are now available" >> $GITHUB_STEP_SUMMARY
echo "The updates to the https://${TARGET_PATH#/}.${CDN_DOMAIN} site are now available." >> $GITHUB_STEP_SUMMARY
fi
Comment thread
cabutlermit marked this conversation as resolved.
Loading