Webp with CloudFront CDN and S3

We are going to convert JPG/PNG images to webp using Edge Lambda in Cloudfront. This will offload our server for handling any webp conversion – the serverless way!

What is webp?

Webp is a new image format introduced by Google in 2018. It is designed to conserve the quality similar to JPG/PNG but a much smaller size. It is currently supported by up to 93% of the browsers. It improves the website performance significantly for both desktop and mobile users. Images are usually the heaviest assets on the website.

Examples:

https://adrenalinmedia-website.s3-ap-southeast-2.amazonaws.com/demo/example1.png (1124KB)

https://adrenalinmedia-website.s3-ap-southeast-2.amazonaws.com/demo/example1.webp (190KB)

https://adrenalinmedia-website.s3-ap-southeast-2.amazonaws.com/demo/example2.jpg (585KB)

https://adrenalinmedia-website.s3-ap-southeast-2.amazonaws.com/demo/example2.webp (351KB)

What is CloudFront?

CloudFront is a Content Delivery Network (CDN) from Amazon Web Services (AWS) that delivers fast, secure content globally (215+ Edge locations). It means your website content can be served as fast as a regional service anywhere in the world. It also offers many other features such as Object Caching, Edge Computing and Firewall.

What is S3?

AWS Simple Cloud Storage (S3) is a cloud storage service that allows your website assets stored in AWS cloud unlimited. For a single file, the maximum size is up to 5 terabytes!

Webp + Cloudfront + Edge Lambda + S3

The combination of these will have:

• Convert webp using pure AWS cloud resources

• Improve the website speed with optimised images webp

• Offload the server for image assets using CloudFront with S3 unlimited storage

• Achieve HA (High Availability) for the image assets

Approach

To support both old and modern browsers, we will have 2 URLs for the same image. One is for the source image and the other is for webp which use the query string ?format=webp. Looks like below.

Source image: /123.jpg
Webp image: /123.jpg?format=webp

When the first webp request hits CloudFront, the Edge Lambda will create a webp image on the fly and store it in S3. If the same request made again, CloudFront will simply retrieve the webp from S3. In addition, It is also cached in CloudFront for faster content delivery.

Let’s Code!

Create Lambda CF_Edge_OriginRequest
Create IAM Lambda Role:

1. Create IAM Role: CF_Edge_OriginRequest-Role

2. Create an inline policy CF_Edge_OriginRequest-RolePolicy. Please change <your-account-number> and <your-s3-bucket> (allow Cloudwatch logs and s3 bucket)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:*:*"
    },
    {
      "Sid": "VisualEditor2",
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::*/*",
        "arn:aws:s3:*:<your-account-number>:job/*",
        "arn:aws:s3:::<your-s3-bucket>",
        "arn:aws:s3:*:<your-account-number>:accesspoint/*"
      ]
    }
  ]
}

1. Add trust relationships:

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Principal": {
    "Service": [
      "edgelambda.amazonaws.com",
      "lambda.amazonaws.com"
    ]
    },
    "Action": "sts:AssumeRole"
  }
  ]
}

Create Node project

1. Create a folder CF_Edge_OriginRequest

2. Initialise node project

npm init
npm install path

1. Create index.js

const path = require('path');

function getQueryVariable(variable, query) {
  var vars = query.split('&');
  for (var i = 0; i < vars.length; i++) {
  var pair = vars[i].split('=');
  if (decodeURIComponent(pair[0]) == variable) {
    return decodeURIComponent(pair[1]);
  }
  }
}

exports.handler = async (event, context, callback) => {  
  const request = event.Records[0].cf.request
  const { uri } = request;
  const headers = request.headers
  const format = getQueryVariable('format', request.querystring);

  if (path.extname(uri).match(/(\.jpg|\.png|\.jpeg)$/g) && format === 'webp') {
  const filename = path.basename(request.uri);
  const olduri = request.uri
  const newuri = olduri + '.webp';
  request.uri = newuri
  }

  callback(null, request);
}

The code above checks if the request is in this format 123.jpg?format=webp. If yes, change the request to the origin with a new route 123.jpg.webp.

1. Create Lambda: CF_Edge_OriginRequest

• Region: us-east-1 – N. Virginia (Edge Lambda must be this region)

• Select an existing IAM Role: CF_Edge_OriginRequest-Role

• Select Node.js: 12.x. (at the time, Edge lambda only supports 12.x)

• Memory: 128 MB

• Timeout: 5 seconds

• Upload your lambda zip file

• Publish a new version

Create Lambda CF_Edge_OriginResponse

Create IAM Lambda Role:

1. Create IAM Role: CF_Edge_OriginResponse-Role

2. Create an inline policy CF_Edge_OriginResponse-RolePolicy. Please change <your-account-number> and <your-s3-bucket> (allow Cloudwatch logs and s3 bucket)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:*:*"
    },
    {
      "Sid": "VisualEditor2",
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::*/*",
        "arn:aws:s3:*:<your-account-number>:job/*",
        "arn:aws:s3:::<your-s3-bucket>",
        "arn:aws:s3:*:<your-account-number>:accesspoint/*"
      ]
    }
  ]
}

1. Add trust relationships:

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Principal": {
    "Service": [
      "edgelambda.amazonaws.com",
      "lambda.amazonaws.com"
    ]
    },
    "Action": "sts:AssumeRole"
  }
  ]
}

Create the second Node project

1. Create a folder CF_Edge_OriginResponse

2. Initialise node project

npm init
#sharp is the node image library for converting webp
npm install path && npm install aws-sdk && npm install sharp

Create index.js

• Update BUCKET to your S3 bucket name and adjust the QUALITY as wished.

const path = require('path')
const AWS = require('aws-sdk')

const S3 = new AWS.S3({
  signatureVersion: 'v4',
})

/*** IMPORTANT ***/
/*
1. This is CloudFront Edge Lambda which CANNOT have environment variables.
2. The sharp package must be linux package in order for CloudFront to run. Thus it is lived in the repo.
*/
const Sharp = require('sharp');
const BUCKET = '<your-s3-bucket-name>'; // Change to your s3 bucket
const QUALITY = 70; // adjust webp quality

function getQueryVariable(variable, query) {
  var vars = query.split('&');
  for (var i = 0; i < vars.length; i++) {
  var pair = vars[i].split('=');
  if (decodeURIComponent(pair[0]) == variable) {
    return decodeURIComponent(pair[1]);
  }
  }  
}

async function getS3Resource(key) {
  try {
  const resource = await S3.getObject({ Bucket: BUCKET, Key: key }).promise()
  return resource;
  } catch (error) {
  if (error.code !== 'NoSuchKey') {
    console.error(error);
  }
  }
  return null;
}

exports.handler = async (event, context, callback) => {
  const { request, response } = event.Records[0].cf;
  const { uri } = request;
  const headers = response.headers;  

  const format = getQueryVariable('format', request.querystring);

  if (path.extname(uri).match(/(\.webp)$/g) && format === 'webp') {  
  if (response.status === "403" || response.status === "404") {
    // handle s3 path. E.g. "/folder/file%201.jpg.webp" to "folder/file 1.jpg.webp"
    const webpS3Key = uri.substring(1).replace(/%20/g, ' ');
    const s3key = webpS3Key.replace(/(\.webp)$/g, '');

    let sharpImageBuffer = null;

    try {
    const resource = await getS3Resource(s3key);
    if (resource) {
      sharpImageBuffer = await Sharp(resource.Body)
      .webp({ quality: +QUALITY })
      .toBuffer();

      await S3.putObject({
      Body: sharpImageBuffer,
      Bucket: BUCKET,
      ContentType: 'image/webp',
      CacheControl: 'max-age=31536000',
      Key: webpS3Key,
      StorageClass: 'STANDARD',
      ACL: 'public-read'
      }).promise();
    }

    if (sharpImageBuffer) {
      response.status = 200
      response.body = sharpImageBuffer.toString('base64')
      response.bodyEncoding = 'base64'
      response.headers['content-type'] = [{ key: 'Content-Type', value: 'image/webp' }]
      response.headers['cache-control'] = [{ key: 'Cache-Control', value: 'max-age=31536000' }]
    }

    } catch (error) {
    console.log(error);
    }
  }
  }

  callback(null, response)
}

The code above checks if the request is in this format 123.jpg.webp, then check if the webp file exists in s3. If not, generate a new webp and store it in s3 with the name 123.jpg.webp. If already existed, just pass the request.

Important: The sharp library works differently depending on the OS. Because Edge Lambda is running in Linux, you must install Node package using Linux to be compatible. See appendix.

1. Create Lambda: CF_Edge_OriginResponse

• Region: us-east-1 – N. Virginia (Edge Lambda must be this region)

• Select an existing IAM Role: CF_Edge_OriginResponse-Role

• Select Node.js 12.x. (At the time, Edge Lambda only supports 12.x)

• Memory: 256 MB

• Timeout: 12 seconds (Important: the image conversion is likely taking more than 3 seconds the default time. Make sure you extend the timeout)

• Upload your published lambda zip file

• Publish a new version

Set up edge Lambda in CloudFront

1. Add behavior to the CloudFront

• Select your s3 as the Origin

• Cache and origin request settings: Use legacy cache settings

• Object Caching: Customize. Leave default TTLs.

• Associate Origin Request and Origin Response with 2 Lambda. Lambda version :x is required in the ARN.

• Query String Forwarding and Caching: Forward all, cache based on all. (You can select whitelist for better cache control)

• Add the 2 function associations below with the latest Lambda version. For further reference about associations, see here.

Test time!

Let’s see the result. You can test it by uploading JPGs and PNGs to s3 and request using the format below.

• Source image: example.jpg

• webp image: example.jpg?format=webp

Things to know

• The Edge Lambda is replicated into AWS edge servers. It means the CloudWatch logs will reside in the edge region. E.g. if you request from Sydney, the CW log will be in ap-southeast-2.

• The Edge Lambda is running under Linux, the node sharp library needs to be installed using Linux.

• The Edge Lambda can’t use Environment variables. You will need to duplicate it per environment.

• Every update made in Lambda will need to publish a new version and re-associate in CloudFront. In addition, each time a new update is made to CloudFront, it takes approx 3 minutes to reflect the update.

• When running Lambda tests within AWS Console, because the Lambda is created in us-east-1, the CW log will be in us-east-1.

• Lambda is charged per execution. With the CloudFront cache, they only execute when the cache is expired. This minimises the cost for Lambda.

• Each image creates 2 assets in S3. The original image and the web image. This will slightly increase the S3 cost.

Summary

We have implemented webp using Edge Lambda in AWS CloudFront – the serverless way. This will significantly improve the website performance. The next step is to reference these webp images on your website. Create a small Javascript function to do browser detection and reference between the source image and webp.

Appendix

Run a Linux-based Node image in docker for installing the sharp library.

Create docker-compose.yml

version: '3.3'

services:
  node:  
  container_name: my-node-server
  image: node:12-buster  
  volumes:
    - ./app:/var/html/www
  tty: true

Run docker using docker compose

## start docker
docker-compose up -d 

## remote to the docker instance
docker exec -it my-node-server bin/bash 

## create node packages
mkdir /var/html/www/CF_Edge_OriginResponse
cd /var/html/www/CF_Edge_OriginResponse
npm init
npm install path && npm install aws-sdk && npm install sharp

Then get all the files including node_modules from your ./app folder.