Update: You can now provision NAT gateways with CloudFormation.
Amazon recently announced their NAT gateway service which allows you to attach an AWS managed NAT gateway to you private subnets instead of managing NAT instances yourself. There is a full rundown of the differences, but for me the big win is eliminating the single point of failure of a single NAT instance without the complexity of a failover setup. It’s also nice not to worry about picking the right size of NAT instance.
We manage our VPC infrastructure with CloudFormation, which doesn’t yet have support for NAT gateways. However CloudFormation has custom lambda resources that can do pretty much anything. Even when CloudFormation does gain support for NAT gateways, hopefully this provides another example of how to create custom resources.
Lambda Backed Custom Resources
When you use a Lambda backed resource, CloudFormation invokes the specified Lambda function asynchronously. It passes some metadata such as the id of the stack, the logical id of the resource (the key of the resource in your template) and of course the resource properties. When you’re done you upload a JSON document describing what you’ve done to a presigned S3 url it provides.
You ALWAYS need to write to this presigned url, even in the case of failure, or CloudFormation will just keep waiting (and eventually timeout). AWS provides the cfn-response snippet that makes this a little easier: you just need to pass the event, context and response data and it will craft the JSON document, upload it and call context.done
for you.
For a NAT gateway the inputs we need are the elastic IP allocation id and the subnet id, so your template looks something like this:
1 2 3 4 5 6 7 8 9 10 |
|
In my template NatIP
is an elastic ip created by the template and NatSubnet
is a subnet created by the template, but how you create those is obviously up to you. CloudFormation doesn’t care about the type (Custom::NatGateway
) beyond the Custom::
prefix. It allows both making the template a little more readable and allows a single Lambda function to handle multiple resource types.
Handling requests
There are 3 types of requests: create, update and delete. The data passed to your function varies slightly.
Upon receiving a create request you should create your resource from the data in event.ResourceProperties
and your response should include the PhysicalResourceId
. This needs to uniquely identify what your Lambda function created, and will be passed back to you in subsequent update or delete requests for the resource. It’s also what {"Ref": "MyGateway"}
will evaluate to in the remainder of your template. The NAT gateway id is the obvious thing to use in this case.
For update requests CloudFormation also passes an OldResourceProperties
attribute that describes the resource properties as they were specified prior to this update. If you can update the resource in-place, then return the same PhysicalResourceID
. If not then create a new resource, return a different PhysicalResourceId
and CloudFormation will follow up with a delete request on the old id. NAT gateways can’t be changed after they are created so we handle create and update requests identically.
Lastly, delete requests should destroy the resource.
The lambda function entry point just dispatches on the request type:
1 2 3 4 5 6 7 8 |
|
The createGateway
method creates a gateway and sends the response back to CloudFormation. Any properties you set in responseData
will become available via Fn::GetAtt
. For my usage there aren’t really any properties of the gateway that I wanted to expose.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
|
The deleteGateway
method just needs to delete the gateway:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
The validation of the physical resource id is to deal with a slight edge case. CloudFormation autoassigns a physical resource id to your resource, which is then overwritten when the response comes back. However if you try and rollback the stack update before that has happened, then it will issue the delete request with this dud physical resource id. In these cases we want the rollback to do nothing, rather than fail because we’ve passed a bogus id to deleteNatGateway
.
Using the gateway
A NAT gateway is no use until you’ve added it to your route table, but unfortunately the AWS::EC2::Route
resource doesn’t support NAT gateways yet, so we need another custom resource to handle the route. I’ve handled this within the same lambda function, so the top level entry point no needs to dispatch on the resource type:
1 2 3 4 5 6 7 |
|
The aws-sdk provides updateRoute
, deleteRoute
and replaceRoute
apis, which map naturally onto the request types but
replaceRoute
can only be used if the destination CIDR and route table id haven’t changed, so we check event.OldResourceProperties
to see if we can use replaceRoute
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
There is no ID that is returned by the createRoute
api: a route is identified by the route table and the cidrblock, so we concatenate these to create a physical identifier. Other than that the createRoute
, replaceRoute
and deleteRoute
functions just call through to the corresponding EC2 api.
You can then put something like this in your template:
1 2 3 4 5 6 7 8 9 10 11 12 |
|
Timing
Some resources (such as routes) create instantly, but a NAT gateway takes a minute or two to become available and you may run into issues if you create routes before it is ready. I haven’t found any better way of dealing with this than have the function poll the DescribeNATGateways
until its state changes. Luckily this happens fast enough to be well clear of a lambda function maximum execution time (5 minutes), although you are of course paying for execution time while your function is in fact just waiting.
You want CloudFormation to get the ID of the gateway as soon as possible rather than waiting for this, to avoid the case where it can’t destroy the gateway because it doesn’t know it’s id. I’ve handled this by responding to CloudFormation as soon as createNatGateway
is complete and then used aWaitCondition
in the template so that CloudFormation waits before creating the route. The lambda function polls DescribeNATGateways
and signals the wait condition when the state changes to available (This required changed the response.send
function from cfn-response
so that it doesn’t terminate the Lambda function.)
I’m not aware of a way of making this self contained.
Packaging up
One last niggle is that at time of writing, the version of the aws-sdk included in the Lambda environment doesn’t include the NAT gateway related functions (it includes 2.2.12 but we require 2.2.24). This means that you need to upload a package containing your source and the required version of aws-sdk. Assuming you have saved your script as nategateway.js
and that you have npm
installed you would do this:
1 2 |
|
And then upload nat_gateway.zip
. In the Lambda config for your function you need to set the handler to nat_gateway/index
.
TLDR;
The full script for a lambda function that handles both the gateway and its route, along with some of the error handling that I omitted here for reasons of brevity, is here along with a CloudFormation template that uses it.
The template creates a VPC, 2 subnets (a public one that the NAT gateway lives in and a private one that uses the gateway for external traffic), the NAT gateway and routes. It also creates the Lambda function and role needed for the custom resources.
The template uses a zip file I have uploaded to S3 in eu-west-1, so you will have to reupload it to an s3 bucket in your region (& update the template) to use it in other regions. I also highly recommend that you check the zip file rather than just blindly running code I’ve written, trustworthy as I may be. This stack will cost a small amount of money to run, since NAT gateways are not free.
Please note that this does have drawbacks compared to “native” cloudformation support for nat gateways, particularly if you were to use this to update an existing stack. Cloudformation doesn’t realize that your Custom::NatGatewayRoute
and AWS::EC2::Route
are the same underlying resource, so it won’t use ReplaceRoute
to update the route. In fact in my experiments I had to do 2 stack updates because it wasn’t deleting the AWS::EC2::Route
before adding the custom route. You would likely face a similar task if you wanted to switch from this to native cloudformation resources in the future. Caveat lector etc.