Paying for idle is so 2015

I had some Lambda Functions that scraped data from the Internet, and stored them in a database. Locking-down the RDS Security Group to only Lambda Functions turned out to be more complicated than I thought.

The Problem

Out of the box, you can't restrict a Security Group to only allow Lambda Functions access. The closest solution is to use VPC-based Lambda (via a VPC Endpoint, and lock the SG down to the ENI that Lambda will use in your VPC. Unfortunately, VPC-based Lambda Functions don't have access to the Internet.

Opening my RDS instance to the world was not an option. Your RDS endpoint is published via DNS, so it's not a question of if someone will find your database, but when.

Wait, RDS?!

Yeah, I tried to get my project working with DynamoDB for that full-server-less feel, but I needed to do lots of scans and queries. I couldn't find a schema that 1) worked, and 2) didn't take ages to respond. As it says on the tin, it's a great key-value store, it's just for this project I needed relationships.

VPC-based Lambda Functions

Since Lambda Functions don't have access to the Internet once they're in your VPC, you need to configure an IGW and NAT Gateways in order to connect them. Setting up the VPC was not the issue, it was more that my you-don't-pay-for-idle-Lambda-Functions no require a suite of services to do their job. It ends up looking something like this:

VPC Lambda

I could roll my own NAT EC2 Instances for slightly lower monthly cost than the Managed NAT service AWS offers, but that's more work than I wanted.

A Solution

As a compromise, I decided to use Lambda to update my database SG so that only Lambda could reach it.

No VPC Lambda

There's an AWS blog post that details how to do something similar with Python, but it seemed quite word-y for the functionality and I thought I could do something simpler in Node.js without any extra dependencies.

The Code

The function runs from a target object that defines what will be updated. It has the following properties:

  • Security Group Id
  • Port
  • Protocol
  • AWS Region
  • AWS service name to allow

You can see an example target object in the gist below:

Note that function this will clobber all your SGs existing ingress rules. It was fine (even desirable) for my use-case, but YMMV.

The handler function that is called is actually quite small. I've used Promises (now that Node.js 4.3.2 is available) to control flow, which means the handler reads pretty succinctly.

IAM Permissions

The IAM Role that your Lambda functions run as will need the ability to describe and modify Security Groups (i.e. EC2 permissions). Unfortunately the managed polices AWS provides are read-only or full-access - neither of which are appropriate for this function.

Here is an IAM Policy document that has just enough permissions to work:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ViewAndUpdateSecurityGroups",
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DescribeSecurityGroups"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Your Lambda Function's role will need that policy in addition to the usual AWSLambdaExecute or AWSLambdaBasicExecutionRole managed policies (which lets it log to CloudWatch Logs, etc).

Subscribing for Updates

You can set up this function to be triggered when the list is updated.

Unfortunately there is no Lambda API call to set up an SNS Event Source for your function. You'll have to do that via the Lambda console.

While looking at your Lambda Function in the console go "Event sources" > "Add event source" > SNS topic > Enter arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged.

Hopefully they'll fix this missing API method soon.

I could also be checking the hash of the IP ranges (which is sent in the SNS Topic), but I haven't got that far yet.

CloudFront

Just like in the AWS blog post linked above, this could be used to protect your CloudFront origin servers. All you'd need to do is update the target object with your own details (e.g. Service name "CLOUDFRONT" and region "GLOBAL") and it would work.

Limitations

Obviously this still leaves my database open to access from other EC2 Instances, but I felt that was a much smaller risk (as at least AWS knows who they are).

Taking it Further

About 3/4 of the way through this solution, I realised I could take this solution even further; I could have a SG with no access by default, and have each Lambda call open a port for its specific IP for each request and close it when it's complete. It would only add 600-900ms per invocation (based on my limited testing), and be much more secure - access would only be available while the request was in-flight, and I still wouldn't have to pay for the idle resources.

I'd also like to convert the functionality in to a module to keep the calling function simple, and put in some more smarts so that functions don't tread on each other's toes, but the majority of the work is already done. This felt like part of the transition in thinking away from a "traditional" application design, to a "new world" server-less approach.