Cleaning up Lambda Logs with CloudFormation

Is your CloudWatch Logs console full of old log groups? Got 50 pages of Lambda log groups? Don't worry, I have the solution for you!

The issue here is not the cost; While there is a cost associated with keeping your log groups around, it's pretty small. The real issue is the administrative overhead - Having a log group for every function you've ever deployed adds up. When you combine this with the default retention period of "indefinitely" (which is very deliberately not "forever", but it might be!) you end up with a lot of log groups. Soon you're paging through pages and pages of groups to find the one you're looking for, and cleaning them up manually is a problem too: Each function has a log group, each log group takes 4 clicks to delete, and you can only delete one log group at a time - there's no way to select multiple groups at once.

When you use CloudFormation or SAM - you are practicing infrastructure as code, aren't you?! - your functions are given unique names to ensure they don't conflict with each other (because every function in an account/region must have a unique name). Since the log group name is based on the function's name each function generates its own log group, and this is good. The problem arises because the log group is created implicitly, rather than explicitly, and as such lives outside the CloudFormation stack. This means that when you delete the function stack, the log group is cut free, left to wander the ether for all eternity, alone and afraid.

As the problem suggests, the key is to make the implicit explicit. By creating the log group ourself - before the Lambda service creates it - we enable management of the resource via CloudFormation. To do that we simply define the log group for each function ourselves:

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/${MyFunction}

Where MyFunction is the logical resource ID for your function. Update and duplicate it as required by your resources. You can see an example SAM template in this gist, but it doesn't matter if you're using SAM or just vanilla CloudFormation.