AWS SAM + Python + Pytest

Update 07.09.2020: Added an AWS DynamoDB.

In this article we will build, test and deploy an example lambda using SAM, Python3 and Pytest, while maintaining maximum code coverage, development velocity and simplicity.

Introduction:
AWS provides an open source framework for building serverless applications — Serverless Application Model.

SAM allows you to easily deploy lambdas on AWS, simplifies the process and saves you lots of time and headache with zipping, installing dependencies and etc.

Prerequisites:
Python3
AWS CLI
SAM CLI
pip packages: pip install pytest jq yq responses requests moto
An S3 bucket(s) to store the lambda artifacts

Why SAM?

While there are other Infrastructure as Code tools out there, I found SAM to be the easiest and most convenient tool for packaging and deploying functions and their resources on AWS.

OK, let’s start digging in.
P.S, the entire code base is available here.

Part 1: code

The lambda downloads a file containing data about all NBA players for the 2019–2020 NBA season, adds each San Antonio Spurs player to DynamoDB database (key value), and uploads the json to an S3 bucket that is created by our SAM template.
I chose to upload the file using smart_open and using boto3 because it’s a 3rd party library that requires us to build the lambda inside a container, which is very easy with SAM, and it’s great to have it as a part of the demo.

Our dependencies are specified in the requirements.txt file and installed by SAM during the sam build command

Part 2: template

The template is straightforward and creates several resources required by our lambda:
1. Lambda function.
2. Bucket for uploading the file.
3. DynamoDB table for storing the San Antonio Spurs roster.
4. Log group for logging the lambda’s output.
5. CloudWatch Event that triggers the lambda every 3 hours.

Part 3: test file

In our test we’re going to mock the response returned by the requests library, and more importantly, the DynamoDB table to which we update the roster details.
We’re also using Decision Table Testing technique for testing multiple scenarios.
For more on decision table testing, read here.

Part 4: deployment script

The deployment script runs multiple stages, and fails the deployment if one of them is not satisfied.
Add your bucket name inside the case switch in line 18.

To script is executed with an environment parameter, for example:
./build.sh dev

Deployment gates:

1. unit tests:
   Let’s make sure that an invalid url indeed generates an error in the test:
   In line 42 we expect the application to raise an exception, so in line 41 we’ll switch the test names (change it to “works properly”) and make our test expect an exception, but it will never happen.

As we can see, the deployment failed and exited without actually deploying

2. Integration Tests:
SAM provides us with a command that runs the lambda locally with an actual event (cloudwatch events, sqs, etc) and environment variables, as a container, just as if it were actually running on AWS.
sam local invoke --template build/template.yaml --env-vars ${ENV}.json --event ${FUNCTION_NAME}-event.json > output

* for the integration tests we’ll be using our lambda’s bucket (from the prerequisites), to make sure it works before the initial deployment, where the infrastructure is created

Let’s change the download link to something invalid:

Again, the deployment exits without doing anything.

OK then, let’s deploy.

(The screenshot contains only the end of the output, due to it’s size)

Let’s navigate to AWS CloudFormation

As expected, our stack is deployed successfully.

Let’s run the lambda from the console:

Verify that our file is present inside the S3 bucket:

Cool!

That’s it! I hope you enjoyed it. feel free to comment and I will do my best to answer.

The next article will discuss mocking AWS resources in Golang using the Golang AWS SDK