Our Goal at RigD is to provide developers with  a convenient experience for automating software deployments, so they can focus on building features that add value to their users. As we continue to add more capabilities and activities we want to check and ensure we are delivering on our objectives. One of the best ways to do this is to compare our solution to alternatives. The best comparison we have found for automated deployments of serverless on AWS is AWS CodePipeline using an AWS Serverless Application Model(SAM) template. Now that we have support for AWS on RigD we can do a very good comparison.

 

For completeness we actually tested three scenarios, one with manual configuration of the lambda and two using SAM.

1. RigD Pipelines without SAM
2. RigD Pipelines with SAM
3. AWS CodePipeline with SAM

 

Now we will show you the test flow and discuss the results.

 

For the test we start with a single lambda function with a python 3.6 runtime that takes in a Name and then returns “Hello [Name]”. Then we modified the Lambda to return “Hello [Name]!” and modified the lambda timeout to 5 seconds.

 

 

Here is a look at the results.

Scenario 2 is used as the benchmark, so items in green were better than scenario 2 and red items were worse.

 

Based on the results using RigD is less effort and faster than AWS CodePipeline. However lets lets review each metric and explore how they might change given changes to the use case.

 

Setup Time

If your looking for the fastest way to get an automated pipeline for a single lambda function up then using a RigD Pipeline without SAM is your best bet. However,say for example you added an API getaway and a DynamoDB table into the architecture. That would result in a lot more setup time for scenario 1 relative to scenario 2 and 3.  So consider SAM as you increase the complexity of your deployment architecture.

 

Config Code

Again if you are building on the fly then RigD Pipelines without SAM is your best bet. Of course if you want and need codified infrastructure then a Sam pipeline is the way to go. One thing to consider if you are experimenting with finding the right infrastructure definition. SAM and CloudFormation give you the ability to rollback which is great. However, this is time consuming. building manually in RigD in the experimentation phase give you the ability to execute each step individually and thus quickly fail and remediate that single step.

 

Config Steps

For Config steps we measured each navigation click or command/field entry. In the case of the two SAM scenarios this would only vary by a few regardless of how complex the deployment architecture became. Scenario 1 would increase significantly with more architecture complexity.

 

Pipeline Run Time

This is the really important metric. While you might only setup a handful of deployment pipelines, you will undoubtedly be running those pipelines 10s, 100s, 1,000s of times. Time savings here is what translates to faster delivery to customers. Again for this simple use case scenario 1 was fastest by a small margin. This is due to the extra time it takes to translate SAM templates into CloudFormation change sets and execute them. What was most surprising was the substantial difference between the RigD scenarios and AWS CodePipeline. Even with SAM RigD was almost 4 times faster. In past tests we had assumed that the use of SAM was the cause for the speed difference, but this round proved us wrong.

 

Additional Manual Steps

Both scenario 1 and 3 required manual steps to complete what scenario 2 could do fully automated. Scenario 1 required us to manually update the lambda timeout, which I likely a relatively low frequency change as compared to code changes. Scenario 3 required us to manually test the lambda function using the UI console. We did investigate invoking the lambda from the code pipeline, however this would have required a lot of extra code in the lambda function as you cannot provide an event structure of your choosing. That said the reality for most is that some other testing solution is going to be used. AWS CodePipeline does has some integrations for testing and we expect to add some to RigD in near future.

 

The Bottomline

If your developing a serverless application and you need deployment automation, then RigD is the fastest and simplest choice. For simple single lambda deployments that you might be using for utility purposes you can likely get by without SAM for a faster setup. If you are building an app with additional architectural components then RigD Pipelines with AWS SAM is the way to go. You’ll save a little time on setup, deploy 75% faster, and have an option for built in lambda testing.

 

Learn more about how RigD can help you get work done easier and faster on our Resources page and you can always sign up and try it for yourself.