Help us to make this transcription better! If you find an error, please
submit a PR with your corrections.
Luciano: Development on AWS is often slowed down by deployment time, particularly in development environments. Often people turn to local simulation of AWS. While this can solve some immediate challenges and speed up your feedback loop, it comes with its own problems. So today we'll try to answer the age-old question, should you simulate AWS locally or go to the real cloud every single time? My name is Luciano, I'm here with Eoin and this is AWS Bites podcast.
AWS Bites is made possible by the support of fourTheorem, an AWS consulting partner that works with you to make AWS migration, architecture and development a success. See fourtheorem.com, this link is in the show notes and this way you can find out more about fourTheorem. So, I remember that when we were working with containers or monolithic applications like Node.js servers or Spring servers or whatever PHP framework you would like, it was always the default to just run things locally, develop that way and when you were happy you had a way to publish things to an actual environment remotely. Now things seem to have become significantly harder because you don't really have that real option of running things locally, you can maybe simulate certain things but since you are dealing with distributed applications it is becoming increasingly harder to be able to simulate everything on your own machine. So what are the options in AWS? Is it true that things are getting harder? What can we do?
Eoin: When Docker became popular I think it made local development a lot easier because the local runtime really matched the production runtime quite closely. You really only had to think about whether you would use a remote database or a local one and then with things like Docker Compose and then other container orchestration, things like Kubernetes, you could run a set of microservices and related resources like ElastCache, RabbitMQ and your databases locally too.
Now particularly with serverless applications, a lot of the application is realized with lots of distributed AWS services outside of the code running in containers and it might be in Lambda functions as well. We try to take advantage of managed services to save on production maintenance and provide that scalability, robustness and security but the trade-off then is that local development has become a lot more challenging. So if you want real SQS, DynamoDB, RDS, IAM, you need to deploy to AWS usually using Terraform or CloudFormation or something else like that. And sometimes deployments are fast and it's not really a problem but often they can take many minutes and when you're doing it multiple times per day this really adds up if you're making lots of changes especially early on in development. So I think this is really the challenge that has led a lot of people to try and simulate things locally. What have you tried and what are the options there?
Luciano: I tried a few things. The first one that comes to mind is LocalStack which has been around for a while and it's an open source project that tries to effectively simulate a bunch of different AWS services and allows you to run them locally using Docker containers and then you can connect all of them and the API tries to match exactly the API that you have in AWS. But of course it's not perfect. I don't think it's humanly possible to try to recreate AWS in its entirety and run it locally.
So the coverage is limited. It might work well for certain services and certain specific use cases, maybe if you're using only certain specific APIs that are the most common, but if you're doing something a little bit more outside the common use cases probably you will find that the coverage is not there yet. There was recently a new version that was released, version 2, which seems like the project is trying to push forward and try to extend that coverage and try to make the whole user experience even better. But I think it's still something that we need to figure out exactly to which extent is going to allow us to cover the full range of things that you might want to do in AWS. I think that the things that I found are generally very hard to simulate are IAM permissions, policies, the whole thing where you kind of generally give up on simulating that locally.
You just say, that's something I'm going to test in the real environment. For now, when I run things locally, I'm going to assume everything is open and there is no permission issue. So there is always some kind of art trade-off there and you kind of think it's working and then you have a permission problem in production and you need to test that part in, sorry, not necessarily production, but like the remote environment on AWS.
And then you'll need to test that until you actually get it right. So maybe that's an area where there might be a little bit more tooling and support for local development. Another similar one is CloudFormation. So I haven't found a way, not that I even tried that much, to test your CloudFormation locally. Generally, CloudFormation is supposed to run against AWS. So that's generally the way I test it. I just write code, try to deploy, see what happens and iterate that way, which can have a very slow feedback loop. So maybe that's an area where there could be some tool as well, maybe in the LocalStack space that can help us to do some more testing locally. Another thing that is worth mentioning about LocalStack is that this used to be a free and open source project. It still is an open source project, but now there is a company behind it and there is a commercial offering as well. So that might have its own pros and cons. Maybe there is more commitment in trying to improve the quality of the open source version, but of course there is also going to be a push for more commercial offerings. So let's see what happens.
Other ones that I tried are serverless local simulation. So these are especially important when you are working with things like Lambdas and you maybe use some frameworks to be able to write and deploy those Lambdas easily. Oftentimes this framework will give you some kind of capability to do some level of local testing. Worth mentioning Serverless offline, which is a plugin for serverless framework and allows you to simulate API Gateway and Lambda locally. So effectively it's going to run for your web server and you can call it, you can send call request, postman, use the browser, whatever you like to actually send request directly to your Lambdas running locally. So it's a very nice way to be able to test your APIs locally as you are developing them using serverless technologies. Very similar if you use SAM. SAM has an option called SAM local and that does pretty much the same thing. And I have been recently playing with Lambdas in Rust and there is another tool called Cargo Lambda, which is kind of the default tool to bootstrap your Lambdas and that comes also with some commands that allow you to simulate your Lambdas locally and integrates very well with SAM. So if you do your project in SAM and you are writing your Lambdas in Rust, you can use Cargo Lambda for all the building and running part.
But then you still use the commands that you are well used to use when you use SAM. So you can still run SAM local and behind the scenes is going to use Cargo Lambda to kind of compile your Lambda and run it correctly. There are other ones in .NET. For instance, there is AWS.NET Mock Lambda test tool. I haven't used this one, but we will have a link in the show notes in case you want to look at that one.
And finally, also worth mentioning that for other services like DynamoDB and Step Functions, there are some packages that you can download directly from AWS. These are generally either Docker containers or JAR executables and they give you some kind of local simulation of that particular service. For instance, for DynamoDB, you don't get a replicated distributed version of DynamoDB, but it is pretty close to the API that you get when you use DynamoDB remotely.
So I think you can rely on that for most of the things that you will do with DynamoDB. And very similar, there is the Step Function local tool, which allows you to run some local simulation of a Step Function. And it's also interesting because it gives you ways to mock. Because with Step Function, we can effectively call other services in AWS, it gives you a way to mock those steps. So you can still execute your Step Function simulation, even though you might be relying on other services that are outside the scope of Step Function. So I guess my comment there is that in general, these simulations are good as a starting point, but you only get so far because you always end up eating some kind of limitation, missing API or difference even in the way that the API works locally with the way it works remotely. And again, there is still the problem of permissions that you don't generally, you don't get any check on your roles or policies. So you kind of assume that everything is correct until you actually run it in AWS. So what do we recommend? Is there any approach that you find works best than others? I used to spend a lot of time on trying to get LocalStack to work.
Eoin: I Remember the early days, trying to make a pull request on LocalStack so I could get event bridge support for some integration tests I was doing. And that was just because I was determined that it was going to work for me and I ended up spending more time on trying to work around the limitations. So this is what you're faced with. And similarly, you mentioned Step Functions local, and recently I tried to use that, but I think there was an issue with support for the new map type because they changed how you can declare maps when they added support for distributed maps, but that's not there in Step Functions local yet. So these tools will always have a challenge keeping up with the latest in the cloud. So I am more now of the opinion that you should put less effort into local simulation and more into trying to optimize your development flow in other ways. So one thing you can do without local simulations is just try and focus more on unit tests. It's a good practice anyway, but if you focus on separating your code well, then you can just unit test the logic and you don't have to deploy every time. So this is just a recommendation really around code structure, following clean code principles, separating the AWS specific stuff from the actual logic, and following some sort of pattern. Like there's plenty, especially in the serverless community, of talks and articles around hexagonal architecture, also known as ports and adapters, which is essentially a fairly simple mechanism to ensure that you separate out the inbound connections and the outbound connections and integrations into your database and other services.
When you are unit testing, for example, in Python, you have moto and moto is a Python library that's actually the internal logic that is used by LocalStack. So it's like the library that makes local stack possible. You can also use moto to mock AWS services in your unit tests. Now I have sometimes used LocalStack and still do from time to time in unit tests and integration tests if the needs are simple and it can be valuable in that case. So what I would also say then is that kind of look at where you're spending your time and how you can just optimize for that. So if your issue is CloudFormation deployment speed, disabling rollbacks can help because sometimes, especially in development, a lot of the reason why it's slow is that you're making mistakes and misconfigurations and then when you deploy, the deployment time is reasonably fast, but the rollback time is the thing that gets you. So by disabling rollbacks, you can make that process a bit faster. And then if you've got lots of resources in your stack, then you don't have to roll back everything just because you've got one configuration value missing in some small resource.
Similarly, you could just try and incrementally build out these stacks and deploy them and kind of resolve the problems one by one. I also think I'm kind of hopeful that CloudFormation will become significantly faster in the future. I notice sometimes how it's just a little, some of the services implementations of the resource providers end up doing things like trying to create a resource. It fails because there's a misconfiguration, but you can see in cloud trail that CloudFormation is retrying this thing with a back off and then it just, it can really slow things down. So those are fairly simple things, low hanging proof that I think the AWS can address. I do see that AWS are trying to make things faster in general. So we talked before about SAM acceleration, SAM sync command.
This is essentially a backdoor to updating things like Lambda functions, API gateway, and Step Functions without using CloudFormation. So it's really just a quick and dirty hack really, but it speeds up development flow. So you don't have to deploy real resources using CloudFormation just because you want to update the code or configuration for a function. And there's a similar thing in CDK called CDK hot swap, which provides support for Lambda Step Functions, S3 assets, CodeBuild and AppSync resolvers too. So I'd say look for optimizations wherever you can.
You can spend a lot of time trying to get the right IAM policy and deploying lots and lots of times to get it right. I think we've all experienced that frustration that can be really slow. So one way to optimize this is to use session policies locally or within a shell on AWS, like in cloud shell or on an EC2 instance. So session policies are like ephemeral policies that you don't create in AWS. You just specify them when you do an assume role and you don't need to create a policy every time. So if you craft your policy locally, just in a JSON file, then you can do an assume role with it, test if it works. It's much quicker to test and make changes. And then once you have the policy right, you can codify it in your infrastructure. You can also, if you're making lots of deployments, you can also just take the ClickOps approach and tweak the resources manually in the AWS management console. And then once you're happy, codify those changes for proper deployment and test that with automated integration tests. So I'd say there's a combination of approaches there you can take, deploy your resources by all means that you need, but then run the code against those resources locally, just talking to the AWS services. Then test your Lambda functions. You can put whatever tests you need in place, whether it's unit test or an ad hoc manual test or some sort of automated integration test.
Luciano: One final comment that I have is that there are some tools that we discussed before, like for instance, Application Composer or Step Function designer, which are tools that sometimes they can help you to get unstuck. So if you are getting some configuration wrong and it's painful to go through this cycle of it is wrong, I'm going to deploy again, fails again, fix something else, deploy again, fails again until you actually get it right. I think these tools can give you kind of a shortcut into figuring out what's the right configuration. You can easily sketch out with the visual designer something that looks like your use case, understand exactly what is the right configuration, then transpose that configuration to your actual code. This is another, maybe it's a bit of a hack, but it's another strategy that sometimes I use to kind of unblock myself when I get stuck into the cycle of failing and retrying, failing and retrying until you actually get what you want to do right. So that's another tip for you that you might want to explore. So in summary, I think for today, what we want to share with you is that things like LocalStack are really, really good projects, but they are not to be trusted 100% as a perfect copy of AWS. Of course, you need to understand that there will be approximations there, there will be missing features. So use them, but with the benefit of the doubt.
So if they help you to speed up your local development, take that value, but be aware there might be instances where that local simulation is not going to be enough and you need to figure out another strategy. And definitely remember to structure your code in such a way that you can test most of it without having to rely on AWS. So you're definitely going to have some business logic, try to make sure that that business logic is as decoupled as possible from the logic integrating with AWS services. So you can test your own business logic without having to rely on AWS. And for all the times where you need to rely on AWS, you can also mock parts of your unit test. So there are libraries like Modo, if you do Python, that can help with that. If you use JavaScript, the new SDK also comes with an additional library that helps you to do mocking. So there are lots of ways there where just by doing unit tests, you can be so much better positioned to trust your code locally before you ship it to production. Now, we probably missed many ideas. I'm sure that people listening have lots of other ideas. Everyone is experimenting all the time with this topic. So if you have something you would like to share with us, we're definitely going to be happy to learn from you. So make sure to use the comments below or reach out to us on Twitter to share your feelings and your ideas about this topic. And as always, if you really enjoyed this episode, remember to thumbs up, subscribe, or if you're listening to the podcast only version, give us a review. Thank you very much, and we'll see you in the next episode.