Help us to make this transcription better! If you find an error, please
submit a PR with your corrections.
Eoin: AWS Application Composer is a free service that helps you architect and build serverless applications. With a simple drag and drop interface, it promises to lower the barrier to entry in building applications using infrastructure as code. But does it live up to that promise? In this episode, we're going to unbox AWS Application Composer, tell you what it's like to use, and let you know whether you should add it to your toolkit or give it a hard pass.
I'm Eoin, I'm here with Luciano and this is the AWS Bites podcast. AWS Bites is sponsored by fourTheorem, and fourTheorem is an AWS partner for migration, architecture and training. Find out more at fourtheorem.com and you can find the link in the show notes. Right, let's get down to it, Luciano. We covered Application Composer briefly when it was announced at reInvent late last year. Since then, you've taken it for a proper test drive and really tried to see where it works and where it doesn't work. So maybe we can just start by telling everybody who doesn't know what is Application Composer and what kind of problem is it actually designed to solve in the first place?
Luciano: Of course, yeah. So let's get started. And as you said in the introduction, Application Composer is a free service. Let's put a little bit of an effort there in that free keyword, because I think it's going to be very important towards the end when we try to draw our conclusions. And the idea is that this service is going to help developers, cloud architects, to streamline and accelerate the architecture configuration of serverless applications.
So the idea is that it's built on top of infrastructure as code. So it uses some of CloudFormation syntax in a more general sense. So it kind of helps you out if you are not familiar with that syntax, gives you a very visual drag and drop interface where you can drag different resources, connect them and behind the scenes, it keeps that infrastructure as code representation of everything that you are drawing up to date with the latest changes.
So the idea is that if you don't know infrastructure as code, CloudFormation specifically, or maybe you know a little bit of it, maybe you don't know specific resources, again, it's that idea that it tries to lower down the barrier to end and trying to speed up achieving something mostly in the serverless space as of today, that you can quickly build and use it and put it in your own AWS account. The way it works is that it is an application that is running on your browser, specifically, you just log in in the AWS console, and it's going to be one of the many services that are available through the AWS console.
And when you start that particular service on the AWS console, you immediately see that there is this concept of a canvas, which looks like one of the resources that you can manage with this service. But it's important to mention that it's not really an actual resource in the classic sense, because it's not something that is persisted long term in your AWS account. It's just the idea that you can start a new session in the browser, but that session is totally lost when you leave that browser window.
And this is an important detail that we will cover a little bit more in the rest of this episode. So the idea is that you have to persist files somewhere because you are creating templates in YAML for your infrastructure as code through this experience, visual experience. So what the application composer gives you is the opportunity to synchronize a local folder in your file system with everything that you are doing in the browser.
And this is what they call connected mode. So when you start a new canvas, the first thing that you see is literally a screen that tells you, do you want to work in connected mode? If you want to do that, you need to give me permission. You need to give the browser permission to read a specific folder in a file system. Now, this is another important detail because being able to give a browser file system permission is something relatively new in the web platform. And at this time, only Chrome and Edge, I believe, implemented that. Definitely doesn't work for Firefox, even though if I was researching a little bit this API called File System Access API, and it seems that Firefox now has an experimental support for that. So maybe eventually this is something that also Firefox will be able to support. So again, at this point, what you can do is start the browser, start the application composer, connect your local folder, give permission, and at that point, you can work on this canvas.
Eoin: OK, so from what I understand, then it's something that you get through the AWS console. It gives you a nice looking UI for building applications, and it works in both directions. So you can generate a visual representation from code, but you can also generate code by dragging and dropping in this application. I know that like many other people, it took me a while to get to grips with CloudFormation when I started many years ago and figuring out how to define components and connect them together. So it's not something you can pick up that quickly. And for that reason, I think the idea of application composer makes sense. And you've mentioned that phrase, lowering the barrier to entry a few times. So it seems like, in theory, at least a good fit. In practice, what is it like? So you've built some applications with it. What are your first impressions? Did you encounter any major roadblocks?
Luciano: Yeah, I think it would be beneficial to try to describe what I try to build, because it's a relatively simple project. But I think there is also enough complexity there, if you were going to do all of that infrastructure as code manually, to have a good feeling for whether this kind of tool can really help us for more realistic projects. So everything that I'm about to say, by the way, is available in our repository.
We will have the link in the show notes. And also the repository details how to basically clone that solution and deploy it, if you want to see it running yourself, and all the steps that you can follow, if you want to replicate everything that I am about to describe yourself. So the idea is that it's a very simple application, where there is an API gateway with just one endpoint. This endpoint will trigger a Lambda, which is responsible for basically doing that Lambda proxy integration and generating an HTTP response.
And before doing that, it's going to go into an S3 bucket and create a new file. So basically for every request, it's going to create a new file with a timestamp, just to simulate some interaction between the Lambda and another service like S3, to answer that particular HTTP request. So effectively, we have three different services, API gateway, Lambda and S3. They are connected together and there is a very specific workflow triggered by an HTTP request, starts the Lambda, the Lambda writes to S3, and then the Lambda responds back to API gateway.
So these are the steps that I follow to try to implement all of this. I started a new project in Application Composer, started in synced mode. So I had to select a folder in my file system. And as soon as I said, okay, keep going, the first thing that I noticed is that I didn't have anything in that folder, but as soon as I clicked okay in Application Composer, a new template.yaml appeared in that folder.
And initially, it's totally emptied. Actually, there is literally like an empty object in the YAML syntax. So that kind of tells you that an empty canvas is equivalent to an empty object in YAML for Application Composer. At this point, you have an Application Composer on the left side, a list of all the resources that are supported. And one of these is API gateway. So I just drag and dropped that resource into the canvas.
And immediately what you could see is that it starts with one endpoint. This is already preconfigured for you, which is a GET endpoint to the slash path, so the root path. Of course, it's something that you can change. You can double click in the canvas object that you just dropped there, and you will see a panel with a bunch of configuration options. So you can add more endpoints if you want to. But for the sake of this project, I was happy with that default endpoint.
And then I started to look at the template and I realized that all of that stuff, so that this new API gateway resource was already codified in Infrastructure as Code. And that was actually something already interesting because that is, I don't know, probably around 15, 20 lines of code. So it's definitely not necessarily simple if you want to write it yourself. And I was able to get all of that in just a few seconds by drag and dropping something in the browser.
So the next step was to start to provision, well, start to create the code related to the Lambda function. So same idea, go to the bar with all the resources supported, drag the Lambda icon into the canvas and immediately you have the definition of a Lambda, both in the canvas visible, but also in your template YAML. And with Lambda is actually very interesting. I feel like it's one of the resources where they spend the most time trying to give you ways to configure it.
In fact, you can double click that Lambda object in the canvas and you will have such a big panel with like many configurations options. And one of these options, the first one that kind of captured my attention was the runtime option. So you can click there and you have a long list of all the possible runtimes that you can use. And actually even more interesting, if you're interested in Node.js, you have runtimes that are not just Node.js, but you can have the same runtime for instance Node.js 18 plus TypeScript.
So I was really curious to try that and I picked up Node.js 18 with TypeScript. And I immediately realized that I didn't just get all the YAML related to that Lambda with that particular runtime, with the metadata that SAM understands to be able to build TypeScript for you using ES build. But also there was a skeleton folder that was created for me called source slash function, where inside of it I already had an handler in TypeScript and a package JSON.
So all of that stuff was very convenient. And again, something else that lowers the barrier to end to even understand how do you organize files with your source code when you're using a template with SAM. All of this stuff was nicely connected together. Then at this point, the next step was S3. So again, drag the S3 icon into the canvas and you get a YAML with the definition of a bucket. And this was also very interesting because this is something if you want to do it yourself, it's actually the simplest of the three resources.
But then you never remember that there are a bunch of configuration best practices that you can do when it comes to S3 buckets. For instance, encryption, for instance, making sure that you don't have communication without encryption when you use the APIs to connect to S3. And all that stuff, all these best practices were already provided by the YAML auto-generated by Application Composer. So it wasn't just the S3 bucket on its own, but there was also all the configuration for encrypting at rest and encrypting in transit, which is, I think, a really good best practice and something that you would easily forget to do it yourself, especially if you are learning CloudFormation for the first time.
So the next step is to connect things together because right now we only have an API gateway. We have a Lambda and we have an S3 bucket. They are not connected with each other and we can easily connect them by dragging. There are like some dots that you can drag on every source to connect with the other one. So basically you create a line between them. And the first line that I created was from API gateway to the Lambda.
And what that did was basically understand, OK, you want to trigger this Lambda from API gateway. So basically the YAML was updated, creating an event description in the specific Lambda saying this Lambda can be triggered by an HTTP event. So that was really convenient as well, especially if you don't really know what's the right syntax. Should they put something on the API gateway side? Should I put something on the Lambda side?
That kind of helps you out to really figure out how to describe that integration in CloudFormation and some. And finally, I did something similar for the bucket. So basically I connected the bucket to the Lambda and that understood that I wanted to use that particular bucket in the context of my Lambda code. So what happened is that it was created a policy for me. So for the Lambda to actually access the S3 bucket, but also this was really cool.
It created two environment variables in the Lambda, referencing the name of the bucket and the ARN of the bucket, which is something very convenient when you need to start to add your code. You don't necessarily know what's going to be the name of the bucket. So generally you would use an environment variable to create a reference, and then you have an abstract way to connect to a specific bucket in your Lambda code.
Now, the last step was to try to deploy all of this. I thought at that point I had, of course, I had to write a little bit more code in my own Lambda because the handler that was generated was just an Hello World. I actually wanted to write code to use the SDK, connect to S3 and create that file and then respond back to the API Gateway request. So for all of that, I needed to install a few extra dependencies because the SDK was not included.
And also there is a dev dependency that you can use to get types for the different events and responses that you can use with the API integration. There was some code there, but the dependencies were not installed. So I needed to install those myself, but eventually I had all the code ready. It's probably like 12 lines of code. You can check that in the repository. And I was able to deploy. I wasn't able to deploy directly from Application Composer itself, but I had all the files locally. So basically locally you can just run SAM, you can do SAM build to build all the TypeScript and convert it into proper JavaScript. And then you can do SAM deploy to initialize a deploy. And after a few minutes, you should have everything running in your AWS account.
Eoin: It's pretty nice the way you can also then use SAM Accelerate. So you could run 'sam sync --watch' and have it just live update. So when you drag and drop components in your architecture, it's actually deploying that infrastructure under the hood and updating the function configuration under the hood. And it could be a fairly productive workflow for the kind of applications that Application Composer supports right now. So you've kind of explained how you start with the visual version, but we talked about this being bi-directional. So what happens if you've got an existing application? Can you import any application you've got that supports CloudFormation or SAM? Does it work flawlessly? Are there any limitations? Yeah, by the way, I didn't try to do that sync idea.
Luciano: I think it's a pretty cool idea. We should try it at some point. And I'd be curious to see what happens if you like break the YAML at some point, because another note that I have is that it's very easy to break the YAML if you create new resources, remove them, disconnect them, reconnect them. You might end up in a state where the YAML is not necessarily 100% correct until you fix the issues. So it would be interesting to try exactly what happens when you use the sync mode and then it tries to deploy continuously at every change.
Anyway, with that aside, I did try to import an existing project that I worked on that is a little bit more... It's not too complicated, but there is a little bit more to it. And this is a project we will also have the repository link in the show notes. It's something that I call earthquake notifier, because it's a small personal project that I built just to be notified when a significantly interesting earthquake is happening close to my family in Sicily. So I can immediately know that that's happening and I can reach out to them to see if everything is fine. And it's a serverless project. Basically, it's built using a Lambda that runs on a schedule. This Lambda will call a specific API every hour that this API has a list of recent earthquakes in Italy and it will check if there is a set of conditions that will match mostly based on position and magnitude of the latest earthquakes. And if there is an earthquake that matches what it does, it creates an event bridge event. Then there is an event bridge rule that captures this kind of events and will deliver them to a CloudWatch log stream, just for me to see all of them historically. And also to an SNS topic where I can easily trigger notifications, for instance, to my email using an email subscription to that SNS topic.
So again, it's not an extremely complicated project. It's actually a very simple one, but there are enough resources there to try to put application composers to test. And I did all of that using some pretty much manually. So it was also a good use case, I think, to try to see that reverse approach when you have something already created manually, what happens when you import it in Application Composer.
So the result was actually a little bit disappointing because when I imported, well, basically I created a new canvas and I connected it to my local folder. But this time the local folder had all the files for this project, including the template.yaml. And what appeared in the canvas was some of the resources, not all of them, and they were not all connected correctly. I had my own architecture diagram and I was expecting to see something very close to it.
Instead, it was very different. There were only a few resources that were connected together, and most of the connections were missing. And so that was a little bit disappointing. And I started to try to fill the gaps myself. For instance, I was trying to connect an event bridge rule to SNS and I got an error message popping up saying this feature is not supported yet. So this is probably one of the reasons why the Application Composer wasn't able to build a correct representation of the architecture or at least the connections between different resources correctly, because probably some of these things are not supported yet. So in general, yeah, I wasn't really happy with the result. I don't think it was giving me value at that point to continue with that approach. I think for that particular project, I was probably better off to just continue with the manual approach.
Eoin: Okay, so there's a mixture of good and bad there. Talk about, you mentioned that it's for creating serverless applications, right? So I'm sure then it supports all of the serverless offerings from AWS, including, I don't know, Fargate, MSK serverless, Redshift serverless, EMR serverless, right? They're all in the same serverless family. Obviously my tongue is in my cheek there, but what is the extent of it? Where do the boundaries exist for this Application Composer serverless world?
Luciano: Yeah, it definitely doesn't support many of the serverless or at least the ones labeled with serverless offering in AWS. For instance, Fargate is the one that I wish was there because it requires significant infrastructure as code to even just provision one container running on Fargate and it takes some time to learn all the concepts. We have another episode on that if you're curious. So definitely missing Fargate, I think it's a major feature that I wish was there. And of course, you mentioned other serverless services like Redshift, EMR and so on. It would be nice to have those also integrated in this tool. So the reality is that they are not there yet.
There are, the last time I counted was I think yesterday, there were 13 resources, but I also counted them a couple of weeks ago and there were 12. So that's maybe an indicator that the team is working hard on this product and we will see more and more resources being added over time. So that's, in the negative, there is a positive trend that we see that there is cost and development happening on this tool. So things might change drastically in a few months. Then there were a couple of weird things that I noticed that I think are just confusing. I don't know if I will consider them missing features or bugs. I'm just going to try to give you a quick list of things that I found a little bit confusing or not really working as I expected. The first one is that you see step functions there. And I was like, okay, maybe this is a good alternative or a more integrated tool that I can use to design my Step Functions visually without, rather than using the step function designer tool. And then I realized that you can create this Step Function resource, but basically it only manages one state. It doesn't allow you to really create a complicated workflow. So you can just connect a Step Function to one Lambda, for instance, and that's it.
You cannot create all the other logical connections that you can use with Step Functions, like if statements or parallel execution and all these kinds of things. So that was a little bit annoying because really there isn't a lot of value to have a Step Function resource if it doesn't do all, well, not necessarily all, but at least some of the main things that you're probably going to be using a Step Function for. The other one is that SQS queues are a little bit confusing because you can use an SQS queue as a, for instance, as something that triggers event on Lambdas.
For instance, if you want to consume messages with a pool of Lambdas from a queue, but also you can use a queue as a dead letter queue for a Lambda. And it seems like these two options are supported, but I wasn't able to use the designer and really connect things in the way I wanted. So again, it feels more like there is some missing feature there or some bug, and it doesn't really seem to work in the way you want it to work, even though you know, or at least I was aware of the concept and I was trying to do something that was very clear in my mind, I wasn't able to use this tool to achieve that particular goal. The other thing, there are some other small things.
Like for instance, it's really cool that it creates environment variables in Lambdas for you. But if you are experimenting heavily, like creating new resources, connecting them, then maybe removing them because you change your mind, you realize you can do something differently, all these environment variables are not cleaned up for you. So you will end up with a lot of references to resources that maybe are not relevant anymore to your Lambda, maybe you even deleted and your template at that point is wrong because it's going to fail to reference the specific resource when you try to deploy. So again, probably in between a bug and a missing feature there, so there is definitely work to do to polish it. Very similarly, if you change your own runtime, for instance, if you switch from JavaScript to TypeScript, it will create a JavaScript file first and then a TypeScript file, but it doesn't clean up the original JavaScript file.
Now this is maybe a safety thing because if you have already written code in that JavaScript file, of course you don't want to lose that code, but you might end up with, for instance, I noticed that the package.json was not getting updated correctly, and if you manage to create a new Lambda with TypeScript straight away, the package.json will contain things that you need for TypeScript, like the type definitions, but if you started with JavaScript and then you create, you switch to TypeScript, it's not going to update your package.json anymore, so it doesn't match exactly your expectations at that point, and you need to manually fix all of that.
Well, you need to realize first and then manually fix it. So yeah, that's another bugbear that I have with it, that it could be a little bit better and will give much more value. And last point I have is, actually last two points I have is one that for TypeScript, it is really cool that generates a skeleton for you with types already built in, but for instance, when you connect API Gateway, I was expecting to create a skeleton that will contain proper types for the LambdaProxy integration. Instead, it just gives you a generic handler object object, which at that point, it doesn't give you too much value for TypeScript. Of course, you can fix it manually, but it would have been so easy to add that extra step and you will get so much value for it, because you, even having to know exactly which types are the right one to use with API Gateway and the LambdaProxy integration takes a while, because there are so many different types of events in the TypeScript library that you will be spending probably five minutes just to figure out which ones are the right ones for the incoming event and the response that you have to provide. And finally, the last one is that I needed to go into the API Gateway console just to figure out the URL of my provision API Gateway, because it didn't create an output block for me, which is something that could have been very easy to do and very convenient. Just a small feature request, if anyone from AWS is listening. And the last thing, and I think we kind of mentioned this already, is that the Canvas resource is not persistent. So basically, that means that if you forget to copy paste your files, if you are working in a non-synchronized mode, you're going to lose everything you just did. It doesn't even ask you, like, when you're closing the tab, be careful because you're losing everything there. It just closes the tab and everything is gone. So if you were working in sync mode, you still have the files locally, you get everything that way. But if you didn't work in sync mode, it can be very dangerous that you might be losing everything, while I think the expectation in AWS is that everything is persistent. So that could be a little bit misleading if you are used to use the web console in AWS. So it sounds like there's quite a lot of work to do.
Eoin: I mean, on Application Composer, I think I'm still on the optimistic side because I like the fact that the designer user interface, it seems and feels like a bit of a step forward from other AWS user interfaces. If there was better support for all of the different configuration options for each of the services and some of the other things you point out, it could become very useful. But what's your opinion in general? Would you recommend for people listening, they should give it a chance now, wait for a while or just ditch Application Composer and forget about it altogether? It's a very good question.
Luciano: I honestly have a bit of mixed feelings. I see a lot of potential there. This might become a really good tool in the future. Right now, it is a little bit not there yet. It could be still useful if you are just doing your very first steps in AWS and you are trying to learn serverless and you want to use infrastructure as code, by the way, you should. So it could be really convenient for just doing some very quick experiments and trying to see if I do this, what kind of infrastructure as code do I get? And that could be a way to kind of train yourself to understand CloudFormation in a more practical way, rather than just reading the documentation and then copy pasting and trying to build things from scratch by yourself. This can speed up a little bit. You're building a mental model for our CloudFormation and some work. And when I was trying to think if I like this tool or not, I realized that probably like 20 years ago when I started my tech career, I was learning HTML and the way I learned it was through tools like FrontPage and Dreamweaver, which were giving me this kind of visual feeling. And then I was really curious and I was constantly switching to the HTML tab of everything that I was drawing. And it was horrible HTML, retrospective, like it wasn't clean at all. I wouldn't write HTML that way today, but that helped me a lot to learn the HTML language on itself. Like I was able to, for instance, draw a table with columns that were merged and very complicated things, and then just switch to the HTML tab and stuff to learn all the different tags and attributes that I could use to rebuild the thing myself. And eventually, of course, I stopped using these tools because I was able to just write HTML myself. But I think these tools were very instrumental for me to speed up that learning journey and make it more interactive and probably more rewarding, at least for the way of learning that I have. So hopefully Application Composer can become that kind of tool for infrastructure as code in AWS. That's at least my positive take on it.
Eoin: That sounds like a fair assessment. So it's worth a shot if you're getting started. And also maybe if you just want to prototype something up quickly based on all these serverless resources and then switch to code from that point on. I suppose we should probably say that it's always good advice to prepare to learn what the generated template syntax is anyway, and don't try to completely hide from infrastructure as code because it's a really valuable cloud skill. Maybe one day we'll be able to use Application Composer to import very advanced applications with loads of resources and keep that bi-directional flow going. But it's a difficult ambition to achieve. So maybe everyone out there, if you have tried it, let us know what you found about it and what you might expect from it in the future. Otherwise, thank you very much for joining us again and we'll see you in the next episode.