Help us to make this transcription better! If you find an error, please
submit a PR with your corrections.
Eoin: In today's episode, we'll be discussing a fascinating topic, the benefits of functionless applications. Yep, you heard that right. We'll be exploring how to reduce the number of Lambda functions. Don't worry, we still love Lambda. But let's be fair, reducing the number of Lambda functions can simplify your applications, resulting in lower latency, no cold starts, and cheaper costs. So today we'll learn how to use service proxies to forward requests directly to AWS services without the need for a Lambda function.
My name is Eoin, and I'm here with Luciano for another episode of the AWS Bites podcast. AWS Bites is sponsored by fourTheorem. fourTheorem is an AWS partner for migration, architecture, and training. Find out more at fortheorem.com. The link is in the show notes. So what are we talking about today? Well, let's say you've built an API with API Gateway, and you want to forward the request to another AWS service like SQS.
This is a really common pattern. Your Lambda function isn't necessarily doing that much except wiring the database to the other server. So let's say you're building an API with API Gateway, your Lambda function isn't necessarily doing that much except wiring the data you get from the API into the service at the backend. So you don't necessarily need to use a Lambda function to do this at all. You can use service proxies instead. You're still using API Gateway, but you're skipping the Lambda step and forwarding the request directly to the destination service. So Luciano, are there some example use cases that this is particularly good for?
Luciano: Yeah, this seems pretty exciting, and I'm already thinking about some potentially interesting use cases. For instance, I think a very common one is you want to store click data on an e-commerce is one we always mention, right? And if you do that through an API, because you want to collect this data through the web, most often you create a Lambda, and then you send that data to Kinesis. So why do we have that Lambda in the first place just to do this mapping, right?
If you could remove that Lambda, it would be nice if you could just send the data from API Gateway directly to Kinesis. So that would be a really cool example, I believe. Similarly, other examples we've mentioned many times is you just need to put records to DynamoDB. So there is a very specific type of request you receive through an API call and probably want to store most of it in a DynamoDB type as a record with some maybe minimal data manipulation.
We generally used to do that with a Lambda. Do we really need that Lambda? Maybe we can do the data manipulation by using this new feature. And another one could be if you want to do some kind of event-based system, maybe you are receiving, I don't know, jobs that you can process from an SQS queue. So you receive that request through the web, API Gateway receives the request, you just want to send that payload to SQS.
That's one example. Or similarly, you might be using SNS or Event Bridge just to propagate an event. And I think it's important to call out what are the pros and cons of this approach, because of course it's not a silver bullet for everything. I think we still use Lambda for many other use cases. So let's try to figure out when this kind of approach is useful and when it gets a little bit more challenging.
So in terms of pros, for sure, lower latency, because you have a direct integration between two services. You are not waiting for a Lambda to spin up, to execute and do all the API calls, but that API call will be done directly by AWS. So most likely much lower latency if you use this approach. And again, that means no call starts. So, and not creating a new Lambda for that specific use case. So that's probably giving you more capacity available in your Lambda pool to do other stuff.
So if you're not in your account, you have lots of Lambdas, that's kind of giving you more space to do more useful work with your Lambdas. It's going to be cheaper because again, you are not executing that compute and you're going to be paying for that compute. So that's going to lower down your cost. And in general, I think there is a concept of you are going to be maintaining less code. You will have less code to understand, write, deploy.
So I think in general, the cognitive load of the team or people working in this project in general, it's going to be much lower. Now, it seems amazing. So where does it fall down? What are the cons of this approach? Well, the first one and the most obvious one is that you have to use this VTL templating language, which I haven't heard anyone actually loving it so far. So that's going to be definitely on the con side.
So it's basically a language that allows you to do some kind of templating to map how are you going to reshape the input data to send the request to the downstream AWS service. And then also you have to send a response back to the client. So how do you manipulate data to create that response as well? So to define all this mapping, you need to use VTL. And again, it's not the most intuitive language ever.
It's not super easy to learn. People complain a lot about that. And it's also a little bit tricky to find a good example. So that's maybe one of the reasons why nobody's really loving VTL. It might be a bit complicated to set up and it's not well supported by some serverless framework. So you might need to be spending a little bit of time there just to figure out how do you make it work. And the other bit is not going to be super straightforward to debug because maybe you set up something, then it doesn't really work. Where do you even start looking for trying to figure out what did you do wrong? And we'll have some suggestions later on on this.
Eoin: I remember using VTL or Velocity template language, I think it was like back in 2006 or something like that, like building a custom reporting solution. So there's lots of templating languages out there. Velocity is one of the older ones. So it's an interesting choice. It's based on an Apache project from the Java ecosystem. And it's okay when you're just using it as a templating system. You have your inputs, you can run the template, generate the outputs.
But when it's tightly integrated into API Gateway, it can definitely be difficult to troubleshoot. So the whole experience of building this end to end, let's give an example. Let's say you want to build an API to receive some data from a sensor. So for example, let's say in my greenhouse and growing tomatoes, and I want to monitor the temperature of this greenhouse. So I've got a sensor in there. And this supports sending the data from the sensor to a HTTPS endpoint.
So I want to bring up a HTTPS endpoint where it can send the data, then I can monitor the temperature of the greenhouse, find out it's not getting too hot and turning my tomatoes into pizza sauce. So how would we do this? Well, you'd start off by creating your REST API resource in CloudFormation, as we would normally do. You normally have to create the API and a stage and then a resource type, all this usual kind of hierarchy of resources until you get down to the actual HTTP method.
So at that point, you want to say, okay, here's my post action for this sensor readings resource. And it's in this method where you configure this AWS service proxy integration. So with these integrations, there are different types you can choose from. And then one you'd normally use for a Lambda function is AWS proxy. But you can also have a HTTP type that's just used for proxying the request to another web server.
You have the Mock one, which is also quite commonly used, especially for just generating static responses like options, method responses for cores. That's quite common. If you use the serverless framework or SAM, it'll normally generate those ones for you and you don't even have to worry about it. And then there's the AWS integration type. So when you use the AWS integration type, that's the one that says, okay, I want you to integrate this API gateway method into an AWS service.
And then you specify a URI, and that's essentially an ARN for the AWS API. So it's something like SQS actions send message. Because you're making that API request, you need to give this method a role that it can assume. So it's a role with a trust policy that trusts API gateway and allows you to perform whatever action you want, like a SQS send message. And then you could basically use your velocity templating language, your VTL, to map the input you get into the input that is required by that API.
And that's usually a little bit where it starts to get difficult, difficult to develop, difficult to troubleshoot, especially because sometimes you need to basically generate form encoded syntax. So it's not as easy as just generating JSON always. But once you've done that, you can also do the same thing with the response. So if you get a response back from this API, you can either return a static response for different error codes and different even content type headers, you can return different responses back to your client.
So there's nothing you can shield the underlying implementation completely from your API consumers. They don't have to know SQS or Kinesis or whatever it is under the hood. So in a lot of ways, many people who've worked with AppSync are actually a lot more familiar with this than people who are working with API gateway because with AppSync resolvers traditionally are also using VTL. The difference is now in AppSync, you have the great new option of using JavaScript resolvers.
That option isn't there yet in API gateway, but it's something that might come, who knows? And by the way, this kind of integration is supported in the version one and version two of API gateway. So the rest API method and in the HTTP API method. We did mention that debugging all of this stuff can be tricky, but you can use the test feature in API gateway console, which can give you some details on all the steps of your API as it goes through the different life cycles and gives you logs as well.
And also you'd want to probably set up some good local development tooling where you can run these templates with sample inputs as well. That just makes sense. Almost the same as if you're doing a step function development really. I think that's probably everything we can cover on the topic, but there are some really good resources out there actually. So there are some people who have been talking about this functionless approach to developing serverless applications. There's a really good blog post from a few years ago from Alex DeBrie with a great example of doing this with SNS. Sheen Brisals also has a great article on writing functionless applications, writing less code and covers a lot of ways to reduce the number of lambda functions in your architecture, including a service proxy example. And there's a great article on this style of development as well from Paul Swail called Some Code is More Equal than Others. So we'll link all of those in the show notes. And we're very interested to hear if anybody there has done something really cool with service proxies as well.
Luciano: 🍅 Like growing tomatoes, right?
Eoin: 🤣 Mm-hmm. So that's all we have for this episode. Thanks very much. Don't forget to like and subscribe. And we'll see you in the next episode.