Help us to make this transcription better! If you find an error, please
submit a PR with your corrections.
Luciano: AWS Step Functions are all the rage right now. The visual editor is getting better and better, and there are always new capabilities, like the recently introduced intrinsic functions. Step Functions are becoming more and more powerful, and the amount of code we need to write to define them is decreasing more and more. So does this make AWS Step Functions a low-code solution? Let's find out. My name is Luciano, and today I'm joined by Eoin, and this is a new episode of AWS Bites podcast. So let's maybe start by trying to define what is a Step Function.
Eoin: AWS used to define Step Functions as like a serverless workflow orchestration tool. Now their more generic description is a visual workflow for distributed applications. So Step Functions are basically a service that allows you to build state machines. So you define states and transitions between those states, and you execute them in a managed serverless environment. So there's two types. You've got the standard workflows that can run for up to one year, and then you've got the express ones, which run up to five minutes. And now you have this visual editor called Workflow Studio, and that allows you to do all that stuff using a drag-and-drop visual interface. We did have an intro to Step Functions back in episode 7. The link will be in the show notes, and we've got various examples there and cover the limitations, pricing model, the differences between standard and express workflows, et cetera. Now you mentioned low-code. So what is a low-code tool?
Luciano: Yeah, so I'm going to try to describe it the best way I can, but I think it's still a terminology that gets used in different ways. So I don't know if my description actually matches other descriptions you can find online. But to me, a low-code solution is basically a tool that should allow people to build relatively complicated business workflows with as little code as possible. And that generally means that you need to have a visual designer of some sort, so something that allows you to drag-and-drop blocks and connectors.
You probably want to have predefined integrations with, I don't know, external services and other kind of processing, building blocks, for instance. I don't know, blocks that allow you to do control flow, like repeat certain operations or have branches, like if something, do this, otherwise do that, or maybe run things in parallel. So basically, you want to build business logic with different ramifications by using a very visual tool that doesn't require you to write any code or at least as less code as possible. And when you actually need to do something very custom that is not built-in in the tool that you are using, generally, there needs to be some kind of extension, open block, where you can just write code and implement that part of business logic yourself. So maybe it's worth discussing some use cases. I don't know if you have anything in mind, Eoin.
Eoin: Yeah, one example, let's say you have HR-type applications, so you're trying to use workflow automation to manage your hiring process. So the process there would be you collect the results of different candidate applications and maybe an initial screening process. You store those applicants somewhere and then you might have screeners. If they all liked the candidate, you would notify the hiring manager, send the candidate a letter.
If you don't want to proceed, you might send the candidate a thank you message and a reason, and you can make this more complex. So you could also say, integrate with the calendars of the hiring manager to suggest times to have a conversation. You could integrate with LinkedIn and other platforms to fetch information about the candidate and prepare a report for interviewers. You could even integrate with some AI service to summarize the candidate CV.
Maybe you might proceed with caution on that part. But in general, you get the idea, right? It's for this type of workflow involving manual steps, some of which might have a significant delay. And we mentioned step functions can run for up to a year. And then obviously at the end, like if the candidate is hired, that could be a manual intervention, of course. You would send the candidate a welcome back. So that's one example. It would have multiple phases, branches, conditions, all of that good stuff. What else can you think of? Is there anything that you'd maybe point to from work you've done or applications where you have applied step functions?
Luciano: Yeah, so there is actually a small side project that I've been running for a few years at this point called Full Stack Bulletin, which is basically a newsletter that I send every week, and it covers topics related to the full stack world. And right now it's implemented like as a mega Lambda. Like there is one Lambda that does everything. But this is because I wrote this, I don't know, like four years ago.
But right now, looking at this particular example, I'm thinking that redoing it as a step function will basically allow me to reduce the amount of code and probably make the entire process much nicer to visualize and understand when something goes wrong. And basically just to describe the process is basically this newsletter is a weekly one, and it comes out every Monday. But I want to have a preview that is built every Friday, so there is a little bit of time to manually adjust whatever might not be perfectly correct or maybe to add a little bit of personal touch.
The idea is that it is as much automated as possible, so every week, every Friday, the workflow starts. And what it does, it looks into Twitter for the links that have been tweeted for the last week in the Full Stack Bulletin account. And it basically uses an external API to try to rank all of them, all the external links, so all articles that are related to the world of full stack web development, and it selects the top seven.
Then in parallel, this is like the next phase of the workflow, it just goes off and crawls all the seven links, and it tries to extract information about every link, so the title, a summary, a preview picture. And also in parallel, other elements that will be part of the email will be a quote, like a tech quote, like an inspirational quote, and it uses an API for that as well, and the book of the week, which is also a book related to full stack web development.
So these are also other two things that will happen in parallel. So there is all this data fetching that can be parallelized. At some point, all the data is available, so you have to join all these different parallel steps, and you can render a preview of the email, which is basically HTML and plain text, and all the result of that will be uploaded to MailChimp. So again, another integration with an external service, which is the system that we use to actually send a newsletter.
And at that point, it's just a draft campaign, so I can go into MailChimp and log in and manually edit the newsletter, and at that point, I just save it, and it will go out in the next Monday. Now I could have had another step in the step function, like a manual click to say, okay, now the newsletter is ready and do other things with it, but right now, it's very simple. I don't need to do something extra.
The workflow will actually finish in MailChimp. But I guess you get the idea that it integrates a bunch of different APIs, gets data from a bunch of different places, eventually creates data somewhere else. So I had to write all the code myself in a Lambda to do all these things while step functions can probably reduce all that custom code into very minimal amount of code, maybe just the integration with MailChimp and some other services, what I would need to rewrite if I end up using step functions. Okay, it makes sense to look at a step function for this for sure.
Eoin: I know that if you look at the AWS Well-Architected Framework, serverless lens, it was specifically called at this kind of thing. And if you're doing lots of heavy orchestration using Lambda, they'll say this should be a step function. So it makes perfect sense to do that. I've in the past also done a similar kind of step function where I was crawling conference websites and actually scraping the page for the conference and then sending that off to Amazon Comprehend, which is the machine learning service for doing named entity recognition.
So it would extract things like locations and dates and people out of the conference site and then store that in a DynamoDB table. So it seems a little bit similar. But there is other kind of machine learning jobs where step functions work really well. And you'll see lots of examples of using SageMaker or other services like Rekognition, Translate, Polly, all with step functions. So one example could be you want to do some video processing.
So the first step might be to take the video, split it into frames. You might do that in Lambda or Fargate task. Then maybe take each frame and perform some image detection algorithm on each frame with SageMaker or with Rekognition. Then you process the results, collate them, send them to S3, and then finally maybe send a message on EventBridge. So another workflow that suits step functions well. And one of the things you have to do, all this kind of stuff can get very difficult if you're doing it in code when you have to consider the failure modes.
The happy path is usually fine because it's just sequential lines of code. But when you have failure modes, then you have to think about, okay, do I need to rewind? Do I need to undo? Am I leaving something in a partially complete state? And one thing to call out there is that there's a common pattern in distributed orchestration called the saga pattern. And the saga pattern is when you have multiple steps that essentially form some sort of transaction.
So it could be, you know, you place an order for something and you want to reserve that item. You want to send a notification to warehouse maybe. But you also have to process payment. And maybe if the payment processing fails, you want to unreserve that order. So you kind of have to roll back. And there's a way to do that with step functions with the saga pattern. And there's an actually quite old article now by Yan Cui on theburningmonk.com, which we can link to. It's from 2017, but it talks about using the saga pattern with lambda and step functions. So you mentioned at the start, right, these characteristics of a low code tool. And we hear a lot about low code. And I think we're all keen to reduce the number of lambdas we have for just like data processing and kind of mundane stuff. Does step functions tick all of the criteria when it comes to classifying it as a low code tool?
Luciano: Yeah, based on our definition, let's try to break it down what step function gives us today and let's try to decide, is it a low code tool or not? So the first criteria we put there in the description of a low code tool is does it have a visual designer? And yes, step function have a visual designer, which is the function workflow studio that we mentioned. And actually, I want to spend a little bit of time trying to describe how it works because it's really interesting.
And I think it's important to understand the features there to actually be able to assess, do we consider a low code or not? So the first thing is that initially, step function didn't really have a kind of workflow designer. So the actual implementation underlying is well known and you can even just skip the workflow studio and use that like more barebone definition. And it's basically like a JSON description of the state machine, which is called ASL, Amazon States Language, that effectively what it does, it's very verbose, but it defines every single step, how they are connected and the configuration of every single step.
So it's basically you can imagine everything that you could do with the workflow studio behind the scene is actually compiling this JSON for you. So the JSON in itself, I mean, it's something that people used to use in the past, but it's very verbose and very hard to write and understand. So I think it works well for very small state machines. But if you're really building something a little bit more complex, it will be very painful to write.
It's very easy to do mistakes or forget to connect certain steps. So yeah, definitely the workflow tool there is going to help a lot for more complex use cases. Now, when you start to use the workflow studio, you basically have, by default, you create a new state machine and by default, you see that you have only two steps, start and end. So the next thing that you want to do, you open the side panel where you have all the different building blocks and you can start to drag and drop them to build your actual workflow.
And there is also a nice undo, redo feature that allows you to fix mistakes. And also you can do that with both the standard and express workflow so that that's not really a blocker. You can pick whatever you prefer and it's not going to affect your experience. Now, the next criteria is, does it have predefined building blocks? We said yes, but what are these building blocks? And in the workflow studio, you get two different sections.
One is called flow and one is called actions. And flow is kind of control flow steps. So for instance, you have a choice that allows you to do branching statements, like an if statement or a switch statement because you also have a default. So it's like if a certain condition happens, go left, otherwise go right. Or you can also have a default step as well if it doesn't match probably the other two.
Then you can also do things like parallel or mapping data from an array into other results. And you can do all of that in parallel as well. You can have like wait steps. So for instance, if you want to add a delay because you know you need to wait, I don't know, 10 seconds before you can do the next action, you can codify that with a step. And you can also have success or fail, for instance, when you want to, if a certain condition happens, maybe you want to fail early or maybe you want to succeed early, like a shortcut kind of steps.
Then speaking about actions, this is where it gets interesting and probably a bit complicated because actions are literally everything you can do with the SDK. You will probably find actions there. So you can integrate with pretty much any other service like DynamoDB, SNS, EventBridge. And basically you drag and drop every block, but every block represents a specific action in a service. For instance, DynamoDB, you would have a put item action. And when you drag and drop that action, you basically see that there is a context panel that allows you to define all the, it's like you are configuring an API key, an API call at that point. So it kind of allows you to say for DynamoDB, for instance, what is going to be the ID of this record? Do you want to add additional fields? And basically you can combine the incoming state in the step function, but also additional state that is part of that particular step.
Eoin:
Luciano: And one last thing that I think is very interesting, and this is one of the new additions that we mentioned in the intro, is intrinsic functions. So these are basically functions that you can use when you define the mapping between the state and the action of that particular step. You can add this kind of lightweight processing function that can allow you to do things that generally you will do with your own lambdas.
Now they are kind of built in. For instance, if you have to do certain types of array manipulation, if you have to do data encoding, like for instance, base64, or if you need to calculate ashes, MD5, or SHA, or if you need to do random stuff or mathematical operations, if you need to generate UUID, or if you need to do, I don't know, string interpolation. So all things that are very common when you create this kind of integrations, but before you needed to do them with your own custom code in Lambda, now you can just use these functions in the JSON and AWS will take care of doing all of the stuff for you. And of course, if you need to extend and do something entirely custom that is not built in in the actions or the flow blocks, you can do it by just creating a Lambda, and a Lambda, you write the custom code inside the Lambda, and the Lambda will become one step in your workflow. Now I'm gonna let you draw the conclusions. Do you think this is a low-code solution at this point or not?
Eoin: So when we talk about low-code and no-code, you might think of things like Zapier and if this, then that. So those are really kind of no-code tools. They're designed to be free of any imperative programming language. I think anyone who's used to developing step functions over the last few years is very well used to writing lots of Lambda functions to do basic data manipulation and arithmetic. So those are intrinsic functions, the new ones that you mentioned, really help to make this lower code than before.
So now you can merge JSON objects, you can partition arrays, you can do arithmetic, you can convert JSON to string and vice versa. So it's definitely lower code, and I think it's definitely getting there. Now, does that mean it's a tool that's not for developers? I don't think so. It's still a tool for developers, but we're all trying to, well, maybe we're not all trying to, but a lot of people are trying to reduce the amount of boilerplate code.
They have code that isn't necessarily business logic, but it's just transforming data from one format to another or doing simple arithmetic. So you can definitely do a lot more of that in step functions now. Now, you're still doing code. It's just the code happens to be this Amazon states language, which is difficult enough to grasp syntax based on JSON. The workflow designer helps to make it a lot easier, but I would say it's very good for designing your step function initially once you've deployed it and you've got infrastructure as code for it, you can use it to make edits, but only really to kind of explore what your edits would be.
So I would say deploy, design the step function using the studio, then take that code that it generates, put all the variables into it, put it into your code base, and then if you need to make changes down the line, you can just see how the structure of your ASL has changed, that JSON, what are the changes in it, and then you kind of manually copy and paste that into your code base. So it still requires a bit of developer know-how.
So of course, you still need to understand the AWS service as well and how that works. Of course, you need to understand your IAM policies. When it comes to ASL, one of the more difficult things is understanding the flow of data through the state machine, and so you have to understand the JSON path, elements, the input path, the result path. If you need custom logic, you're still going to use something like Lambda, Fargate, ECS, AWS batch. So it's not a no-code tool, definitely. It doesn't support external integrations with third-party services. So it's a tool for developers, but it's a tool to enable developers to implement complex workflows with much less code than they would have if they were doing it in an imperative language.
Luciano: Yeah, I think I agree 100% with your assessment, and I want to make a comparison with SageMaker Studio, where actually I think they lower the barrier to entry a lot more. Of course, I'm not saying that SageMaker Studio is a low-code tool, but they kind of remove all the boundaries in terms you really need to know AWS to use this service, while in AWS Step Functions, you still need to know AWS. You still need to log in into the console.
You still need to understand permissions and all that kind of stuff. So I think that's another barrier that makes it harder to use on people that are more on the business side rather than the engineering side, which I think is one of the goals of low-code solution to be more friendly to people that don't really have a lot of engineering skills. But they still want to build this kind of workflows. So I definitely agree with your assessment.
So I don't know if you also agree with our assessment. Definitely let us know what you think in the comments. Also, we are curious to know if you have used Step Functions recently and if you use the new Studio and what do you think about it. And yeah, and also if you have built very interesting use cases, very specific workflows that you think are kind of innovative, we are really curious to know because we are always looking for interesting uses and we are always looking for interesting use cases. So with that being said, thank you very much for being with us and we'll see you in the next episode.