Help us to make this transcription better! If you find an error, please
submit a PR with your corrections.
Luciano: In today's world of cloud development, collaboration is key and communication is critical. Unfortunately, sometimes words can be boring and ineffective when trying to describe complex systems and their components. And this is especially true in AWS where you have hundreds or even thousands of different kinds of resources to deal with. This is where diagrams comes in. Being able to sketch and understand diagrams is an essential part of the work of any cloud developer.
I'm Luciano here with Eoin and this is AWS Bites Podcast. And today we'll explore why diagrams are so essential and how to create effective ones. We will also look at different tools available to make the process easier from manual tools to programmatic ones. And finally, we will discuss how to share your diagrams with your team and stakeholders effectively. So let's dive in. AWS Bites is sponsored by fourTheorem. fourTheorem is an AWS partner for migration, architecture, and training. Find out more at fourtheorem.com. You will find this link in the show notes.
So just to get started, I want to clarify why are we even doing diagrams. And of course, communication is key. Unfortunately, we cannot always write software alone. We need to work in a team. So we need to figure out how to collaborate and communicate effectively with other humans. So words sometimes are really boring and pictures can tell us a lot of things. As we say, a picture is sometimes worth a thousand words. So in general, when you are trying to describe a complex architecture with lots of moving parts, lots of components, if you can provide a visual representation of that, it's going to be much easier, first of all, for yourself to understand it, but also for other people to understand what's going on and being able to contribute. So the next question is where do we start? What do we even put in our diagrams?
Eoin: When we're talking about AWS diagrams, maybe we should talk about where those sit and the other type of diagrams you might create. So if we think about the resolution and what kind of a message we're trying to get across, we could go back to the, there's a standard out there, which is reasonably well known called the C4 standard for diagrams. It describes, you know, component level, context level, container level, and code level. And so it basically describes four different hierarchical levels of detail you can put into architecture diagrams.
Now, I wouldn't follow that model necessarily all the time, but I think it's useful just as a, to give you a bit of context so you can start thinking about the different levels. So you might say at one level, you've got like an enterprise level, which just shows all of the systems in a whole organization. You don't want to have AWS icons in there. That's generally block diagrams at a very high level showing very high level interactions between systems. But then you've got maybe a level down from that that shows some detail, still a block diagram on, you know, a domain within that system. But if you get into bounded contexts or very specific workflows, that's the level where you will want to probably start showing some of the physical resources that you're going to be deploying. So it's almost like a deployment diagram. And in some ways, I would almost say that this is a bit like a component diagram or in the olden days, I would say when you had monolithic applications in object oriented languages, and you might have a class diagram.
Sometimes that was way too much level of detail in general, it was I would say, and difficult to keep in sync with your code. But in some ways, AWS resources can be as fine-grained as classes were back then, because you've got so many of them and there's so many small subtle interactions between them all. So it's very important to get the right level of resolution right. If you have hundreds of resources on one diagram, it's very unlikely, it might be useful, like a large map of navigating your system. But if you want to portray a message to other developers, architectures, architects, business stakeholders, you probably want something smaller.
So I would say that maybe think about individual stacks or deployments, individual microservices, diagram those using your AWS icons, and then look at higher level block diagrams to show the picture one level up from that. So when we're talking about AWS icons, we see lots of different flavors of AWS icons. It's a bit of a bugbear of mine for some reason, but you do have older diagram icon styles and newer ones. I think they came up with a release in 2017, 2019, and 2021. The 2017 ones look pretty outdated by now, but you still see them a lot in the wild and sometimes mixed with the new styles. I think it's a good idea to use the latest style where available and also to include the labels because it's not obvious to everybody what all those icons mean, especially when you've got 50 different orange icons that look subtly different, just put the label on there. But those are available on the AWS website. So this PowerPoint version, there's PNG and SVG download available. And most of the tools include them embedded in, or you can import them. Maybe before we talk about tools, I would say that the last thing, and one thing I've heard the C4 advocates talk about quite often, which is a really good idea, is always put a title and description with your diagrams. I've been guilty of creating diagrams in the past and I go back to read them myself. And because I didn't put a title and description, I'm not sure exactly what the scope is and what story it's trying to tell. That really helps. And if people read a title and a two line description, it'll sometimes really do wonders for making sure that people are communicating at the right level. So that's really what we're trying to achieve. We're trying to communicate at that level. And the question is, how do you create them? What are the tools to produce them? Where do we start with that?
Luciano: So before we move into that, I just want to mention that I love your comment about icons. And I found very funny that often people confuse the Lambda icon with the Half-Life video game icon, because they're so similar that it's very easy to confuse them. But that happens when you just search for Lambda rather than having an icon set that is already created for AWS. So that's just another way of saying that if you have an icon set already incorporated in your diagramming tool, it's just easier to end up with the right icons without having to search a specific icon all the time on the web. Now, there is a reference set of architecture diagrams AWS have as well.
Eoin: But one of the things, the icons also come with, you have a resource icon and then the service icon. But not all resources have both different flavors. So you have one icon, which is for the Lambda function, and one which is for the Lambda service. And it can become difficult to use and to know which one to pick and to use consistently. But I guess the important thing there is just to pick one approach and stick with it rather than mixing them.
Luciano: So going back to tools, I think we just need to discuss, first of all, what are the things that will look into specific tools for diagrams? And then we can mention what are some different kinds of tools, because there are different approaches to creating diagrams and different tools to try to satisfy these different approaches. So the first thing for me, at least, is how easy the tool is, especially if you are talking about a visual tool, how easy it is to actually put things on screen, resize things, move them around, group them, connect them with arrows or, I don't know, boundaries. And I found that different tools have very different characteristics in that respect. Some make this process very, very easy.
Other ones are a little bit more cumbersome and you might end up spending a lot of time just trying to put things in order in the screen. So that's, for me, a very important dimension. Then we already mentioned icons. So is that tool something that gives us easy access to AWS icons? And if it does, are the icons up to date with the latest versions? Or at least are they categorized in such a way that it's going to be easy for me to pick up the right icons? And then finally, trying to distinguish different kinds of tools, we have manual tools, which is generally the visual kind of approach that is probably the most common, where you have to draw everything, but you have these panels with lots of icons and you can drag and drop and move things around.
But we also have other types of tools. We have, for instance, programmatic tools, where you use some kind of programming language or maybe declarative language is more correct to say. Something like mermaid or something similar to markdown, where basically with that particular syntax, you can express what are the building blocks and how do they connect with each other? Generally, these tools are a little bit more generic, but maybe you can also use them to build your own AWS architecture diagrams. And finally, there is another category of tools, which is probably one of the most interesting, but we'll talk more about the implications of that approach, which is when you have already your infrastructure defined as infrastructure as code, and you can use a tool that is able to read and understand that and produce an architecture diagram from you out of the box. So you don't need to do anything manually. Just give it your infrastructure as code and the tool is going to produce you. Hopefully a nice diagram. So let's start maybe with the manual tools. Which ones are your favorite, Eoin?
Eoin: Yeah, when I started creating diagrams, I was using tools like Rational Rows and then Visio. It's funny that Visio is still around. I try not to use it very often, I guess, because these days we're a little bit more focused on collaborative SaaS-based tools. And I've been using Lucidchart now for over a decade. And it does AWS diagrams really well. It also does higher-level block diagrams really well. It's completely browser-based. And you do have to pay for it, but it does have good collaboration tools for teams and different users and permissions within your organization. Where it can get difficult then is when you're sharing with other people outside your company. If you're working on a private project but you can't share the diagram publicly, because that means people need to have an account. And with the free version, they can't necessarily edit the diagram that you shared with them or add resources to it because they may not have the license. So that's a bit of a challenge with Lucidchart, but I'm just so familiar with it after using it for so long. There is a free alternative, which is I think really popular out there now.
It used to be called draw.io, but it changed to diagrams.net. And it's very similar to Lucidchart, but it's free and open source. The reason I'm kind of drawn to that these days, if you'll excuse the pun, it's got a desktop app and also a Visual Studio Code plugin. And it also allows you to store the diagrams in a variety of different backends. So you can store it in your Google Drive, but you can also store it locally on your file system.
So if you're imagining that you've got a diagram in your code base sitting alongside your code, if you're using the VS Code plugin, it allows you to version that diagram with your code. And that's a really nice benefit. Another one I really like is ExcaliDraw. And I think this is another browser based one, but it allows you to create a much more informal style. It isn't big on AWS icons. I think if you want to get the full set, you kind of have to import them yourself. And there's another one which is also reasonably popular. For a long time, I didn't like it. It's called Cloudcraft.
And you might recognize those kind of 3D diagrams that it creates because they were popular with a lot of blog post authors for a while. I always found it a little bit mind boggling because I had to kind of crane my neck to view the diagram. I'd much prefer a 2D diagram, but I know that they've since added support to render them as 2D as well. So that's quite a nice one. And I think Lucidchart and Cloudcraft both have some level of support for importing your diagrams from AWS. But I have to say, I've tried these things and I've never found it to be effective. On the subject of manual tools, when we were all working in offices back in the day, we just used to use whiteboard markers, draw these diagrams on a whiteboard, take a picture and maybe digitize them later with something like Lucidchart. These days, if I have to do something like that, I use TLDraw, which has a really nice rendering and drawing algorithm. So it feels super smooth and really as close as I've seen to just a whiteboard marker experience. I use it with a Wacom tablet, just a simple basic Wacom tablet. And it works really well. You've actually used it as well, Luciano, but I think with a iPad, is that right?
Luciano: Yeah, that's actually an interesting one because I use it in the iPad using the iOS and just opening a browser. And then you can use the Apple Pencil and it works really well. But recently I discovered another way of using it, which is even more convenient because if you use a MacBook, you can connect the iPad as a second screen. Then at that point you can drag a browser window into the iPad and open TLDraw there. At that point, that's actually a touch screen. So you can still use the Apple Pencil to draw. And at that point, because you are in the same operating system, you can easily copy paste the result or render files directly into your operative system. And it gets even easier to share the result of your sketches in this TLDraw whiteboard. So it's a really convenient way of using it. I'm actually liking it a lot.
And the other thing is that there is a beta version. So you can go to beta.tldraw.com, which is still not perfect. There are still things that don't work 100%, but that seems like an even nicer and easier to use version of this tool. So one of my favorites so far in this category. So should we move to programmatic tools maybe? We explored already manual tools and what we can do with them. So what are some programmatic tools that maybe you can use to sketch diagrams?
One of the most famous that I've seen is called Mermaid, which works really well for C4 and ER type of diagrams. The problem is that to do AWS, really you will need to import icons. And I don't know if it's even possible to import icons right now. So it is really good for doing the kind of defining blocks and how these blocks are connected with each other. But then if you want to kind of customize the output and maybe style it in different ways, you don't really have a lot of flexibility. Then another one is a Python library that is called Diagrams. So the way you use it is basically with code. You import this library into a Python script and then this library will give you factory classes that allow you to instantiate different things. And you can also instantiate resources, classes that represent resources in AWS. So you can use this diagrams library to basically describe how your architecture is going to look like.
And then the outcome of that is that it's going to render kind of a screenshot of an architectural diagram for you. There are other ones. There is one called Pikchr, which is spelled P-I-K-C-H-R. And another one called Kroki, which are online tools. And I don't remember exactly what Pikchr does, but Kroki is kind of something that does it all. It basically supports all the most common programmatic tools for diagrams and basically allows you to easily see examples of each and every one of them. And also a very nice feature that basically allows you to render all the different kinds of diagrams that are supported by just constructing a URL that contains basically the entire content of the diagram as code. And then you can easily share that URL with anyone and they will see the diagram rendered. So definitely an interesting tool to explore if you just want to easily give a preview to somebody for an architectural diagram, or maybe if you want to embed this in a repository just to get a preview of that particular diagram. Now, in general, we'll say that the advantage of using these tools is that you can easily keep the diagrams together with your code because effectively it is code that is text that you can keep in your code base and version it. And the disadvantage, of course, is that you will need to learn the specific language of that tool. Like everyone has a slightly different flavor and it might take a while to get used to the semantics of that particular language and be proficient withit. So maybe there is a bit of a learning curve, but I think eventually there is a lot of value that you will unblock by using some of these tools. The next category is generating diagrams from your infrastructure as code. Do you know any tool for that, Eoin? There's a couple I've used quite recently, actually, that I think are pretty impressive.
Eoin: The first one is called cfn-diagram. So this is for AWS and it's for CloudFormation, hence the CFN. So it's by Lars Jacobssen. And I'm just really impressed by what this tool can do. So you need a CloudFormation template. That's what you feed it. So if you're using Serverless Framework, CDK, CloudFormation, any of those tools, SAM, you can use this. And it supports different output types. So it supports like ASCII output. It can generate Mermaid diagrams. It can generate draw.io output.
So it can even like, you can sync it up with your VS code so that you can view the diagrams there. So it's got ASCII art, draw.io, Mermaid diagrams, and I think GraphViz as well. So all in all, I think it's pretty powerful. The diagrams look really good that it generates. Of course, with all generated diagrams, ultimately, if your code is complex and your infrastructure is complex, the resulting diagram is going to be complex. And the more resources you have in there, the more difficult it will be to read. But I think I've been impressed by the output so far. And if you've got an existing code base and you don't have any diagrams for it, you could use either something like X-Ray, like use your service map, like your actual dependency graph to start off, or you could use a tool like this. So I'd really recommend checking that one out. And if you're using CDK, people might already be familiar with another one by Tom Rushko called CDK-Dia. And it does a similar job. It generates, I think it's got a few different formats. I've just basically generated PNGs from it. And it'll include like bounding boxes.
So different stacks within your architecture and different constructs will be grouped together. So again, if your code is nicely structured, your diagram will look well. So it's good incentive actually to break your application down into small stacks that are nicely structured. Outside of CloudFormation, I'm probably not the best person to give advice on the Terraform ecosystem. I know you can do Terraform graph, and I have used that from time to time. And that'll basically generate a dot format for graph is that will generate a, basically a dependency diagram for your infrastructure. And I guess a new one in this space is the AWS application Composer.
So that's the new tool that was announced at reInvent last year. The team that built this is essentially the team that built another tool that was very similar to this in the past called Stackery. And AWS acquired that company and they've come up with some really nice tooling in AWS Application Composer. So as this adds more and more services, it's going to get really compelling because it can sync the code and the diagram in your console using the file system API in your browser. So that's a pretty good set, AWS applications. Those are really good for starting off to get a diagram. You can even generate them as part of your build system and include them in your readme then as part of an automated process. We mentioned that these are ultimately a tool for communication between members of a team and people in the company. So what do we do about sharing them and keeping them up to date and all that kind of stuff?
Luciano: So we already said that programmatic tools are better because you can keep them in a repository. So when you are sharing the code, if you have these diagrams in line with the code, they are immediately available to at least people that are trying to understand the code or contribute to the code. So that's definitely one way you can use programmatic tools. Manual tools are I think that they have kind of a lower barrier to entry because it's easier to just you open this kind of white canvas and it's easier to understand how to drag and drop things and connect them. But then you have the problem that how do you share them? Sometimes you need to give people access to that particular tool. For instance, we mentioned already Lucidchart, which has that particular problem. And the tool kind of becomes effectively a repository for you to host all your diagrams, which might get very messy.
And if you are a company that deals with different projects and you want to share the diagrams with your customers, how do you do that? So sometimes you just end up exporting pictures and putting these pictures somewhere. And then there is the problem of how do you keep things in sync? Every time you change the diagram, you need to re-export the picture and make sure to copy it in the right place, which I've seen is kind of a common issue and you generally end up with pictures that are out of sync with the actual architecture diagram. So I don't necessarily have a solution for that problem except trying to be diligent with this process, but just be aware that when you use programmatic tools, you can kind of create pipelines that render the pictures and put them in the right places.
When you're doing things manually using a manual tool, it's much harder to achieve that. So you have to be more diligent and trying to propagate the rendering of that diagram correctly in the right place. And then the other point is where is the place that people are expected to go in that particular organization to consume these architecture diagrams? Sometimes certain companies rely a lot on wikis, something like Confluence and there are specific sections for architecture, so they do expect to go there and be able to find some kind of visualization of this architecture. So in that case, you don't really have a lot of options. You still need to export pictures and embed them in the wiki system. I don't know if certain wiki systems maybe have plugins that can integrate with specific tools like Lucidchart or Visio or other things.
Maybe there could be an option to try to keep things a little bit more in sync, but generally it gets more complicated to work with these kind of integrations. Another thing that I really like is to have diagrams in the readme. So again, if you work in a repository, don't just put the code there, but try to come up with a rendered version so that you can just click and see it straight away without having to clone the code and run some kind of script in your own machine to actually see the effect of that particular code-generated diagram. And for that, I really like the tool that we mentioned before, kroki.io, because you don't necessarily even have to create a pipeline, it's just literally the URL itself will become the picture that you can easily embed in your markdown. So I think this is all we have for today. I hope that you found some useful information in this chat, but also I'm really curious to hear if we did miss anything. If you have other tools that we were not aware about, we'd love to learn from you, we'd love to know if you have any best practice that you would recommend, so definitely reach out to us on Twitter or leave us a comment and we'll make sure to get back to you. So thank you very much and we'll see you in the next episode.