Help us to make this transcription better! If you find an error, please
submit a PR with your corrections.
Luciano: Imagine this scenario. You were live streaming your AWS application build on Twitch, and because you are amazing and people love you, you have thousands of people that are watching you, and everything is going great. Then for a split second, you accidentally show your audience your environment variables, and that includes amazing things like your temporary AWS role credentials, like access key, AD, secret access key, and session token, which is basically everything, right?
And you start to be worried, your heart is racing, and you are sweating, and you are wondering, okay, what do I do now? Because in a matter of minutes or seconds, people are going to actually try to use these credentials to do something nasty with my AWS account. So what do you do now? Fear not, because today we're going to give you a super quick tip on how to revoke AWS credentials. And by the way, if you're thinking that you want to get the AWS Solution and you want to get an architect certification exam, this is one of the questions that might come up.
So be ready to take notes. My name is Luciano, and I'm here with Eoin for another AWS Bites podcast episode. AWS Bites is brought to you by fourTheorem, an AWS consulting partner that doesn't suck, or rarely suck at least. So if you want to check us out, go to fourTheorem.com, and also feel free to reach out to us. We have all the links in the show notes. Okay, let's talk this through. Maybe you have been following our 100-plus episodes, so you know that you shouldn't be using hardcoded credentials.
You should be using temporary credentials. Still, you can make a simple mistake, and you can still accidentally commit them to Git, or maybe if you're doing a screen share during a video call, you can still leak them. So what is the problem here, right? Is that a big deal? Because temporary credentials, as the name suggests, is something that's going to be temporary, but how long are they going to last?
They might last hours, right? So is that going to be a problem? Well, probably yes, right? Because if you have hours, there is still a lot of damage that you can do in a few hours, especially if that role is something that has very broad credentials. Maybe, I don't know, it's an administrative credential, probably is the worst use case, but you might have a role, for instance, that you use for deployment reasons, so you might be able to provision new Lambdas or EC2s or create VPCs, all things that can lead to actually giving pretty broad permissions to an attacker that might be able to get those credentials.
So what do you do in this case? Because the first worry that you have is that you might have a massive AWS bill if somebody tries to use these credentials, and they can do other kinds of damage that might affect the reputation of a company or might destroy some of your environments that you use for work. So yes, what do we do here now, right? And the first thing that you might realize is that this kind of credentials cannot be invalidated. You don't really have an API or a panel where you can say, well, invalidate this particular set of credentials, as you can do with the un-coded ones. So the only thing it seems that we have at end is the expiration time, but we don't want to trust that because it's still pretty open. So what do we do now, Eoin? Any idea?
Eoin: Yeah, like you say, you can't invalidate them, they exist until they expire. So you need to think about what permissions really these credentials are granting and how can you change the permissions level. And still, the valid credentials will still be usable, but maybe just prevent what they can do with those credentials in the time until the expiration occurs. So the only thing you can do really is change the role and the policies within it.
I suppose you could delete the role, that would be a brute force way of dealing with the problem. Or you can also update the role and maybe just remove all of its permission, so it doesn't have any permissions then. This would have the immediate effect of blocking out anyone with the temporary credentials. Actually, I say immediate effect, you should also be aware that IAM is eventually consistent.
So it might take seconds or even minutes sometimes for the effect of your changes to occur. That's something you don't really control, because IAM has to be reflected out in all of the regions and all of the nodes it's distributed across all of AWS's massive infrastructure. And that doesn't happen in the blink of an eye. So by doing that, you've blocked out your attacker, but it also has side effects, you might not want any valid holders of credentials are also blocked out, including anyone who just gets new credentials for the role after you've made that change.
So it's not ideal. You could also attach a policy with an explicit deny. So you could say, deny everything on every resource, and just attach that to the policy. And we know that an explicit deny always wins in these policies, it will trump everything. And it's better because you don't have to remove all the other permissions, you don't have to delete the role, but it still has that problem that it blocks out valid identities too.
So the solution to this really is to attach a policy with a conditional deny. And there's a special clever condition that you can use that denies access to identities that assume the role before a specific timestamp. And this is basically a way of saying, deny everybody who assumed this role before the current timestamp right now. And there's a condition key called date less than the condition predicate.
And then the attribute you're putting into that predicate is the AWS token issue time. So if the token issue time is less than the current timestamp, then deny them everything. What that means is that valid and invalid, like spurious attackers have no permissions for all sessions until the current time. But it does allow valid users to get a new session. And the new session will be valid and will continue to work. And business will operate as normal. Or if it's a service linked, a service role, then the service can get a new role as well. And the session will be valid. But anyone with the leaked credentials is locked out. So that's your problem solved. Yeah, all of this, the way we've described it, it sounds a bit complicated. And if you're in a situation where you're panicked, it's difficult to remember it all and do it correctly. So how do we simplify this process so we don't have to remember everything in that real situation?
Luciano: So you described already that the policy very well. Anyway, we're going to make that policy available in a gist. And you will find the link in the show notes just if you want to have a more easy to follow reference. And also if you need to copy paste the structure of it. But of course, if you need to remember it from scratch, from the top of your mind, and you're panicking, it might not be the easiest thing to do ever.
But luckily, there is a feature which is built in the AWS console. And basically, if you go to your role in IAM, in the page for that specific role, you can select the there is an item called revoke session. And that revoke session will give you something like revoke active sessions. And this will actually do something very similar to what we described, because it's going to create an inline policy called AWS revoke all the sessions, and it's going to inject the current time into that condition.
And we also going to have a link in the show notes that basically document this particular feature. But it pretty much works in a similar way to what we described. So while this is something that is good to have available in the AWS console, is it's good to know how it works under the wood. So it's something that if you ever want to automate with your own script, or maybe you want to add, I don't know, something that you can easily do from a CLI or a lambda, because it's maybe easier for you to trigger it that way, you can recreate that feature by yourself. Now, that was the answer for temporary credentials. What about if you have long lived credentials? What do we do? Do we try not to use them? But we know that the reality they are often something that exists in companies and deployments. So what can we do in those cases? The fact that it's harder to do this with long lived credentials is one of the reasons why you should avoid them all together.
Eoin: Because your only option is to just deactivate that access key and create a new one. So it's like a revocation, but if you have long lived credentials, by their nature, that probably means that they're hard coded in multiple places, then you have to go updating those places. And that might mean you're updating an application, there might be some downtime involved or a period where the application isn't working fully.
So I would say avoid them if possible. But basically, your only option is to delete the access key, and recreate a new one and update any references to it. That would go for IAM user access keys, but also dreaded root user access keys as well, which hopefully you shouldn't have. There are some organizations that are still using HashiCorp Vault to manage AWS secrets. And I suppose it's worth mentioning because it sometimes uses long lived secrets under the hood, but they're not really long lived in the same sense, because Vault is managing the lifecycle of those secrets for you, and is able to rotate them and revoke them automatically. So it's not much of an issue there, you're just relying on Vault to deliver that feature for you and manage your security. So if you're worried about us complaining about long lived credentials, but you're using Vault, don't worry, as long as you're following the best practices. I don't think there's much of an issue there.
Luciano: This brings us to the end of this short episode, I hope that now you have a little bit more peace of mind that even if you accidentally leak your credentials, at least you know what to do and where you can find some kind of solutions to remediate quickly for that particular issue. It's important, of course, to think about if those credentials are part of an application, if you can guarantee zero downtime to that application while you are revoking the credentials.
It's, of course, something you should be doing, but it's not always possible if you're using long lived credentials. So just keep that in mind. And maybe one more reason to think, am I using long lived credentials? Do I really need to use them? Maybe I can transition to temporary credentials. That's probably going to be a good thing to do anyway. So if you have other tips about how to rotate credentials, or if you have any horror story that happened to you, maybe you leaked credentials and something funny or not so funny happened to you after that, it might be interesting to know. It might be a nice story, might teach us something that we didn't know about. So please share all of that stuff either in the comments or if you don't feel like particularly happy to share this stuff publicly, you can always reach out in private and we can have a nice conversation about that and maybe learn from each other. So that's all we have for today. Thank you very much for following along. If you found value, always remember to share or like. This stuff always helps a little bit. So thank you for that and we'll see you in the next episode.