47. Preview: How to build a File Transfer application - live!
In this special episode, we announce our next initiative: starting some YouTube coding live streams where we build something on AWS. Specifically, we are going to build a file transfer service, just like WeTransfer or Dropbox Transfer! In this episode, we announce why we decided to start this, the logistics, and some of the details of the product we are going to build. We are really looking forward to pairing with you all live on this build! Make sure you are subscribed to our YouTube channel so you are notified when we go live on Wednesday 17 August!
In this episode, we mentioned the following resources:
Hello everyone, today we want to give you a heads up for a new kind of episode that we are working on. So instead of giving you a pre-recorded episode, we are taking a short break for the summer, but then we will be back with actually coding live streams. Our plan is to go live on Wednesday, 17th of August, and we're gonna be live on YouTube at 5pm Ireland or UK time, which should be 6pm standard time, 12 Eastern time, and 9am Pacific time. Hopefully all my time zone maths are correct. Our point of reference is definitely 5pm Ireland, UK, if you want to double check all these times. So basically in this live episode, what we want to do is we are going to start to build an application, and this of course is an application running on AWS, and over the course of the series we actually want to build a file transfer service, something that looks a little bit like WeTransfer or DropboxTransfer if you ever use those kind of services. So I suppose the main question is why are we doing this?
Right, well we get lots of feedback from viewers and listeners and people who just really like the real world use cases. So the one you mentioned about migrating an application to AWS was really popular. We've also covered, like we spent 50 episodes covering all the different AWS services and how they work. It's probably time to start putting these into practice and showing people really how it works when you've got hands on keyboard. And I suppose it's also good just to build the services you use, like file transfer services, because you understand exactly what the idea is, and you can learn a lot like trying to understand how you might build that from scratch. So what would it look like?
Sure, so just to put things into context, we are not going to try to build like a fully fledged Dropbox with file syncing and stuff like that. We're going to keep it a little bit more minimalistic, but still we want to build something that is going to be useful for people and you can use it in different contexts. So the idea is that it's kind of a very simple file sharing product. So you just want to upload one file and then have a URL that you can share with other people for them to download that particular file. And if you are thinking of different kind of examples or use cases, for instance, when we record these episodes, we always need to share video file and audio files between ourselves before we do the final production. So imagine that we will be using a service like this to just upload this video and audio file somewhere, and then we have an easy link that we can easily share between ourselves. Similarly, you might be thinking, okay, I need to send a bunch of receipts to my accountant. I have this big zip file, but it's too big to send it in an email. What do I do? My accountant probably doesn't have Dropbox, so let's send them a URL. Similarly, and this is actually a very common use case I had many, many times, and you probably can relate to that. You have multiple devices and you just want to transfer a big file between them. Every time it's like, how do I do that? If you have a service like this, you can easily just upload on one side and download from the URL on the other side. And at this point, why not to use Dropbox? Well, there might be many reasons. Maybe you just don't want to pay for Dropbox, or maybe you want to be in control of your own data and prefer to avoid third-party services and have something where you really understand what's going on, how the data is actually being processed and used. Or maybe this is actually a realistic use case I've seen in many offices that the network is actually blocking services like WeTransfer or Dropbox. In that case, you literally cannot use those services. One final reason that we can find is maybe you just want to use your own custom URL either for vanity or because you're actually building a business and it might make sense to have a little bit more trust by giving URLs that represent your company or your product. So I suppose, yeah, just to finish off, we are building this product mostly for ourselves to have fun, but in reality, a feature like this, it's something that could be integrated in every other product. Imagine you are building, I don't know, a chat application and you need people to be able to exchange files in a chat session, or you have just a file management system integrated in a CMS or other kinds of products. So it is actually a very, very common use case to having to deal with uploads and downloads in all sorts of SaaS products. So I think by doing this exercise, we're going to, well, first of all, learn ourself how to do this, but hopefully also teach and share with other people this learning so that you can go off and build the same functionality in other applications. So do you want to describe, Eoin, what's the plan?
The starting point, I think what we can do as the MVP for our file sharing would be, imagine that you call it an API that we deploy onto AWS and it gives you an upload URL and a download URL. So you upload the file with your upload URL, and then you share the download URL with your friend or with your other device. And that's enough, right? For our MVP, that should work. It might be a little bit rough around the edges in terms of user experience, but fundamentally it achieves the goal. So for that, we could use things like S3, of course, maybe signed URLs integrated with API gateway, a bit of Lambda, a sprinkling of IAM and put it all together with Node.js Lambda functions, serverless framework. And since it's an MVP, you can just use curl to do the uploading and downloading. But that's the first MVP. And I think that would be the core of it from that point. Okay, you mentioned vanity URLs. So maybe the next thing would be to register a domain name, integrate it so that we have much nicer links that are branded and we could use Route 53 and certificate manager to help us achieve that. Maybe we have to pick a name for this product. So maybe people can let us know over the next couple of weeks, what funky name we could use for our WeTransfer clone. We can get the domain registered in time for the first episode. Beyond that, I guess we could start building a bit of a front end, right? Maybe not a React front end. How about a command line interface, something like an NPM CLI? What do you think?
Yeah, I suppose that the simplest CLI I can think of is literally you just call the command name, let's call it file upload for now until we have an actual name. And you just say file upload this part. And then you just see the progress bar and eventually you see, okay, this is the URL for your file. This is probably the simplest thing I could think of in terms of CLI. And of course, we can make it nicer. We can have, I don't know, spinning progress bar and all that kind of stuff.
But the very minimum is that you just easily are able to upload a file by just giving a path to the CLI. And we'll probably gonna do all of that with Node.js again, because it's the easiest for us. So yeah, it's probably gonna be like a package that you can just NPM install global and then you can use it from any bash terminal. After that, I think that there is a lot of room for different kinds of improvements. We can probably also listen to feedback and try to understand what are things that people are mostly interested in. But the ideas that we came up with are, for instance, showing how we could build a deployment pipeline. We also talk a lot about using GitHub Actions and OIDC providers. So that can be another excuse to showcase how we could actually deploy this product from a GitHub Action using the OIDC integration that we have been talking about. We could also think about securing this entire API, because up to this point, we haven't mentioned any level of security. As long as you know the host, you can just upload and download. But we might want to limit that. Maybe we can... The easiest thing we could do is just use API keys in API Gateway, but we might end up exploring other options as well. Maybe Cognito or something a little bit more sophisticated. And then another topic that we were thinking about is probably you don't want these files to live in a mystery packet forever. Most likely, if you are using this kind of service, you are transferring big files, you probably don't load them only once and then not using them again. So it probably makes sense to make this file somehow disposable. Maybe make sure they get deleted automatically after a week, a day, or whatever is the deadline that makes sense for the kind of application. We might discuss a little bit about quota, limits, pricing, and things like that. So definitely let us know if you already have specific topics in mind that you would like us to actually explore in a kind of live coding fashion, because that is going to drive us in selecting what can we do next after this. Yep. That's it. That sounds good.
Maybe people want to see a bit more front end. It might be nice if you share the link that it gives people your download page and they have an option with what to do with the content. You can even preview the content in some way. There's a lot of different directions we could take this. And I'm really interested to hear what people want to see. So yeah, maybe in terms of feedback, since we're looking for it, one thing is for podcast listeners, we've got a question for you, because we know we have a lot of people who listen to AWS Bites in podcast audio only format. And this presents a bit of a challenge when you're doing a live coding demo, live build. And so I sometimes listen to YouTube videos of that kind of thing when I'm in the bus or driving. But does it make sense for you? Do you want us to publish these streams as audio only podcasts? Let us know. We're happy to do so if people want to hear it. If people think it's annoying and pointless, we're happy to skip them as well. We're probably going to do a summary episode at the end anyway, where we talk about all the things we learned. So just let us know what you think. We're happy to be guided by you. So we're really looking forward to pairing with you all live on this build. Make sure you're subscribed to the YouTube channel so you're going to be notified when we go live on Wednesday, the 17th of August. Until then, goodbye.
Use the arrow keys to move between transcript segments, Home or End to jump to the first or last segment, and Enter to seek in the video.
Help us to make this transcription better! If you find an error, please submit a PR with your corrections.