From the course: Linux Tips

Remote files: curl and wget - Linux Tutorial

From the course: Linux Tips

Start my 1-month free trial Buy for my team

Remote files: curl and wget

“

- [Instructor] Curl and wget are two tools that we can use to download files at the command line. Curl is a very flexible tool with many options. And it can both send and receive information. In this video though, we'll focus on retrieving information. wget on the other hand is a little bit more specialized and it's designed to tolerate unstable network connections for downloading. It'll retry if it encounters an error. First let's take a look at using curl to download something. I need something to download, so I'll visit the Kernel Archives at kernel.org. And I'll right-click on this button and choose copy link location. I'm using this link to the kernel gzip because it'll always be accessible and it'll take a while to download, so we can see how the tool shows up status updates. Before I use curl, I need to make sure that it's installed. So I'll write sudo apt install curl. Alright. Now I can write curl and paste the URL with Ctrl+Shift+V. And then I'll press enter and well, what the heck's going on? By default, curl sends the contents of the file to the standard output. And if we're trying to download a file, that's not what we want. I've pressed Ctrl+C to cancel this and I'll clear the screen with clear. Sometimes we do want to be able to have a file be piped into another command, but let's download this to a file instead. For that, we can use the dash-lowercase O option to set the file name. So again, I'll write curl and paste the URL but this time I'll add -o and I'll call this kernel.tar.xz. When I press enter to run that, you'll see curl start to download the file and give some statistics about the operation. Depending on the speed of your connection, this could take a minute or so. In the interface, we see the percent completed and the total size and the amount of data that's been received. Over here, there's an average speed and there's a listing of the time spent transferring data as well. After that, there's an estimate of how much longer the transfer will take, and a snapshot of the current transfer rate. Curl offers another option in naming the downloaded file as well. Instead of using a lowercase o in a file name, I can use capital O and then curl will use the name of the file in the URL as the final name of the file. To use that, I'll write curl and once again I'll paste the URL and then use -O. Now if I take a look at this folder, and look at the .xz files, I can see the first one that I downloaded had the name kernel.tar.xz but the second one, where I didn't specify a name, took its file name from the URL. Curl is an extremely versatile tool and it can send and receive data with many different protocols. Check out the command page for a very thorough listing of what it can do. wget is also useful to know about. Like curl, it's often used for downloading files from URLs and scripts and so on. Wget can operate well in the background and it's tolerant of intermittent problems when trying to download a file. Like curl, it has many useful features, and the command page details them all. Let's take a look at basic usage of wget with the kernel file from earlier. I'll write wget and I'll paste in the URL. Here I can see the activity that wget is taking. It's resolving the host, connecting, sending the request, getting the response and then I can see the progress as the file is downloaded, along with the file size received so far and the speed with which the file is being sent. And when it's done, I get a little summary here. If for some reason the transfer was interrupted, wget would try to retrieve the download a few times before giving up. In the event that the download fails partway through, wget can continue from where it left off, once the download source is available again, using the -c command. Let's take a look at that. First, I'll make sure that I don't have any copies of this kernel file downloaded. I'll see what I have with ls -lh *.xz*. Here's the first two that I downloaded and the version that I downloaded with wget, because it has the same name as an existing file and a .1 appended to the end. So I'll remove those with rm *.xz*. Now I'll recall my wget command. And I'll press enter, but this time I'll interrupt it partway through the download with Ctrl+C, leaving a broken partial file. Let's take a look at what we have with ls -lh *.xz and here's a file with 35 MB of information. I can use the same command again to download it, but this time adding -c and notice the wget is now using a partial content response. And the portion that's remaining gets downloaded. That part's represented with + characters here and the rest of the progress is represented with = signs like normal. Both curl and wget are extremely common tools for working with URLs of the command line. So I recommend that you become familiar with at least the basics of how to use each of them.

Contents