Downloading files in linux with Curl and Wget – Some simple examples

How to download files with Curl

The simplest format for curl is:

curl -O [URL]

That’s a capital “O” (Oh)

Eg:

curl -O http://droidzone.in/roms/supernova.zip

 

will download the file supernova.zip from remote and store it in the same name in the current location.

 

Resuming a Broken download with curl:

curl -C - -O android-ndk-r8c-linux-x86.tar.bz2 http://dl.google.com/android/ndk/android-ndk-r8c-linux-x86.tar.bz2

will resume downloading from http://dl.google.com/android/ndk/android-ndk-r8c-linux-x86.tar.bz2 to the local file android-ndk-r8c-linux-x86.tar.bz2

Note that that is a capital C and a small O (Oh)

Now here’s an interesting observation. Wget and curl can resume download of a download started by the other too!

How to download files with Wget

The basic syntax for downloading a file with wget is:

wget [URL]

 

Eg:

wget http://droidzone.in/roms/supernova.zip

Resuming a Broken download with wget:

wget -c http://droidzone.in/roms/supernova.zip[/code]
will resume the download into the same filename received from server, i.e supernova.zip, and will not overwrite it.

Specify a destination directory with wget:

This is done with the -P option. Eg: Getting a specific patch from Github
roidzone@gooey:~/Note2Core-GT-N710x_Kernel$ wget https://github.com/glewarne/Note2Core-GT-N710x_Kernel/commit/2c305d77b1ebd8a45242312ae693764de11630a9.patch -P ../patches/
--2012-11-27 22:21:17--  https://github.com/glewarne/Note2Core-GT-N710x_Kernel/commit/2c305d77b1ebd8a45242312ae693764de11630a9.patch
Resolving github.com... 207.97.227.239
Connecting to github.com|207.97.227.239|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1177 (1.1K) 1
Saving to: `../patches/2c305d77b1ebd8a45242312ae693764de11630a9.patch'

100%[============================================================================================================================>] 1,177       --.-K/s   in 0s      

2012-11-27 22:21:18 (461 MB/s) - `../patches/2c305d77b1ebd8a45242312ae693764de11630a9.patch' saved [1177/1177]

Download using proper file name from server

Often, while using curl or wget to download files with urls like http://download.oracle.com/otn-pub/java/jdk/7u9-b05/jdk-7u9-linux-i586.tar.gz?AuthParam=1354451808_536b13c3c7c8eb652e674f6f7c23467b, you will find that it actually downloads file with a filename like 7u9-b05/jdk-7u9-linux-i586.tar.gz?AuthParam=1354451808_536b13c3c7c8eb652e674f6f7c23467b. Frankly it's a chore to rename it back to the correct filename jdk-7u9-linux-i586.tar.gz and wreaks havoc with any sort of automation you had in mind.

So it turns out that both wget and curl have experimental options to resolve the correct filename from server.

If we want to use curl to get a file with proper filename and also resume it if the file already exists, and get the proper filename from server we might do something like:

curl -C - -O -J http://download.oracle.com/otn-pub/java/jdk/7u9-b05/jdk-7u9-linux-i586.tar.gz?AuthParam=1354451808_536b13c3c7c8eb652e674f6f7c23467b

Here, -C is the flag to "Continue (resume) a file download if a partial file already exists"

-O specifies the output is to a file rather than stdout

-J makes sure that filename is resolved properly

 

The corresponding option for wget is to use the --content-disposition flag.

So our command would be:

wget --content-disposition -c http://download.oracle.com/otn-pub/java/jdk/7u9-b05/jdk-7u9-linux-i586.tar.gz?AuthParam=1354451808_536b13c3c7c8eb652e674f6f7c23467b

 

r


You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.