How do I download a file over HTTP using Python?

Posted on

Solving problem is about exposing yourself to as many situations as possible like How do I download a file over HTTP using Python? and practice these strategies over and over. With time, it becomes second nature and a natural way you approach any problems in general. Big or small, always start with a plan, use other strategies mentioned here till you are confident and ready to code the solution.
In this post, my aim is to share an overview the topic about How do I download a file over HTTP using Python?, which can be followed any time. Take easy to follow this discuss.

How do I download a file over HTTP using Python?

I have a small utility that I use to download an MP3 file from a website on a schedule and then builds/updates a podcast XML file which I’ve added to iTunes.

The text processing that creates/updates the XML file is written in Python. However, I use wget inside a Windows .bat file to download the actual MP3 file. I would prefer to have the entire utility written in Python.

I struggled to find a way to actually download the file in Python, thus why I resorted to using wget.

So, how do I download the file using Python?

Asked By: Owen


Answer #1:

Use urllib.request.urlopen():

import urllib.request
with urllib.request.urlopen('') as f:
    html ='utf-8')

This is the most basic way to use the library, minus any error handling. You can also do more complex stuff such as changing headers.

On Python 2, the method is in urllib2:

import urllib2
response = urllib2.urlopen('')
html =
Answered By: Corey

Answer #2:

One more, using urlretrieve:

import urllib
urllib.urlretrieve ("", "mp3.mp3")

(for Python 3+ use import urllib.request and urllib.request.urlretrieve)

Yet another one, with a “progressbar”

import urllib2
url = ""
file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
meta =
file_size = int(meta.getheaders("Content-Length")[0])
print "Downloading: %s Bytes: %s" % (file_name, file_size)
file_size_dl = 0
block_sz = 8192
while True:
    buffer =
    if not buffer:
    file_size_dl += len(buffer)
    status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
    status = status + chr(8)*(len(status)+1)
    print status,
Answered By: PabloG

Answer #3:

In 2012, use the python requests library

>>> import requests
>>> url = ""
>>> r = requests.get(url)
>>> print len(r.content)

You can run pip install requests to get it.

Requests has many advantages over the alternatives because the API is much simpler. This is especially true if you have to do authentication. urllib and urllib2 are pretty unintuitive and painful in this case.


People have expressed admiration for the progress bar. It’s cool, sure. There are several off-the-shelf solutions now, including tqdm:

from tqdm import tqdm
import requests
url = ""
response = requests.get(url, stream=True)
with open("10MB", "wb") as handle:
    for data in tqdm(response.iter_content()):

This is essentially the implementation @kvance described 30 months ago.

Answered By: hughdbrown

Answer #4:

import urllib2
mp3file = urllib2.urlopen("")
with open('test.mp3','wb') as output:

The wb in open('test.mp3','wb') opens a file (and erases any existing file) in binary mode so you can save data with it instead of just text.

Answered By: Grant

Answer #5:

Python 3

  • urllib.request.urlopen

    import urllib.request
    response = urllib.request.urlopen('')
    html =
  • urllib.request.urlretrieve

    import urllib.request
    urllib.request.urlretrieve('', 'mp3.mp3')

    Note: According to the documentation, urllib.request.urlretrieve is a “legacy interface” and “might become deprecated in the future” (thanks gerrit)

Python 2

  • urllib2.urlopen (thanks Corey)

    import urllib2
    response = urllib2.urlopen('')
    html =
  • urllib.urlretrieve (thanks PabloG)

    import urllib
    urllib.urlretrieve('', 'mp3.mp3')
Answered By: bmaupin

Answer #6:

use wget module:

import wget'url')
Answered By: Sara Santana

Answer #7:

import os,requests
def download(url):
    get_response = requests.get(url,stream=True)
    file_name  = url.split("/")[-1]
    with open(file_name, 'wb') as f:
        for chunk in get_response.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
Answered By: H S Umer farooq

Answer #8:

An improved version of the PabloG code for Python 2/3:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import ( division, absolute_import, print_function, unicode_literals )
import sys, os, tempfile, logging
if sys.version_info >= (3,):
    import urllib.request as urllib2
    import urllib.parse as urlparse
    import urllib2
    import urlparse
def download_file(url, dest=None):
    Download and save a file specified by url to dest directory,
    u = urllib2.urlopen(url)
    scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
    filename = os.path.basename(path)
    if not filename:
        filename = 'downloaded.file'
    if dest:
        filename = os.path.join(dest, filename)
    with open(filename, 'wb') as f:
        meta =
        meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all
        meta_length = meta_func("Content-Length")
        file_size = None
        if meta_length:
            file_size = int(meta_length[0])
        print("Downloading: {0} Bytes: {1}".format(url, file_size))
        file_size_dl = 0
        block_sz = 8192
        while True:
            buffer =
            if not buffer:
            file_size_dl += len(buffer)
            status = "{0:16}".format(file_size_dl)
            if file_size:
                status += "   [{0:6.2f}%]".format(file_size_dl * 100 / file_size)
            status += chr(13)
            print(status, end="")
    return filename
if __name__ == "__main__":  # Only run if this file is called directly
    print("Testing with 10MB download")
    url = ""
    filename = download_file(url)
Answered By: Stan

Leave a Reply

Your email address will not be published. Required fields are marked *