I’ve been struggling with this simple problem for too long, so I thought I’d ask for help. I am trying to read a list of journal articles from National Library of Medicine ftp site into Python 3.3.2 (on Windows 7). The journal articles are in a .csv file.
I have tried the following code:
import csv import urllib.request url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv" ftpstream = urllib.request.urlopen(url) csvfile = csv.reader(ftpstream) data = [row for row in csvfile]
It results in the following error:
Traceback (most recent call last): File "<pyshell#4>", line 1, in <module> data = [row for row in csvfile] File "<pyshell#4>", line 1, in <listcomp> data = [row for row in csvfile] _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
I presume I should be working with strings not bytes? Any help with the simple problem, and an explanation as to what is going wrong would be greatly appreciated.
The problem relies on
urllib returning bytes. As a proof, you can try to download the csv file with your browser and opening it as a regular file and the problem is gone.
A similar problem was addressed here.
It can be solved decoding bytes to strings with the appropriate encoding. For example:
import csv import urllib.request url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv" ftpstream = urllib.request.urlopen(url) csvfile = csv.reader(ftpstream.read().decode('utf-8')) # with the appropriate encoding data = [row for row in csvfile]
The last line could also be:
data = list(csvfile) which can be easier to read.
By the way, since the csv file is very big, it can slow and memory-consuming. Maybe it would be preferable to use a generator.
Using codecs as proposed by Steven Rumbalski so it’s not necessary to read the whole file to decode. Memory consumption reduced and speed increased.
import csv import urllib.request import codecs url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv" ftpstream = urllib.request.urlopen(url) csvfile = csv.reader(codecs.iterdecode(ftpstream, 'utf-8')) for line in csvfile: print(line) # do something with line
Note that the list is not created either for the same reason.
Even though there is already an accepted answer, I thought I’d add to the body of knowledge by showing how I achieved something similar using the
requests package (which is sometimes seen as an alternative to
The basis of using
codecs.itercode() to solve the original problem is still the same as in the accepted answer.
import codecs from contextlib import closing import csv import requests url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv" with closing(requests.get(url, stream=True)) as r: reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8')) for row in reader: print row
Here we also see the use of streaming provided through the
requests package in order to avoid having to load the entire file over the network into memory first (which could take long if the file is large).
I thought it might be useful since it helped me, as I was using
requests rather than
urllib.request in Python 3.6.
Some of the ideas (e.g using
closing()) are picked from this similar post
I had a similar problem using
requests package and
The response from post request was type
In order to user
csv library, first I a stored them as a string file in memory (in my case the size was small), decoded utf-8.
import io import csv import requests response = requests.post(url, data) # response.content is something like: # b'"City","Awb","Total"rn"Bucuresti","6733338850003","32.57"rn' csv_bytes = response.content # write in-memory string file from bytes, decoded (utf-8) str_file = io.StringIO(csv_bytes.decode('utf-8'), newline='n') reader = csv.reader(str_file) for row_list in reader: print(row_list) # Once the file is closed, # any operation on the file (e.g. reading or writing) will raise a ValueError str_file.close()
Printed something like:
['City', 'Awb', 'Total'] ['Bucuresti', '6733338850003', '32.57']
urlopen will return a
urllib.response.addinfourl instance for an ftp request.
For ftp, file, and data urls and requests explicity handled by legacy
URLopener and FancyURLopener classes, this function returns a
urllib.response.addinfourl object which can work as context manager…
48868168L whose fp = <addclosehook at 48777416L whose fp = <socket._fileobject object at 0x0000000002E52B88>>>urllib2.urlopen(url) <addinfourl at
At this point
ftpstream is a file like object, using
.read() would return the contents however
csv.reader requires an iterable in this case:
Defining a generator like so:
def to_lines(f): line = f.readline() while line: yield line line = f.readline()
We can create our csv reader like so:
reader = csv.reader(to_lines(ftps))
And with a url
url = "http://pic.dhe.ibm.com/infocenter/tivihelp/v41r1/topic/com.ibm.ismsaas.doc/reference/CIsImportMinimumSample.csv"
for row in reader: print row
'simpleci'] ['SCI.APPSERVER'] ['SRM_SaaS_ES', 'MXCIImport', 'AddChange', 'EN'] ['CI_CINUM'] ['unique_identifier1'] ['unique_identifier2'][