Fastest way to convert an iterator to a list

Posted on

Question :

Fastest way to convert an iterator to a list

Having an iterator object, is there something faster, better or more correct than a list comprehension to get a list of the objects returned by the iterator?

user_list = [user for user in user_iterator]

Answer #1:

Answered By: mikerobi

Answer #2:

since python 3.5 you can use * iterable unpacking operator:

user_list = [*your_iterator]

but the pythonic way to do it is:

user_list  = list(your_iterator)
Answered By: kederrac

Answer #3:

@Robino was suggesting to add some tests which make sense, so here is a simple benchmark between 3 possible ways (maybe the most used ones) to convert an iterator to a list:

  1. by type constructor


  1. by unpacking


  1. using list comprehension

[e for e in my_iterator]

I have been using simple_bechmark library

from simple_benchmark import BenchmarkBuilder
from heapq import nsmallest

b = BenchmarkBuilder()

def convert_by_type_constructor(size):

def convert_by_list_comprehension(size):
    [e for e in iter(range(size))]

def convert_by_unpacking(size):

@b.add_arguments('Convert an iterator to a list')
def argument_provider():
    for exp in range(2, 22):
        size = 2**exp
        yield size, size

r =

enter image description here

As you can see there is very hard to make a difference between conversion by the constructor and conversion by unpacking, conversion by list comprehension is the “slowest” approach.

I have been testing also across different Python versions (3.6, 3.7, 3.8, 3.9) by using the following simple script:

import argparse
import timeit

parser = argparse.ArgumentParser(
    description='Test convert iterator to list')
    '--size', help='The number of elements from iterator')

args = parser.parse_args()

size = int(args.size)
repeat_number = 10000

# do not wait too much if the size is too big
if size > 10000:
    repeat_number = 100

def test_convert_by_type_constructor():

def test_convert_by_list_comprehension():
    [e for e in iter(range(size))]

def test_convert_by_unpacking():

def get_avg_time_in_ms(func):
    avg_time = timeit.timeit(func, number=repeat_number) * 1000 / repeat_number
    return round(avg_time, 6)

funcs = [test_convert_by_type_constructor,
         test_convert_by_unpacking, test_convert_by_list_comprehension]

print(*map(get_avg_time_in_ms, funcs))

The script will be executed via a subprocess from a Jupyter Notebook (or a script), the size parameter will be passed through command-line arguments and the script results will be taken from standard output.

from subprocess import PIPE, run

import pandas

simple_data = {'constructor': [], 'unpacking': [], 'comprehension': [],
        'size': [], 'python version': []}

size_test = 100, 1000, 10_000, 100_000, 1_000_000
for version in ['3.6', '3.7', '3.8', '3.9']:
    print('test for python', version)
    for size in size_test:
        command = [f'python{version}', '', f'--size={size}']
        result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
        constructor, unpacking,  comprehension = result.stdout.split()
        simple_data['python version'].append(version)

df_ = pandas.DataFrame(simple_data)

enter image description here

You can get my full notebook from here.

In most of the cases, in my tests, unpacking shows to be faster, but the difference is so small that the results may change from a run to the other. Again, the comprehension approach is the slowest, in fact, the other 2 methods are up to ~ 60% faster.

Answered By: kederrac

Leave a Reply

Your email address will not be published. Required fields are marked *