Elegant way to remove items from sequence in Python? [duplicate]

Posted on

Question :

Elegant way to remove items from sequence in Python? [duplicate]

When I am writing code in Python, I often need to remove items from a list or other sequence type based on some criteria. I haven’t found a solution that is elegant and efficient, as removing items from a list you are currently iterating through is bad. For example, you can’t do this:

for name in names:
    if name[-5:] == 'Smith':

I usually end up doing something like this:

toremove = []
for name in names:
    if name[-5:] == 'Smith':
for name in toremove:
del toremove

This is innefficient, fairly ugly and possibly buggy (how does it handle multiple ‘John Smith’ entries?). Does anyone have a more elegant solution, or at least a more efficient one?

How about one that works with dictionaries?

Answer #1:

Two easy ways to accomplish just the filtering are:

  1. Using filter:

    names = filter(lambda name: name[-5:] != "Smith", names)

  2. Using list comprehensions:

    names = [name for name in names if name[-5:] != "Smith"]

Note that both cases keep the values for which the predicate function evaluates to True, so you have to reverse the logic (i.e. you say “keep the people who do not have the last name Smith” instead of “remove the people who have the last name Smith”).

Edit Funny… two people individually posted both of the answers I suggested as I was posting mine.

Answered By: John

Answer #2:

You can also iterate backwards over the list:

for name in reversed(names):
    if name[-5:] == 'Smith':

This has the advantage that it does not create a new list (like filter or a list comprehension) and uses an iterator instead of a list copy (like [:]).

Note that although removing elements while iterating backwards is safe, inserting them is somewhat trickier.

Answer #3:

The obvious answer is the one that John and a couple other people gave, namely:

>>> names = [name for name in names if name[-5:] != "Smith"]       # <-- slower

But that has the disadvantage that it creates a new list object, rather than reusing the original object. I did some profiling and experimentation, and the most efficient method I came up with is:

>>> names[:] = (name for name in names if name[-5:] != "Smith")    # <-- faster

Assigning to “names[:]” basically means “replace the contents of the names list with the following value”. It’s different from just assigning to names, in that it doesn’t create a new list object. The right hand side of the assignment is a generator expression (note the use of parentheses rather than square brackets). This will cause Python to iterate across the list.

Some quick profiling suggests that this is about 30% faster than the list comprehension approach, and about 40% faster than the filter approach.

Caveat: while this solution is faster than the obvious solution, it is more obscure, and relies on more advanced Python techniques. If you do use it, I recommend accompanying it with a comment. It’s probably only worth using in cases where you really care about the performance of this particular operation (which is pretty fast no matter what). (In the case where I used this, I was doing A* beam search, and used this to remove search points from the search beam.)

Answered By: Edward Loper

Answer #4:

Using a list comprehension

list = [x for x in list if x[-5:] != "smith"]
Answered By: Corey

Answer #5:

There are times when filtering (either using filter or a list comprehension) doesn’t work. This happens when some other object is holding a reference to the list you’re modifying and you need to modify the list in place.

for name in names[:]:
    if name[-5:] == 'Smith':

The only difference from the original code is the use of names[:] instead of names in the for loop. That way the code iterates over a (shallow) copy of the list and the removals work as expected. Since the list copying is shallow, it’s fairly quick.

Answered By: elifiner

Answer #6:

filter would be awesome for this. Simple example:

names = ['mike', 'dave', 'jim']
filter(lambda x: x != 'mike', names)
['dave', 'jim']

Edit: Corey’s list comprehension is awesome too.

Answered By: mk.

Answer #7:

names = filter(lambda x: x[-5:] != "Smith", names);
Answered By: pottedmeat

Answer #8:

Both solutions, filter and comprehension requires building a new list. I don’t know enough of the Python internals to be sure, but I think that a more traditional (but less elegant) approach could be more efficient:

names = ['Jones', 'Vai', 'Smith', 'Perez']

item = 0
while item <> len(names):
    name = names [item]
    if name=='Smith':
        item += 1

print names

Anyway, for short lists, I stick with either of the two solutions proposed earlier.

Answered By: PabloG

Leave a Reply

Your email address will not be published. Required fields are marked *