Modifying a Python dict while iterating over it

Posted on

Question :

Modifying a Python dict while iterating over it

Let’s say we have a Python dictionary d, and we’re iterating over it like so:

for k,v in d.iteritems():
    del d[f(k)] # remove some item
    d[g(k)] = v # add a new item

(f and g are just some black-box transformations.)

In other words, we try to add/remove items to d while iterating over it using iteritems.

Is this well defined? Could you provide some references to support your answer?

(It’s pretty obvious how to fix this if it’s broken, so this isn’t the angle I am after.)

Asked By: NPE

||

Answer #1:

It is explicitly mentioned on the Python doc page (for Python 2.7) that

Using iteritems() while adding or deleting entries in the dictionary may raise a RuntimeError or fail to iterate over all entries.

Similarly for Python 3.

The same holds for iter(d), d.iterkeys() and d.itervalues(), and I’ll go as far as saying that it does for for k, v in d.items(): (I can’t remember exactly what for does, but I would not be surprised if the implementation called iter(d)).

Answer #2:

Alex Martelli weighs in on this here.

It may not be safe to change the container (e.g. dict) while looping over the container.
So del d[f(k)] may not be safe. As you know, the workaround is to use d.items() (to loop over an independent copy of the container) instead of d.iteritems() (which uses the same underlying container).

It is okay to modify the value at an existing index of the dict, but inserting values at new indices (e.g. d[g(k)]=v) may not work.

Answered By: unutbu

Answer #3:

You cannot do that, at least with d.iteritems(). I tried it, and Python fails with

RuntimeError: dictionary changed size during iteration

If you instead use d.items(), then it works.

In Python 3, d.items() is a view into the dictionary, like d.iteritems() in Python 2. To do this in Python 3, instead use d.copy().items(). This will similarly allow us to iterate over a copy of the dictionary in order to avoid modifying the data structure we are iterating over.

Answered By: murgatroid99

Answer #4:

I have a large dictionary containing Numpy arrays, so the dict.copy().keys() thing suggested by @murgatroid99 was not feasible (though it worked). Instead, I just converted the keys_view to a list and it worked fine (in Python 3.4):

for item in list(dict_d.keys()):
    temp = dict_d.pop(item)
    dict_d['some_key'] = 1  # Some value

I realize this doesn’t dive into the philosophical realm of Python’s inner workings like the answers above, but it does provide a practical solution to the stated problem.

Answered By: 2cynykyl

Answer #5:

The following code shows that this is not well defined:

def f(x):
    return x

def g(x):
    return x+1

def h(x):
    return x+10

try:
    d = {1:"a", 2:"b", 3:"c"}
    for k, v in d.iteritems():
        del d[f(k)]
        d[g(k)] = v+"x"
    print d
except Exception as e:
    print "Exception:", e

try:
    d = {1:"a", 2:"b", 3:"c"}
    for k, v in d.iteritems():
        del d[f(k)]
        d[h(k)] = v+"x"
    print d
except Exception as e:
    print "Exception:", e

The first example calls g(k), and throws an exception (dictionary changed size during iteration).

The second example calls h(k) and throws no exception, but outputs:

{21: 'axx', 22: 'bxx', 23: 'cxx'}

Which, looking at the code, seems wrong – I would have expected something like:

{11: 'ax', 12: 'bx', 13: 'cx'}
Answered By: combatdave

Answer #6:

I got the same problem and I used following procedure to solve this issue.

Python List can be iterate even if you modify during iterating over it.
so for following code it will print 1’s infinitely.

for i in list:
   list.append(1)
   print 1

So using list and dict collaboratively you can solve this problem.

d_list=[]
 d_dict = {} 
 for k in d_list:
    if d_dict[k] is not -1:
       d_dict[f(k)] = -1 # rather than deleting it mark it with -1 or other value to specify that it will be not considered further(deleted)
       d_dict[g(k)] = v # add a new item 
       d_list.append(g(k))
Answered By: Zeel Shah

Answer #7:

Python 3 you should just:

prefix = 'item_'
t = {'f1': 'ffw', 'f2': 'fca'}
t2 = dict() 
for k,v in t.items():
    t2[k] = prefix + v

or use:

t2 = t1.copy()

You should never modify original dictionary, it leads to confusion as well as potential bugs or RunTimeErrors. Unless you just append to the dictionary with new key names.

Answered By: Dexter

Answer #8:

Today I had a similar use-case, but instead of simply materializing the keys on the dictionary at the beginning of the loop, I wanted changes to the dict to affect the iteration of the dict, which was an ordered dict.

I ended up building the following routine, which can also be found in jaraco.itertools:

def _mutable_iter(dict):
    """
    Iterate over items in the dict, yielding the first one, but allowing
    it to be mutated during the process.
    >>> d = dict(a=1)
    >>> it = _mutable_iter(d)
    >>> next(it)
    ('a', 1)
    >>> d
    {}
    >>> d.update(b=2)
    >>> list(it)
    [('b', 2)]
    """
    while dict:
        prev_key = next(iter(dict))
        yield prev_key, dict.pop(prev_key)

The docstring illustrates the usage. This function could be used in place of d.iteritems() above to have the desired effect.

Answered By: Jason R. Coombs

Leave a Reply

Your email address will not be published. Required fields are marked *