Common use-cases for pickle in Python

Posted on

Question :

Common use-cases for pickle in Python

I’ve looked at the pickle documentation, but I don’t understand where pickle is useful.

What are some common use-cases for pickle?

Asked By: satoru


Answer #1:

Some uses that I have come across:

1) saving a program’s state data to disk so that it can carry on where it left off when restarted (persistence)

2) sending python data over a TCP connection in a multi-core or distributed system (marshalling)

3) storing python objects in a database

4) converting an arbitrary python object to a string so that it can be used as a dictionary key (e.g. for caching & memoization).

There are some issues with the last one – two identical objects can be pickled and result in different strings – or even the same object pickled twice can have different representations. This is because the pickle can include reference count information.

To emphasise @lunaryorn’s comment – you should never unpickle a string from an untrusted source, since a carefully crafted pickle could execute arbitrary code on your system. For example see

Answered By: Dave Kirby

Answer #2:

Minimal roundtrip example..

>>> import pickle
>>> a = Anon()
>>> = 'bar'
>>> pickled = pickle.dumps(a)
>>> unpickled = pickle.loads(pickled)

Edit: but as for the question of real-world examples of pickling, perhaps the most advanced use of pickling (you’d have to dig quite deep into the source) is ZODB:

Otherwise, PyPI mentions several:

I have personally seen several examples of pickled objects being sent over the network as an easy to use network transfer protocol.

Answered By: Jacob Oscarson

Answer #3:

Pickling is absolutely necessary for distributed and parallel computing.

Say you wanted to do a parallel map-reduce with multiprocessing (or across cluster nodes with pyina), then you need to make sure the function you want to have mapped across the parallel resources will pickle. If it doesn’t pickle, you can’t send it to the other resources on another process, computer, etc. Also see here for a good example.

To do this, I use dill, which can serialize almost anything in python. Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

And, yes, people use picking to save the state of a calculation, or your ipython session, or whatever.

Answered By: Mike McKerns

Answer #4:

I have used it in one of my projects. If the app was terminated during it’s working (it did a lengthy task and processed lots of data), I needed to save the whole data structure and reload it after the app was run again. I used cPickle for this, as speed was a crucial thing and the size of data was really big.

Answered By: taskinoor

Answer #5:

For the beginner (as is the case with me) it’s really hard to understand why use pickle in the first place when reading the official documentation. It’s maybe because the docs imply that you already know the whole purpose of serialization. Only after reading the general description of serialization have I understood the reason for this module and its common use cases. Also broad explanations of serialization disregarding a particular programming language may help:, What is serialization?,

Answered By: Bad

Answer #6:

Pickle is like “Save As..” and “Open..” for your data structures and classes. Let’s say I want to save my data structures so that it is persistent between program runs.


with open("save.p", "wb") as f:    
    pickle.dump(myStuff, f)        


    with open("save.p", "rb") as f:
        myStuff = pickle.load(f)
    myStuff = defaultdict(dict)

Now I don’t have to build myStuff from scratch all over again, and I can just pick(le) up from where I left off.

Answered By: spartan

Answer #7:

To add a real-world example: The Sphinx documentation tool for Python uses pickle to cache parsed documents and cross-references between documents, to speed up subsequent builds of the documentation.

Answered By: lunaryorn

Answer #8:

I can tell you the uses I use it for and have seen it used for:

  • Game profile saves
  • Game data saves like lives and health
  • Previous records of say numbers inputed to a program

Those are the ones I use it for at least

Answered By: Austin Gummy

Leave a Reply

Your email address will not be published.