Input and output numpy arrays to h5py

Posted on

Question :

Input and output numpy arrays to h5py

I have a Python code whose output is a enter image description here sized matrix, whose entries are all of the type float. If I save it with the extension .dat the file size is of the order of 500 MB. I read that using h5py reduces the file size considerably. So, let’s say I have the 2D numpy array named A. How do I save it to an h5py file?
Also, how do I read the same file and put it as a numpy array in a different code, as I need to do manipulations with the array?

Answer #1:

h5py provides a model of datasets and groups. The former is basically arrays and the latter you can think of as directories. Each is named. You should look at the documentation for the API and examples:

A simple example where you are creating all of the data upfront and just want to save it to an hdf5 file would look something like:

In [1]: import numpy as np
In [2]: import h5py
In [3]: a = np.random.random(size=(100,20))
In [4]: h5f = h5py.File('data.h5', 'w')
In [5]: h5f.create_dataset('dataset_1', data=a)
Out[5]: <HDF5 dataset "dataset_1": shape (100, 20), type "<f8">

In [6]: h5f.close()

You can then load that data back in using:

In [10]: h5f = h5py.File('data.h5','r')
In [11]: b = h5f['dataset_1'][:]
In [12]: h5f.close()

In [13]: np.allclose(a,b)
Out[13]: True

Definitely check out the docs:

Writing to hdf5 file depends either on h5py or pytables (each has a different python API that sits on top of the hdf5 file specification). You should also take a look at other simple binary formats provided by numpy natively such as, np.savez etc:

Answered By: JoshAdel

Answer #2:

A cleaner way to handle file open/close and avoid memory leaks:


import numpy as np
import h5py

data_to_write = np.random.random(size=(100,20)) # or some such


with h5py.File('name-of-file.h5', 'w') as hf:
    hf.create_dataset("name-of-dataset",  data=data_to_write)


with h5py.File('name-of-file.h5', 'r') as hf:
    data = hf['name-of-dataset'][:]
Answered By: Lavi Avigdor

Leave a Reply

Your email address will not be published. Required fields are marked *