I am trying to set members of an array that are below a threshold to nan. This is part of a QA/QC process and the incoming data may already have slots that are nan.
So as an example my threshold might be -1000 and hence I would want to set -3000 to nan in the following array
x = np.array([np.nan,1.,2.,-3000.,np.nan,5.])
x[x < -1000.] = np.nan
produces the correct behavior, but also a RuntimeWarning, but the overhead of disabling the warning
warnings.filterwarnings("ignore") ... warnints.resetwarnings()
is kind of heavy an potentially a bit unsafe.
Trying to index twice with fancy indexing as follows doesn’t produce any effect:
nonan = np.where(~np.isnan(x)) x[nonan][x[nonan] < -1000.] = np.nan
I assume this is because a copy is made due to the integer index or the use of indexing twice.
Does anyone have a relatively simple solution? It would be fine to use a masked array in the process, but the final product has to be an ndarray and I can’t introduce new dependencies. Thanks.
Any comparison (other than
!=) of a NaN to a non-NaN value will always return False:
1000 array([False, False, False, True, False, False], dtype=bool)x < -
So you can simply ignore the fact that there are NaNs already in your array and do:
1000] = np.nan x array([ nan, 1., 2., nan, nan, 5.])x[x < -
EDIT I don’t see any warning when I ran the above, but if you really need to stay away from the NaNs, you can do something like:
mask = ~np.isnan(x) mask[mask] &= x[mask] < -1000 x[mask] = np.nan
One option is to disable the relevant warnings with
with numpy.errstate(invalid='ignore'): ...
To turn off the relevant warnings globally, use
np.less() has a
where argument that controls where the operation will be applied. So you could do:
x[np.less(x, -1000., where=~np.isnan(x))] = np.nan
I personally ignore the warnings using the np.errstate context manager in the answer already given, as the code clarity is worth the extra time, but here is an alternative.
# given x = np.array([np.nan, 1., 2., -3000., np.nan, 5.]) # apply NaNs as desired mask = np.zeros(x.shape, dtype=bool) np.less(x, -1000, out=mask, where=~np.isnan(x)) x[mask] = np.nan # expected output and comparison y = np.array([np.nan, 1., 2., np.nan, np.nan, 5.]) assert np.allclose(x, y, rtol=0., atol=1e-14, equal_nan=True)
less ufunc takes the optional argument
where, and only evaluates it where true, unlike the
np.where function which evaluates both options and then picks the relevant one. You then set the desired output when it’s not true by using the
A little bit late, but this is how I would do:
x = np.array([np.nan,1.,2.,-3000.,np.nan,5.]) igood=np.where(~np.isnan(x)) x[igood[x[igood]<-1000.]]=np.nan