python numpy ValueError: operands could not be broadcast together with shapes

Posted on

Question :

python numpy ValueError: operands could not be broadcast together with shapes

In numpy, I have two “arrays”, X is (m,n) and y is a vector (n,1)

using

X*y

I am getting the error

ValueError: operands could not be broadcast together with shapes (97,2) (2,1) 

When (97,2)x(2,1) is clearly a legal matrix operation and should give me a (97,1) vector

EDIT:

I have corrected this using X.dot(y) but the original question still remains.

Asked By: yayu

||

Answer #1:

dot is matrix multiplication, but * does something else.

We have two arrays:

  • X, shape (97,2)
  • y, shape (2,1)

With Numpy arrays, the operation

X * y

is done element-wise, but one or both of the values can be expanded in one or more dimensions to make them compatible. This operation is called broadcasting. Dimensions, where size is 1 or which are missing, can be used in broadcasting.

In the example above the dimensions are incompatible, because:

97   2
 2   1

Here there are conflicting numbers in the first dimension (97 and 2). That is what the ValueError above is complaining about. The second dimension would be ok, as number 1 does not conflict with anything.

For more information on broadcasting rules: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html

(Please note that if X and y are of type numpy.matrix, then asterisk can be used as matrix multiplication. My recommendation is to keep away from numpy.matrix, it tends to complicate more than simplifying things.)

Your arrays should be fine with numpy.dot; if you get an error on numpy.dot, you must have some other bug. If the shapes are wrong for numpy.dot, you get a different exception:

ValueError: matrices are not aligned

If you still get this error, please post a minimal example of the problem. An example multiplication with arrays shaped like yours succeeds:

In [1]: import numpy

In [2]: numpy.dot(numpy.ones([97, 2]), numpy.ones([2, 1])).shape
Out[2]: (97, 1)
Answered By: DrV

Answer #2:

Per numpy docs:

When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when:

  • they are equal, or
  • one of them is 1

In other words, if you are trying to multiply two matrices (in the linear algebra sense) then you want X.dot(y) but if you are trying to broadcast scalars from matrix y onto X then you need to perform X * y.T.

Example:

>>> import numpy as np
>>>
>>> X = np.arange(8).reshape(4, 2)
>>> y = np.arange(2).reshape(1, 2)  # create a 1x2 matrix
>>> X * y
array([[0,1],
       [0,3],
       [0,5],
       [0,7]])
Answered By: gobrewers14

Answer #3:

You are looking for np.matmul(X, y). In Python 3.5+ you can use X @ y.

Answered By: iamanigeeit

Answer #4:

It’s possible that the error didn’t occur in the dot product, but after.
For example try this

a = np.random.randn(12,1)
b = np.random.randn(1,5)
c = np.random.randn(5,12)
d = np.dot(a,b) * c

np.dot(a,b) will be fine; however np.dot(a, b) * c is clearly wrong (12×1 X 1×5 = 12×5 which cannot element-wise multiply 5×12) but numpy will give you

ValueError: operands could not be broadcast together with shapes (12,1) (1,5)

The error is misleading; however there is an issue on that line.

Answered By: Kenan

Answer #5:

Use np.mat(x) * np.mat(y), that’ll work.

Answered By: user3101695

Answer #6:

We might confuse ourselves that a * b is a dot product.

But in fact, it is broadcast.

Dot Product :
a.dot(b)

Broadcast:

The term broadcasting refers to how numpy treats arrays with different
dimensions during arithmetic operations which lead to certain
constraints, the smaller array is broadcast across the larger array so
that they have compatible shapes.

(m,n) +-/* (1,n) ? (m,n) : the operation will be applied to m rows

Answered By: chia yongkang

Answer #7:

Convert the arrays to matrices, and then perform the multiplication.

X = np.matrix(X)

y = np.matrix(y)

X*y
Answered By: Azzam Radman

Leave a Reply

Your email address will not be published. Required fields are marked *