Correlate a single time series with a large number of time series

Question

I have a large number (M) of time series, each with N time points, stored in an MxN matrix. Then I also have a separate time series with N time points that I would like to correlate with all the time series in the matrix.

An easy solution is to go through the matrix row by row and run numpy.corrcoef. However, I was wondering if there is a faster or more concise way to do this?

Answer 1

Let's use this correlation formula :

You can implement this for X as the M x N array and Y as the other separate time series array of N elements to be correlated with X. So, assuming X and Y as A and B respectively, a vectorized implementation would look something like this -

import numpy as np

# Rowwise mean of input arrays & subtract from input arrays themeselves
A_mA = A - A.mean(1)[:,None]
B_mB = B - B.mean()

# Sum of squares across rows
ssA = (A_mA**2).sum(1)
ssB = (B_mB**2).sum()

# Finally get corr coeff
out = np.dot(A_mA,B_mB.T).ravel()/np.sqrt(ssA*ssB)
# OR out = np.einsum('ij,j->i',A_mA,B_mB)/np.sqrt(ssA*ssB)

Verify results -

In [115]: A
Out[115]: 
array([[ 0.1001229 ,  0.77201334,  0.19108671,  0.83574124],
       [ 0.23873773,  0.14254842,  0.1878178 ,  0.32542199],
       [ 0.62674274,  0.42252403,  0.52145288,  0.75656695],
       [ 0.24917321,  0.73416177,  0.40779406,  0.58225605],
       [ 0.91376553,  0.37977182,  0.38417424,  0.16035635]])

In [116]: B
Out[116]: array([ 0.18675642,  0.3073746 ,  0.32381341,  0.01424491])

In [117]: out
Out[117]: array([-0.39788555, -0.95916359, -0.93824771,  0.02198139,  0.23052277])

In [118]: np.corrcoef(A[0],B), np.corrcoef(A[1],B), np.corrcoef(A[2],B)
Out[118]: 
(array([[ 1.        , -0.39788555],
       [-0.39788555,  1.        ]]),
 array([[ 1.        , -0.95916359],
       [-0.95916359,  1.        ]]),
 array([[ 1.        , -0.93824771],
       [-0.93824771,  1.        ]]))