I have a certain filtering routine
import scipy as sp
def remove_wiggles(x, y, p):
""" Wiggle removing routine. Idea is to interpolate between the points in which the
curvature goes to 0 to obtain the trend of the oscillation """
# Interpolate the given curve (x, y)
# The trend of the curve should be where the curvature of the signal goes to 0
fwiggle = sp.interpolate.UnivariateSpline(
x,
y,
k=3,
s=0
)
# fwiggle is a cubic spline.
# derivs is an array which contains the 3 derivatives (0, 1 and 2 order) at each point of x
derivs = np.array(
[fwiggle.derivatives(_k) for _k in x]
).T
# Get and interpolate the second derivative
d2 = sp.interpolate.UnivariateSpline(x,
derivs[2],
k=3,
s=1.0
)
wzeros = d2.roots()
wtrend = sp.interpolate.UnivariateSpline(
wzeros,
fwiggle(wzeros),
k=3,
s=0
)
return wtrend(x)
which works perfectly fine on the tests I run on my laptop. However, when I try to use this filtering method on the cluster, I get the following error dfitpack.error: (m>k) failed for hidden m: fpcurf0:m=2, with the signal y being exactly the same.
I used this filtering for a past project, and encountered the same problem. Luckily I did not need to use the cluster to filter this. However for my present project filtering needs to be applied each time a new signal is generated and for that I need the cluster.
Seems to be a problem of the cluster on how it calculates the interpolation. For example, for the case of the y's I am using right now, in my laptop it yields a (correct) filtering and the wzeros array (the points where the signal crosses the trend line, the one we are interested in) has a len() in the order of the hundreds, while in the cluster it just has 2 elements (which is what raises the error).
For reference
and the filtered signal by my laptop
The red line you can ignore, it is just the second integral of this curve.