Some columns became NaN after scaling

626 Views Asked by At

I'm trying to scale features with such a function

def featureNormalize(X):
'''
This function takes the features as input and
returns the normalized values, the mean, as well 
as the standard deviation for each feature.
'''
X_norm = (X - np.mean(X))/np.std(X)
mu = np.mean(X)
sigma = np.std(X)
return X_norm, mu, sigma

and then call it

X, mean, std = featureNormalize(X) ## We call the function over the features
Train Set Test Set
enter image description here enter image description here

So that it works ok for my train set.

But when I call it for a test set], some columns completely turn into Nan Both sets have no Nan or null values. I've tried to rewrite this function with nanmean and nanstd, but it didn't work :

def featureNormalize(X):
    X_norm = (X - np.nanmean(X, dtype = 'float32')) / np.nanstd(X, dtype = 'float32')
    mu = np.nanmean(X, dtype = 'float32')
    sigma = np.nanstd(X, dtype = 'float32')
    return X_norm, mu, sigma

What can cause this problem and what should I do

0

There are 0 best solutions below