Identifying memory leaks in python

102 Views Asked by At

I've a method in python, which isn't memory intensive at all, but each time my code snippet reaches that method, after around 30-40 iterations, the system's RAM reaches 100% and my system freezes. The same method wasnt giving any issues before, but now I simple cant seem to find a way out of method. Any specific reason? I've got i5 processor (CPU usage is not even 50%) and 8Gb RAM. The code snippet is as follows:

def modify_data(self, data, col):
    """
    Convert Raw scores to Z-scores.

    Args:
        data (Pandas DataFrame): DataFrame storing raw scores
        col (String): The column on which conversion is being done

    Returns:
        z_score (Pandas DataFrame): New DataFrame storing z-scores instead of raw scores.
    """
    print "inside modify_data"
    # Now calculate z-score based on rolling windows for columns calculated above.

    z_score = pd.DataFrame()

    score = data
    z_scores = []

    ctr = 0

    indeces = score.index.tolist()

    for idx in indeces:
        row = score.ix[idx]

        n = min(idx, self.window)

        if self.use_window:
            subdata = score.iloc[ctr-n:ctr+1]
        else:
            subdata = score.iloc[:ctr+1]

        row = score.ix[idx]
        x = row[0]

        mu = subdata.mean()
        sigma = subdata.std()
        new_score = (x-mu)/sigma

        z_scores = np.append(z_score, new_score)

        ctr += 1

        z_scores = np.array(z_scores)
        tmpDF = pd.DataFrame(data=z_scores, columns=[col])

        z_score = z_score.append(tmpDF)

    return z_score

Earlier I had implemented:

for index, row in score.iterrows():

to iterate over loop, and thought maybe iterrows is slow performance based, and hence switched over to iterating over index list. But even that didnt seem to help. Any inputs will be really helpful

Thanks..

0

There are 0 best solutions below