vaderSentiment TypeError: 'float' object is not iterable

Question

I have a twitter dataset and want to do sentiment analysis using VaderSentiment. I want to iterate over the column where the preprocessed tweets are but get TypeError.

My code is like this

!pip install vaderSentiment
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import time
import re
import warnings
warnings.filterwarnings('ignore')

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()
df=pd.read_csv("full_data.csv", parse_dates=['Date'],engine='python')

scores = []
# Declare variables for scores
compound_list = []
positive_list = []
negative_list = []
neutral_list = []
for i in range(df['PreProcessed_Tweets'].shape[0]):
#print(analyser.polarity_scores(sentiments_pd['text'][i]))
    compound = analyzer.polarity_scores(df['PreProcessed_Tweets'][i])["compound"]
    pos = analyzer.polarity_scores(df['PreProcessed_Tweets'][i])["pos"]
    neu = analyzer.polarity_scores(df['PreProcessed_Tweets'][i])["neu"]
    neg = analyzer.polarity_scores(df['PreProcessed_Tweets'][i])["neg"]
    
    scores.append({"Compound": compound,
                       "Positive": pos,
                       "Negative": neg,
                       "Neutral": neu
                  })
    
sentiments_score = pd.DataFrame.from_dict(scores)
df = df.join(sentiments_score)
df.head(20)

The error message: TypeError: 'float' object is not iterable

How can I deal with this error?

Answer 1

I think you try to use a float in your range()

Look at this variable : df['PreProcessed_Tweets'].shape[0]

Regards,

Answer 2

It's because of NaN values PreProcessed_tweets column. I had the same problem. When I removed stopwords during preprocessing, there was one value of a text that contained only stopwords so after the preprocessing, it contained NaN value. First, it didn't come to my mind, because I checked for NaN values beforehand. You probably had the same issue. I know that it's quite late now, but it might help someone else facing the same issue.

Answer 3

I just had the same error and solved it by checking the relevant column (the column that contains the text you want to analyse) for NaNs:

value = df['column'].isnull()
print(sum(value))

This will be zero if there are no NaNs.

To fix this, I replaced the NaNs with an empty string:

df['column'] = df['column'].replace(np.nan, '', regex=True)