Flume not accepting keywords for Twitter stream

Question

a Hadoop neophyte here, using this tutorial: https://acadgild.com/blog/streaming-twitter-data-using-flume/ to capture tweets. Here is my flume.conf file:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey=xxxx
TwitterAgent.sources.Twitter.consumerSecret=xxxx
TwitterAgent.sources.Twitter.accessToken=xxxx
TwitterAgent.sources.Twitter.accessTokenSecret=xxxx

TwitterAgent.sources.Twitter.keywords= #canpoli

TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs:/xxxx/user/flume/tweets
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600

TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=1000

TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel`

It streams tweets fine, it saves into my desired directory correctly, but it seems to be streaming everything without filtering for my keyword. I get tweets from all over the world except with that hashtag.

What might be the issue?

Answer 1

First, it's accepting all hashtags because you gave an empty list.

# character is a comment, so everything after the equals is seen as ignored. At least I think that's how it is parsed.

You linked to a site that doesn't use the #, so I would follow that tutorial until it worked.

Secondly, that source is considered experimental, and doesn't seem to mention keywords.

https://flume.apache.org/FlumeUserGuide.html#twitter-1-firehose-source-experimental

Your code looks almost exactly like this example from Cloudera that includes comments in the config and has keywords, which if you check, there's a different source class.

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource

https://github.com/cloudera/cdh-twitter-example/blob/master/flume-sources/flume.conf

You need to download that java code there, package it to a JAR, and place it in the Flume lib directory