Why is Flume not moving my locally stored data to hdfs?

35 Views Asked by At

I am testing out Flume, and want to use it for moving data I have stored locally to hdfs, but I have run in to a problem that I don't know how to troubleshoot, as it gives no direct errors;

When I run 'flume-ng agent -n agent1 -c conf -f C://Flume/conf/flume.conf -property flume.root.logger-DEBUG,console' I get the following response;

C:\Flume>flume-ng agent -n agent1 -c conf -f C://Flume/conf/flume.conf -property flume.root.logger-DEBUG,console

C:\Flume>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File C:\Flume\bin\flume-ng.ps1 agent -n agent1 -c conf -f C://Flume/conf/flume.conf -property flume.root.logger-DEBUG,console

WARN: Config directory not set. Defaulting to C:\Flume\conf
Sourcing environment configuration script C:\Flume\conf\flume-env.ps1
Including Hadoop libraries found in (C:\hadoop-3.2.4) for DFS access
WARN: HBASE_HOME not found
WARN: HIVE_HOME not found

  Running FLUME agent :
    class: org.apache.flume.node.Application
    arguments: -n agent1 -f "C:\Flume\conf\flume.conf"

I get no other error, but it doesn't seem like it is moving the data either as the console remains static, and the file explorer is my hdfs localhost is still empty.

This is my flume.conf file;

agent1.sources = tail
agent1.channels = Channel-2
agent1.sinks = sink-1
agent1.sources.tail.type = exec
agent1.sources.tail.command = cat /Flume/twitterUserLogs/log.1.txt
agent1.sources.tail.channels = Channel-2
agent1.sinks.sink-1.channel = Channel-2
agent1.sinks.sink-1.type = hdfs
agent1.sinks.sink-1.hdfs.path = hdfs://localhost:9000/user/Hadoop/twitter_data/
agent1.sinks.sink-1.hdfs.fileType = DataStream
agent1.sinks.sink-1.hdfs.writeFormat = Text
agent1.sinks.sink-1.hdfs.rollInterval = 60
agent1.sinks.sink-1.hdfs.rollSize = 0
agent1.sinks.sink-1.hdfs.rollCount = 0
agent1.channels.Channel-2.type = memory

I hope someone knows what is happening and can give me some advice on how to fix this, thanks in advance.

0

There are 0 best solutions below