Use Flume to stream a webpage data to HDFS

Question

I have 3 node cluster, using latest cloudera parcels for 5.9 version. OS is CentOS 6.7 on all three of them. I am using Flume for the first time.

My purpose is to stream webpage data into HDFS. However this webpage is a third party website, news site in my case so I don't know which Port to use to connect.

Curl and telnet was happening on port 80, hence i used it. But getting error.

My Flume.conf is:

tier1.sources  = http-source
tier1.channels = mem-channel-1
tier1.sinks    = hdfs-sink
tier1.sources.http-source.type     = http
tier1.sources.http-source.handler = org.apache.flume.source.http.JSONHandler
tier1.sources.http-source.bind     = 132.247.1.32
tier1.sources.http-source.port     = 80
tier1.sources.http-source.channels = mem-channel-1
tier1.channels.mem-channel-1.type   = memory
tier1.sinks.hdfs-sink.type         = hdfs
tier1.sinks.hdfs-sink.channel      = mem-channel-1
tier1.sinks.hdfs-sink.hdfs.path    = /flume/events/%y-%m-%d/%H%M/%S
# Other properties are specific to each type of
# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.
tier1.channels.mem-channel-1.capacity = 100

Error

2016-12-19 16:45:00,353 WARN org.mortbay.log: failed [email protected]:80: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 WARN org.mortbay.log: failed Server@36772002: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 ERROR org.apache.flume.source.http.HTTPSource: Error while starting HTTPSource. Exception follows.
java.net.BindException: Cannot assign requested address**
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.mortbay.jetty.nio.SelectChannelConnector.doStart2016-12-19 16:45:00,364 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{
name:http-source,state:IDLE} } - Exception follows.
java.lang.RuntimeException: java.net.BindException: Cannot assign requested address(SelectChannelConnector.java:315)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:207)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Answer 1

Try changing the source config as below : httpagent.sources.http-source.port = 80 httpagent.sources.http-source.bind = localhost httpagent.sources.http-source.url = 132.247.1.32

Note : If 132.247.1.32 doesn't work, try giving the hostname.

Answer 2

The HTTP source for Flume provides a way to use GET/POST to send data into a flume agent. The HTTP source does not go out and fetch data from websites for you, it sets up a HTTP server and waits for GET/POST and accepts the data from that endpoint.

My recommendation would be to create a custom source that fetches the webpage you require.