kylo feed keeps running and does not finish issue

Question

I am new to Kylo

I manually deployed Kylo on Test cluster of Hortonworks HDP 2.5 and have reused my Nifi instance.

I made a sample feed by following like ingestion tutorial (User Signups) and was successful.

However, when I drop sample data file in /var/dropzone/ the file is removed (assuming its fetched and read by Nifi) and feed is running for the last 18 hours. It seems too much for a small file.

I looked at the nifi-app.log and could only find belo error which is certainly not helpful

2017-05-29 09:38:41,045 INFO [Timer-Driven Process Thread-2] c.t.nifi.v2.common.FeedProcessor Resolving id ed2033a4-4bd5-4ea1-b9bc-2a48d69afef9 for feed /
2017-05-29 09:38:41,046 INFO [Timer-Driven Process Thread-5] c.t.nifi.v2.common.FeedProcessor Resolving ID for feed /
2017-05-29 09:38:41,048 INFO [Timer-Driven Process Thread-4] o.a.n.processors.standard.LogAttribute LogAttribute[id=01a8415a-0b38-14f2-8190-30f49a59ce2c] logging for flow file StandardFlowFileRecord[uuid=545885ac-eaa9-4387-b02a-a127594adc32,claim=,offset=0,name=19269191970708822,size=0]
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
        Value: 'Mon May 29 09:38:40 AST 2017'
Key: 'lineageStartDate'
        Value: 'Mon May 29 09:38:40 AST 2017'
Key: 'fileSize'
        Value: '0'
FlowFile Attribute Map Content
Key: 'feedId'
        Value: 'ed2033a4-4bd5-4ea1-b9bc-2a48d69afef9'
Key: 'filename'
        Value: '19269191970708822'
Key: 'path'
        Value: './'
Key: 'uuid'
        Value: '545885ac-eaa9-4387-b02a-a127594adc32'

2017-05-29 09:38:41,088 INFO [Timer-Driven Process Thread-8] c.t.n.p.cache.FeedFlowFileCacheUtil Ending the Job for Feed null and flowfile: 47e9d23c-1b7f-4d64-8b2c-feb9e944838f.  Event: ProvenanceEventRecordDTO{eventId=16975572, processorName=Load High Watermark Failure, componentId=01a8415a-0b38-14f2-8190-30f49a59ce2c, flowFile=47e9d23c-1b7f-4d64-8b2c-feb9e944838f, previous=null, eventType=DROP, eventDetails=Auto-Terminated by success Relationship, isEndOfJob=true, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 ERROR [Timer-Driven Process Thread-8] c.t.n.p.ProvenanceEventCollector Provenance: Cant find Feed for ProvenanceEventRecordDTO{eventId=16975572, processorName=Load High Watermark Failure, componentId=01a8415a-0b38-14f2-8190-30f49a59ce2c, flowFile=47e9d23c-1b7f-4d64-8b2c-feb9e944838f, previous=null, eventType=DROP, eventDetails=Auto-Terminated by success Relationship, isEndOfJob=true, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 ERROR [Timer-Driven Process Thread-8] c.t.n.p.ProvenanceEventCollector Provenance: Cant find Feed for ProvenanceEventRecordDTO{eventId=16975573, processorName=Load High Watermark, componentId=01a84157-0b38-14f2-d63d-c41fbd9c38a3, flowFile=574f778e-40f1-4fd4-863a-6d19613c127e, previous=null, eventType=CREATE, eventDetails=null, isEndOfJob=false, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 INFO [Timer-Driven Process Thread-8] c.t.n.p.cache.FeedFlowFileCacheUtil Ending the Job for Feed null and flowfile: 574f778e-40f1-4fd4-863a-6d19613c127e.  Event: ProvenanceEventRecordDTO{eventId=16975574, processorName=Load High Watermark Failure, componentId=01a8415a-0b38-14f2-8190-30f49a59ce2c, flowFile=574f778e-40f1-4fd4-863a-6d19613c127e, previous=null, eventType=DROP, eventDetails=Auto-Terminated by success Relationship, isEndOfJob=true, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 ERROR [Timer-Driven Process Thread-8] c.t.n.p.ProvenanceEventCollector Provenance: Cant find Feed for ProvenanceEventRecordDTO{eventId=16975574, processorName=Load High Watermark Failure, componentId=01a8415a-0b38-14f2-8190-30f49a59ce2c, flowFile=574f778e-40f1-4fd4-863a-6d19613c127e, previous=null, eventType=DROP, eventDetails=Auto-Terminated by success Relationship, isEndOfJob=true, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 ERROR [Timer-Driven Process Thread-8] c.t.n.p.ProvenanceEventCollector Provenance: Cant find Feed for ProvenanceEventRecordDTO{eventId=16975575, processorName=Load High Watermark, componentId=01a84157-0b38-14f2-d63d-c41fbd9c38a3, flowFile=2236a2a0-27e7-4f7d-8fcf-79441fe43e86, previous=null, eventType=CREATE, eventDetails=null, isEndOfJob=false, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 INFO [Timer-Driven Process Thread-8] c.t.n.p.cache.FeedFlowFileCacheUtil Ending the Job for Feed null and flowfile: 2236a2a0-27e7-4f7d-8fcf-79441fe43e86.  Event: ProvenanceEventRecordDTO{eventId=16975576, processorName=Load High Watermark Failure, componentId=01a8415a-0b38-14f2-8190-30f49a59ce2c, flowFile=2236a2a0-27e7-4f7d-8fcf-79441fe43e86, previous=null, eventType=DROP, eventDetails=Auto-Terminated by success Relationship, isEndOfJob=true, isBatch=true, isStream=false, feed=null}
2017-05-29 09:38:41,088 ERROR [Timer-Driven Process Thread-8] c.t.n.p.ProvenanceEventCollector Provenance: Cant find Feed for ProvenanceEventRecordDTO{eventId=16975576, processorName=Load High Watermark Failure, componentId=01a8415a-0b38-14f2-8190-30f49a59ce2c, flowFile=2236a2a0-27e7-4f7d-8fcf-79441fe43e86, previous=null, eventType=DROP, eventDetails=Auto-Terminated by success Relationship, isEndOfJob=true, isBatch=true, isStream=false, feed=null}

Below is the nifi processes. I don't see much here.

Any idea?

Answer 1

It sounds like the file may have gotten stuck in NiFi. Please go into NiFi and see if you can find the processor that is failing or stopped.