Failing to index csv file based data in opendistro elasticsearch

Question

I am trying to index sample csv based data into opendistro elasticsearch but failing to create the index. Could you please let me what i am missing here.

csv file to index

[admin@fedser32 logstashoss-docker]$ cat /tmp/student.csv 
"aaa","bbb",27,"Day Street"
"xxx","yyy",33,"Web Street"
"sss","mmm",29,"Adam Street"

logstash.conf

[admin@fedser32 logstashoss-docker]$ cat logstash.conf
input {
  file {
    path => "/tmp/student.csv"
    start_position => "beginning"
}
}
filter {
  csv {
    columns => ["firstname", "lastname", "age", "address"]
  }
}
output {
    elasticsearch {
        hosts => ["https://fedser32.stack.com:9200"]
        index => "sampledata"
        ssl => true
        ssl_certificate_verification => false
        user => "admin"
        password => "admin@1234"
    }
}

My Opendistro cluster is listening on 9200 as shown below.

[admin@fedser32 logstashoss-docker]$ curl -X GET -u admin:admin@1234 -k https://fedser32.stack.com:9200
{
  "name" : "odfe-node1",
  "cluster_name" : "odfe-cluster",
  "cluster_uuid" : "5GOEtg12S6qM5eaBkmzUXg",
  "version" : {
    "number" : "7.10.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
    "build_date" : "2020-11-09T21:30:33.964949Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

As per the logs it does indicate it is able to find the csv file as shown below.

logstash_1  | [2022-03-03T12:11:44,716][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
logstash_1  | [2022-03-03T12:11:44,716][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
logstash_1  | [2022-03-03T12:11:44,725][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x5c537d14 run>"}
logstash_1  | [2022-03-03T12:11:45,439][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.71}
logstash_1  | [2022-03-03T12:11:45,676][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_20d37e3ca625c7debb90eb1c70f994d6", :path=>["/tmp/student.csv"]}
logstash_1  | [2022-03-03T12:11:45,697][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1  | [2022-03-03T12:11:45,738][INFO ][filewatch.observingtail  ][main][2f140d63e9cab8ddc711daddee17a77865645a8de3d2be55737aa0da8790511c] START, creating Discoverer, Watch with file and sincedb collections
logstash_1  | [2022-03-03T12:11:45,761][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1  | [2022-03-03T12:11:45,921][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Answer 1

Could you check the access right for /tmp/student.csv file? it must be readable by user logstash. check with this command:

#ls -l /tmp

Other way, if you have already indexed the file path, you have to clean up the sincedb

Answer 2

The thing that i was missing is i had to volume mount my CSV file into the logstash container as shown below after which i was able to index my csv data.

[admin@fedser opensearch-logstash-docker]$ cat docker-compose.yml 
version: '2.1'
services:
  logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
    ports:
      - "5044:5044"
    volumes:
      - $PWD/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - $PWD/student.csv:/tmp/student.csv