Apache drill fails to read this csv

Question

Trying select * query on the below csv file using apache drill 1.11.0.

id,email,first_name,last_name,middle_name,suffix,work_phone,mobile_phone,gender,picture,speciality,taxonomy_code,education_details,experience_details,keywords,doctor_npi,wait_time,created_tstamp,created_by,last_updated_tstamp,last_updated_by,is_deleted
1,[email protected],XXXXX,XXXX,,Dr,912225711234,,M,assets/images/doctorIcon.png,Primary Care Physician,Primary Care Doctor,M.D,3 years,Primary Care Doctor,1043259765,10,2015-04-22 17:20:48.0,,2015-12-16 12:06:27.0,,N
2,[email protected],XXXX,XXXX,,Dr,913375311234,,M,assets/images/doctorIcon.png,Eye Doctor,EYE Care Doctor,MD,5 years,,1619932076,20,2015-04-30 11:07:57.0,,2015-11-07 08:49:57.0,,N

I get this error :

org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: Error processing input: , line=1, char=292. Content parsed: [ ] Failure while reading file file:/..... Happened at or shortly before byte position 292. Fragment 0:0 [Error Id: 1ce7d94a-c06e-4633-af97-f3eceb1b5350 on 172.16.16.57:31010]

What is wrong here ?

Answer 1

Somehow column header name "suffix" is not working. If I use any other header name it works.

Answer 2

It seems to be a bug in Apache Drill but Praveen is right the problem is connected with suffix column. Suffix column is among four implicit columns in Drill (filename, suffix, fqn, filepath) [1]. Though the expected behavior here was supposed to be the result of implicit column suffix output (i.e. csv) rather than error. I'll file Jira for that.

If a column name has the same name as an implicit column, you can change the default implicit column name using the ALTER SYSTEM|SESSION SET command. For example: ALTER session SET `drill.exec.storage.implicit.suffix.column.label` = 'appendix';

[1] https://drill.apache.org/docs/querying-a-file-system-introduction/