In Spark 2.2 extracting date not working from unix_timestamp Input Data:
+-------------------------+
|UPDATE_TS |
+-------------------------+
|26NOV2009:03:27:01.154410|
|24DEC2012:00:47:46.805710|
|02MAY2013:00:45:33.233844|
|21NOV2014:00:33:39.350140|
|10DEC2013:00:30:30.532446|
I tried following approaches but output Im getting as null
Query Tired:
Spark sql
sqlContext.sql("select from_unixtime(unix_timestamp(UPDATE_TS,'ddMMMyyyy:HH:MM:SS.ssssss'), 'yyyy') as new_date from df_vendor_tab").show()
DSL:
df_id.withColumn('part_date', from_unixtime(unix_timestamp(df_id.UPDATE_TS, "ddMMMyyyy:HH:MM:SS.sss"), "yyyy"))
expected output:
2009
2012
2013
2014
2013
You're using the incorrect format string. Capital
M
is for month. Lower casem
is for minute.The following would work: