Is it possible to interpolate and ffill different columns in a Koalas dataframe something like this?
%%spark -s sparkenv2
kdf = ks.DataFrame({
'id':[1,2,3,4],
'A': [None, 3, None, None],
'B': [2, 4, None, 3],
'C': [99, None, None, 1],
'D': [0, 1, 5, 4]
},
columns=['id','A', 'B', 'C', 'D'])
kdf['A']=kdf['A'].ffill()
kdf['B']=kdf['B'].interpolate()
For ffill, this is taken from John Paton's blog
I have no answer for interpolate - still trying to find it myself.
PS - You can switch to backfill, by changing rowsBetween(0, max.size) and using first() rather than last().