It will become very difficult to read the output if it is not in the good formated manner like shown in the image in this link -> required output. I am running my code on jupyter notebook in pyspark with apache-toree kernel. Currently when i show my dataframe it looks somewhat like as shown in the image in this link -> Current Table.
I know there is a way using "magic" as mentioned on apache toree website but it is also not working in pyspark. It works if i load a normal python notebook in jupyter. It gives the error shown in the image with pyspark
I am not able to find anything on internet about this. Any help is very much appreciated.
I work around this problem by converting pyspark DataFrame to pandas DataFrame with toPandas method.