I'm working in an .iypnb with pyspark and pandas.
I'd like to be able to take my dataset (df) and loop through various columns, outputting a freq table of the values within that column.
Doing so with .show() is easy enough:
columz = ['col1', 'col2', etc]
for c in columz:
df.groupBy(c).count().orderBy(c).show(200, False)
However, I don't want my freq table (dataframe) output via .show(), I want .toPandas() (it's cleaner, easier to read, and much easier to copy out).
Outputting just 1 df to Pandas is no problem, either:
df.groupBy('col1').count().orderBy('col1').toPandas()
But iteratively outputting Pandas doesn't work.
columz = ['col1', 'col2', etc]
for c in columz:
df.groupBy(c).count().orderBy(c).toPandas()
I feel like there has to be a workaround to this, but no matter how I try rewording my searches, I get nothing but results that aren't what I'm asking for.
(My df is usually too large to convert the whole thing to Pandas, btw)
Additional clarification based on first answer:
Thanks. print() does successfully print each table as you loop through.
Unfortunately, when you do print(df.groupBy(c).count().orderBy(c).toPandas()) it does not output in the same format as it prints when you do df.groupBy('col1').count().orderBy('col1').toPandas() It outputs in an annoying format that is (to me) better than what you get from using .show(), but not as good as what you get from using .toPandas()
(My primary complaint is that when you output via .toPandas(), you normally get a table that you can just highlight and paste directly into something like Excel and it automatically puts each column of your output into a separate column of the spreadsheet. Clean and easy. When you output via .show() or print( .toPandas()), if you copy and paste the result, it puts everything into a single column. Maybe there's a Paste option that works, but I tried lots of different ones with no success.)
There may not be a way to get what I want out of this, but I'm going to leave it open, just in case.
