How to (de)serialize Polars DataFrame to send it over REST call?

Question

I would like to convert polars::frame::DataFrame into a serializable format to send it over a REST call. For performance reasons, I would want to avoid JSON or CSV and instead use the arrow format.

While I see that there is a to_arrow() and from_arrow() in Python. I didn't find any such functions in Rust.

How do I serialize and de-serialize a Polars DataFrame into binary format?

Answer 1

You can use either the Arrow IPC or the Parquet formats. The differences between them are outlined in this Stack Overflow answer.

For IPC, decoding from a Read is done with IpcStreamReader, and encoding to a Write is done with IpcStreamWriter.

For Parquet, decoding is done with ParquetReader, and encoding with ParquetWriter.