how to create a dataframe in Rust so it can be used in DataFusion?

Question

I want to convert existing in-memory data (some columns and a schema) to a dataframe in Rust so it can be processed by DataFusion with its dataframe API. How can we do that?

I see the only constructor has the signature:

pub fn new(session_state: SessionState, plan: LogicalPlan) -> Self {

which requires the SessionState and LogicalPlan. However, the logical plan in my case is not known in advance.

One workaround is to write the data to disk as CSV and then read it back. Certainly it does not work for large data.

Answer 1

Here is a solution from the datafusion repo:

https://github.com/apache/arrow-datafusion/blob/main/datafusion-examples/examples/dataframe_in_memory.rs