How to ensure my dependency's dependency version (where they use `>=XY`)?

90 Views Asked by At

I have two crates (one for this example and second is the large prod version) with identical code, and identical following dependency's versions in my Cargo.toml Example script compiles, and the other one doesn't.

My crates have:

polars = {version="0.35.4", features=["lazy", "serde", "serde-lazy", "is_in", "diagonal_concat", "streaming", "object"]}
polars-arrow = {version = "0.35.4", features=["arrow_rs"]}
connectorx = {git="https://github.com/sfu-db/connector-x", features=["dst_arrow", "src_mysql"]}
arrow-array = ">=46"

Note that polars uses

arrow-array = {version = ">=41"}

And connectorx:

arrow = {version = "46", features = ["prettyprint", "ffi"]}

PROBLEM See code snippet below (unfortunately, to reproduce you'd need a running MySQL server). My second crate does not compile with this error: the trait bound polars::prelude::ArrowField: std::convert::From<&arrow_schema::field::Field> is not satisfied.

Whereas the my first crate compiles fine.

What seems to be happening is that in my second crate, ArrowField implements From<Field>, where Field comes from arrow-array="49" and since connectorx returns arrow-array="46" Field - there is a conflict. However, my first crate is somehow able to resolve this issue. In my first crate, polars's ArrowField implements From<Field> for arrrow-array="46", and code compiles.

So the question ultimately is - how do I enforce polars-arrow to use arrow-array="46" , like it does in my first crate?

FYI this is the code (which doesn't compile in my second crate)

use connectorx::{source_router::SourceConn, sql::CXQuery, get_arrow};
use polars::{prelude::{DataFrame, Field}, series::Series, functions::concat_df_diagonal};
use polars_arrow::{array::Array, datatypes::{Field as PolarsArrowField}};
use arrow_array::RecordBatch;

pub fn conn() {

    let uri=format!("mysql://{0}:{1}@{2}:{3}/{4}?cxprotocol=binary",
        "username" , "password", "localhost", 3306, "dbname");

    let mut source_conn = SourceConn::try_from(uri.as_str()).expect("parse conn str failed");

    let queries = &[CXQuery::from("SELECT DISTINCT * FROM frtb")];
    let destination = get_arrow::get_arrow(&source_conn, None, queries).expect("run failed");

    let mut data = destination.arrow().unwrap();

    let dt_cl = data.clone();

    let res = record_batches_to_df(dt_cl);

}
    
pub fn record_batches_to_df<I>(batches: I) -> DataFrame
    where I: IntoIterator<Item=RecordBatch>
{
    let mut batches_iter: <I as IntoIterator>::IntoIter = batches.into_iter();

    let mut dfs = vec![];

    for next_batch in batches_iter {
        dfs.push(batch_to_df(next_batch));
    }

    let res = concat_df_diagonal(&dfs).unwrap();

    res
}

pub fn batch_to_df(batch: RecordBatch) -> DataFrame {
    let mut columns = vec![];
    batch.schema().all_fields().into_iter()
            .zip(batch.columns())
            .for_each(|(f, c)|{
                let polars_field = Field::from(&PolarsArrowField::from(f));
                let chunk: Box<dyn Array> = From::from(c.as_ref());
                let s = unsafe {Series::from_chunks_and_dtype_unchecked(polars_field.name.as_str(), vec![chunk], &polars_field.data_type())};
                columns.push(s);
            });
    DataFrame::from_iter(columns)
}
0

There are 0 best solutions below