Rust Polars WebAssembly CSVReader

Question

I am having an issue below when trying to upload a CSV file and parse it in web assembly polars using rust.

Thanks

Error:

Grid.js:498 panicked at 'unsafe precondition(s) violated: ptr::read requires that the pointer argument is aligned and non-null', C:\Users\61414\.rustup\toolchains\nightly-2022-11-20-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\panicking.rs:89:58

I have example code below.

Furthermore, I am using Svelte as a front end, but don't think that will make much of a difference.

Rust:

    pub fn load_csv(&mut self, buff: &[u8]) -> String {

        let cursor = Cursor::new(buff);

        let lf = CsvReader::new(cursor).with_ignore_parser_errors(true).finish().unwrap().lazy();

        return lf.describe_plan();
    }

Typescript:

Just a function triggered when a file is uploaded to a file input.

    function load_csv_file(event){

        const file = event.target.files[0]

        let reader = new FileReader();
        reader.readAsArrayBuffer(file);
        reader.onload = data1 => {
            console.log(data1.target.result)

            console.log(data.load_csv(new Uint8Array(data1.target.result)))
        };
    }

WASM Trace:

Not sure how to get anything better

Error
    at http://localhost:5173/rustFunctions/grid/pkg/Grid.js:504:21
    at logError (http://localhost:5173/rustFunctions/grid/pkg/Grid.js:134:18)
    at imports.wbg.__wbg_new_abda76e883ba8a5f (http://localhost:5173/rustFunctions/grid/pkg/Grid.js:503:66)
    at console_error_panic_hook::Error::new::h5d7996250e9efb8e (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[154789]:0x30fc8e8)
    at console_error_panic_hook::hook_impl::h667dd0ae102fb048 (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[29585]:0x1c5c717)
    at console_error_panic_hook::hook::hc3def586df00a6f2 (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[169823]:0x31c84b3)
    at core::ops::function::Fn::call::h397369bf956b8712 (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[149254]:0x30a5529)
    at <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hfcda8f4a283bd42a (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[125833]:0x2ee5fac)
    at std::panicking::rust_panic_with_hook::h7e6939e50e26b51d (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[914]:0x4641c8)
    at std::panicking::begin_panic_handler::{{closure}}::hfff77ccb8fcf1114 (http://localhost:5173/rustFunctions/grid/pkg/grid_bg.wasm:wasm-function[34760]:0x1e31219)

So far:

• I have checked the array buffer is not null by, checking the cursor using .is_empty() and printing out the [u8] which matches a [u8] array for a CSV parser which does work when I run the polars rust code locally without Wasm.
• as I mentioned, I used the same rust code locally without Wasm where the file is loaded correctly.

So I am guessing there is something going wrong with the cursor creation or passing to polars when using WASM, but that is just a guess.

Answer 1

you can take a look at the ` js-polars implementation for reference.

Generally when exposing rust functions to JS, you should either use wasm-bindgen or use ffi via extern "C" functions. wasm-bindgen is much easier to use than manually handling memory & pointers though.

Some other things to note:

If processing large datasets, you will like run into some max memory limits with wasm & rust. You can modify it by updating your .cargo/config.toml

There is also significant overhead in spinning up a web worker threadpool for parsing CSV's, and you will need to run the parse_csv function within a worker. the @polars/browser npm package simplifies a lot of this, and you can take a look in there for some guidelines on how to run polars in the browser.