I am working on a web assembly application in rust. The program relies on streaming a file and storing the data in memory. However I run into a rust_oom
issue.
This issue only arises when recompiling the std library with atomic, bulk-memory, and mutable-global flags. Reproducable via .cargo/config.toml:
[target.wasm32-unknown-unknown]
rustflags = ["-C", "target-feature=+atomics,+bulk-memory,+mutable-globals"]
[unstable]
build-std = ["panic_abort", "std"]
Compiling without these flags works fine.
The relevant rust code:
#[wasm_bindgen]
pub async fn start(r: JsValue) {
// javascript response into websys response
let resp: Response = r.dyn_into().unwrap();
// from example here https://github.com/MattiasBuelens/wasm-streams/blob/master/examples/fetch_as_stream.rs
let raw_body = resp.body().unwrap();
let body = ReadableStream::from_raw(raw_body.dyn_into().unwrap());
let mut stream = body.into_stream();
// store the data in memory
let mut v = vec![];
log("start streaming");
while let Some(Ok(chunk)) = stream.next().await {
//convert to vec<u8>
let mut x = chunk.dyn_ref::<Uint8Array>().unwrap().to_vec();
v.append(&mut x);
}
log(&format!("{}", v.len()));
log("done streaming");
}
The full error message provided
at rust_oom (wasm_memtest_bg.wasm:0xfa10)
at __rg_oom (wasm_memtest_bg.wasm:0xfdd4)
at __rust_alloc_error_handler (wasm_memtest_bg.wasm:0xfd38)
at alloc::alloc::handle_alloc_error::rt_error::hf991f317b52eeff2 (wasm_memtest_bg.wasm:0xfdb3)
at core::ops::function::FnOnce::call_once::ha90352dededaa31f (wasm_memtest_bg.wasm:0xfdc9)
at core::intrinsics::const_eval_select::h04f9b6091fe1f42f (wasm_memtest_bg.wasm:0xfdbe)
at alloc::alloc::handle_alloc_error::h82c7beb21e18f5f3 (wasm_memtest_bg.wasm:0xfda8)
at alloc::raw_vec::RawVec<T,A>::reserve::do_reserve_and_handle::h65538cba507bb5c9 (wasm_memtest_bg.wasm:0xc149)
at <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he4014043296c78b9 (wasm_memtest_bg.wasm:0x1c85)
at wasm_bindgen_futures::task::multithread::Task::run::h77f1075b3537ddf9 (wasm_memtest_bg.wasm:0x5a0e)
Here is the full project if you want to test it out.
https://github.com/KivalM/memory-wasm
EDIT: The code required to reproduce the issue is provided in the first snippet. The full wasm application is provided in the repository so that you do not have to create one from scratch. I will include the rust code in the question.
The issue arises when loading a simple 500MB file into memory. Which should be possible via this approach. Yet Chrome(and chromium-based browsers) tend to run out of memory, whereas when the rustflags are not present the program runs just fine.
I will move the links here as they are not part of the question, but can be relevant.
https://rustwasm.github.io/2018/10/24/multithreading-rust-and-wasm.html https://rustwasm.github.io/wasm-bindgen/examples/raytrace.html