How do I construct an instance of a DST struct whose size is only known at runtime?

Question

I have a struct that can be used in a fixed-size or unsized way, where the last element is an array of u8 items. We can write the following struct definition:

struct MyStructImpl<E: ?Sized> {
    field: char,
    extendable: E,
}
type MyStructFixed<const N: usize> = MyStructImpl<[u8; N]>;
type MyStructUnsized = MyStructImpl<[u8]>;

It is possible to acquire a handle to a unsized version of this struct if we know the size at compilation time. For example:

impl MyStructUnsized {
    fn new_from_fixed_array<const N: usize>(field: char, arr: [u8; N]) -> Box<Self> {
        Box::new(MyStructFixed {
            field,
            extendable: arr,
        })
    }
}

However, I can't find a way to construct the struct if you only know the size at runtime. For example:

impl MyStruct<[u8]> {
    fn new_from_slice(field: char, arr: &[u8]) -> Box<Self> {
        todo!(); // help!
    }
}

In theory this should be possible to do: We just need to allocate some space of the correct size, then construct the struct in there, and finally copy the u8s over. Box<T> is supposed to know how to free an unsized T, so this shouldn't be a problem.

However, in practice, this is not at all simple to do in Rust. (This is very simple to do in C or C++, owing to the fact that struct layout in those languages is pretty well-defined.)

std::boxed::Box has from_raw(), which takes ownership of a *mut T and puts it in a Box. An allocated *mut T can be gotten from a call to std::alloc::alloc(). However, std::alloc::alloc() needs a std::alloc::Layout, which is essentially a size and an alignment, but there seems to be no way to calculate the size and alignment of a MyStructFixed<N> if N is only known at runtime, since the layout of a regular #[repr(Rust)] type is unspecified. In theory the compiler should know it, since it needs to emit code for any member functions of MyStructUnsized, so it would seem reasonable for a compiler intrinsic with the following signature to exist:

fn get_layout<T: ?Sized>(len: usize) -> Layout;

However, I can't find such a thing.

I think something like this might work:

fn guess_required_layout(len: usize) -> Layout {
    Layout::from_size_align(
        std::mem::size_of::<MyStructFixed<0>>() + len * std::mem::size_of::<u8>(),
        std::mem::align_of::<MyStructFixed<0>>(),
    )
    .unwrap()
}

... but that makes an assumption about the layout of extendable structs, which seems to not be meant to be guaranteed.

What is the idiomatic way to construct a Box of an unsized struct whose size is only known at runtime, ideally with the latest stable Rust (1.74.0 at the time of writing)?

Answer 1

Custom DSTs are a half-baked feature.

On stable, AFAIK, the only way is to mark the struct #[repr(C)]. Then you can calculate the layout manually like:

use std::alloc::Layout;
use std::ptr::addr_of_mut;

#[repr(C)]
struct MyStructImpl<E: ?Sized> {
    field: char,
    extendable: E,
}
type MyStructFixed<const N: usize> = MyStructImpl<[u8; N]>;
type MyStructUnsized = MyStructImpl<[u8]>;

impl MyStructUnsized {
    fn new_from_iterator(field: char, arr: &[u8]) -> Box<Self> {
        let layout = Layout::new::<char>()
            .extend(Layout::for_value(arr))
            .unwrap()
            .0
            .pad_to_align();

        unsafe {
            let data = std::alloc::alloc(layout);
            if data.is_null() {
                std::alloc::handle_alloc_error(layout);
            }

            let data =
                std::ptr::slice_from_raw_parts_mut::<u8>(data, arr.len()) as *mut MyStructUnsized;
            addr_of_mut!((*data).field).write(field);
            addr_of_mut!((*data).extendable)
                .cast::<u8>()
                .copy_from_nonoverlapping(arr.as_ptr(), arr.len());

            Box::from_raw(data)
        }
    }
}

On nightly, you could in theory do slightly better with Layout::for_value_raw(), however, one of its safety precondition is that the size of the entire thing will fit in isize, and to calculate that to know the size of the prefix of the DST, which is the fields (known, size of char) but also the padding, and you can calculate the padding via the size of the whole struct but then we're back to point one, or you can calculate it via offset_of!() of extendable, but offset_of!() is unstable and doesn't currently allows DSTs, and while there are crates to implement it on stable they don't support unsized structs... So in conclusion I'm not sure the hassle is worth it.