Can Rust lazy-load dynamically linked (.dll/.so/.dylib) crates?

Question

So, im just curious. For example: i have a crate my_lib with the following code in Cargo.toml:

[lib]
crate-type = ["dylib"]

Then i can use the library in other crate:

[dependencies]
my_lib = { path = "../my_lib" }

This allows me to use my_lib in the program just like any statically-linked crate. And my_lib is compiled to a separate shared object.

But can rust actually lazy load dylib crates OR at least allow to still boot an executable even when required shared object is not present? By running i mean that program will be able to check that it lacks the library and will run without some features.

You might think that manually loading symbols via dlopen (or dlopen wrappers) is the solution. But there is the problem that in that case i cannot use types and structs and traits, etc, from the crate. I can only use loaded symbols.

What solutions, i think, might exist:

• Being able to pass some RUSTFLAGS, something remotely like /DELAYLOAD in windows.

• Being able to use types from other crates without linking those crates at all. Just manually creating wrappers for each function.

• Being able to manually create some sort of rust compiler plugin, that will in the end of the compilation collect a list of all the symbols and types and generate a wrapper-crate based on this.

• Some code-generating crate that generates wrappers of other crates. Those wrappers would manually load symbols from other crates, while still providing functions and types.

Do NOT care about being able to C-FFI types and ABI stuff. For me its acceptable to only use .dll/.so/.dylib-s that are compiled with the exact same version of the compiler, so ABI is the same.

Answer 1

For the purpose of this example, I would organize the crates like this.

.
├── lazy_app
│   ├── Cargo.toml
│   └── src
│       └── main.rs
├── lazy_impl_a
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── lazy_impl_b
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
└── lazy_interface
    ├── Cargo.toml
    └── src
        └── lib.rs

lazy_app is the application that needs to optionally load a crate at runtime.
lazy_interface defines the elements that anyone could expect from such a dynamically loaded crate: at least a trait and some utilities to obtain a dynamic object implementing this trait.
lazy_impl_a and lazy_impl_b are two examples of dynamic libraries implementing such an interface.

lazy_interface/src/lib.rs looks like this

/*
[dependencies]
libc = "0.2"
*/

use std::ffi::{CStr, CString};

pub trait LazyTrait {
    fn show_something(&self) -> String;
    fn do_something(&mut self);
}

pub struct LazyInterface {
    lib: *mut libc::c_void,
    make:
        fn(&[&str]) -> Result<Box<dyn LazyTrait>, Box<dyn std::error::Error>>,
}
impl LazyInterface {
    pub fn load(path: &str) -> Result<Self, Box<dyn std::error::Error>> {
        let lib_name = CString::new(path)?;
        let lib = unsafe {
            libc::dlopen(
                lib_name.as_ptr() as *const libc::c_char,
                libc::RTLD_LAZY,
            )
        };
        if lib.is_null() {
            let mut msg = format!("cannot load {:?}", path);
            Self::error_detail(&mut msg);
            Err(msg)?
        }
        let fnct_name = CString::new("make_lazy")?;
        let symb = unsafe {
            libc::dlsym(lib, fnct_name.as_ptr() as *const libc::c_char)
        };
        if symb.is_null() {
            let mut msg =
                format!("cannot find {:?} in {:?}", fnct_name, path);
            Self::error_detail(&mut msg);
            Self::close(lib);
            Err(msg)?
        }
        Ok(Self {
            lib,
            make: unsafe { std::mem::transmute(symb) },
        })
    }
    pub fn make(
        &self,
        params: &[&str],
    ) -> Result<Box<dyn LazyTrait>, Box<dyn std::error::Error>> {
        (self.make)(params)
    }
    fn error_detail(msg: &mut String) {
        let c_ptr = unsafe { libc::dlerror() };
        if !c_ptr.is_null() {
            msg.push_str(": ");
            msg.push_str(&unsafe { CStr::from_ptr(c_ptr) }.to_string_lossy());
        }
    }
    fn close(lib: *mut libc::c_void) {
        unsafe { libc::dlclose(lib) };
    }
}
impl Drop for LazyInterface {
    fn drop(&mut self) {
        Self::close(self.lib);
    }
}

LazyTrait should expose everything that could be expected from a dynamically loaded crate; this example relies only on two trivial functions.
LazyInterface provides the boilerplate to obtain a dynamic object implementing this trait.
I used libc::dlopen() to keep it simple, but some other solutions might help portability.
Note that this crate is a usual "lib" crate.

lazy_impl_a/src/lib.rs and lazy_impl_b/src/lib.rs could be defined like this

/*
[lib]
crate-type = ["cdylib"]

[dependencies]
lazy_interface = { path = "../lazy_interface" }
*/

use lazy_interface::LazyTrait;

struct LazyA {
    a: i32,
    b: String,
}
impl LazyTrait for LazyA {
    fn show_something(&self) -> String {
        format!("LazyA with {:?} and {:?}", self.a, self.b)
    }

    fn do_something(&mut self) {
        self.b.push_str(&format!("\nchanged {}", self.a));
        self.a += 5;
        self.b.push_str(&format!(" into {}", self.a));
    }
}

#[no_mangle]
pub fn make_lazy(
    params: &[&str]
) -> Result<Box<dyn LazyTrait>, Box<dyn std::error::Error>> {
    let first = *params.get(0).ok_or("missing first parameter")?;
    let second = *params.get(1).ok_or("missing second parameter")?;
    Ok(Box::new(LazyA {
        a: first.parse()?,
        b: second.to_owned(),
    }))
}

/*
[lib]
crate-type = ["cdylib"]

[dependencies]
lazy_interface = { path = "../lazy_interface" }
*/

use lazy_interface::LazyTrait;

struct LazyB {
    v: Vec<String>,
}
impl LazyTrait for LazyB {
    fn show_something(&self) -> String {
        format!("LazyB with {:?}", self.v)
    }

    fn do_something(&mut self) {
        self.v.push(format!("element {}", self.v.len()));
    }
}

#[no_mangle]
pub fn make_lazy(
    params: &[&str]
) -> Result<Box<dyn LazyTrait>, Box<dyn std::error::Error>> {
    Ok(Box::new(LazyB {
        v: params.iter().map(|&s| s.to_owned()).collect(),
    }))
}

The implementation of the trait does not have anything specific to dynamic code loading.
We just need to expose with no_mangle a function providing a dynamic object implementing this trait; this function must match exactly the prototype expected in the LazyInterface::make function pointer.
Note that such crates have the "cdylib" crate type and must be built as well as the application (the application won't build these crates because they are not part of its dependencies).

Finally, lazy_app/src/main.rs could be like this

/*
[dependencies]
lazy_interface = { path = "../lazy_interface" }
*/

use lazy_interface::LazyInterface;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    for lib in [
        "../lazy_impl_a/target/debug/liblazy_impl_a.so",
        "../lazy_impl_b/target/debug/liblazy_impl_b.so",
    ] {
        match LazyInterface::load(lib) {
            Ok(itf) => {
                let mut lazy = itf.make(&["123", "hello"])?;
                lazy.do_something();
                println!("~~> {}", lazy.show_something());
            }
            Err(e) => {
                println!("{}", e);
            }
        }
    }
    Ok(())
}
/*
~~> LazyA with 128 and "hello\nchanged 123 into 128"
~~> LazyB with ["123", "hello", "element 2"]
*/

Only the resources of lazy_interface are known, and the implementations are loaded only if they are available.

Note that all of this relies on a bunch of unsafe invocations.
It is very easy to introduce bugs.
For example, if the prototype of the make_lazy function exposed in the implementation does not match exactly what is expected in the LazyInterface::make function pointer, anything can happen.
Another terrible thing could happen if the LazyInterface is dropped in the application while some resources it created (directly or indirectly) are still in use.
This example is just an idea; it should be double-checked in order to detect anything that is or could become unsound.