How to wrap a Linux library to be used in Wine? I keep getting BAD_IMAGE_FORMAT

Question

I have a Windows-only game that can be extended by plugins and a Linux library I would like to use. The application (Unity game) runs fine under Wine/Proton. I've found that Winelib can do this, and I was following the User's Guide, notably the 5th section which mentions my exact use case.

After some attempts (like removing --single-target arg from winemaker), I got to the point where the Proton debug log keeps showing c000007b which seems to be STATUS_BAD_IMAGE_FORMAT.

Note that I renamed the Linux library (and linked against that), to avoid a potential conflict between that and my wrapper.

I only have a 64-bit version of the library and it is called from C# code using the cdecl calling convention. This is what I've tried so far (and combinations of these):

• Added a .spec file with contents (first without the -arch and with different ordinals):

1 cdecl -arch=x86_64 LibraryFunc (long ptr ptr) LibraryFuncWine

• Added the library's header and a .c file:

#include "library_header.h"
#include <windef.h> /* Part of the Wine header files */

enum ELibResult WINAPI LibraryFuncWine(Version version, struct CreateParams* params, struct ICore** result)
{
    enum ELibResult ret = LibraryFunc(version, params, result);

    return ret;
}

• Used winemaker as winemaker --nosource-fix --dll --nomfc -I. -L. -llinux_library . then make
• Copied the .dll.so file all around the game
• Copied the .dll.so file to steamapps/common/Proton 5.0/dist/lib64/wine/ and the native library to one level higher (lib64/) and removed the others
• Also tried to copy to lib/ instead of lib64/
• Added a dll override for the file and set it to builtin
• Removed the referenced DLLs (odbc32, ole32, oleaut32, winspool, odbccp32) and libraries (uuid) from the makefile
• Add --mno-cygwin to compiler flags
• Add -m64 and -fPIC to compiler/linker flags (based on Wine spec files)
• I've also tried making a .def file and using that
• Used winemaker again with the --nomsvcrt argument added and ran make (the header file includes string.h)

From the Proton log (+module, the others didn't provide any more info for me):

00dc:trace:module:load_dll looking for L"Z:\\D\\a\\library\\path\\library_name" in L"Z:\\D\\gamepath;C:\\windows\\system32;C:\\windows\\system;C:\\windows;C:\\Program Files (x86)\\Steam;.;C:\\windows\\system32;C:\\windows;C:\\windows\\system32\\wbem;C:\\windows\\system32\\WindowsPowershell\\v1.0"
00dc:trace:module:get_load_order looking for L"Z:\\D\\a\\library\\path\\library_name.dll"
00dc:trace:module:get_load_order_value got standard key b for L"library_name"
00dc:trace:module:load_builtin_dll Trying built-in L"library_name.dll"
00dc:trace:module:load_so_dll loading L"\\??\\Z:\\D\\a\\library\\path\\library_name.dll" from so lib "/D/Games/SteamLibrary/steamapps/common/Proton 5.0/dist/bin/../lib64/wine/library_name.dll.so"
00dc:warn:module:load_dll Failed to load module L"Z:\\D\\a\\library\\path\\library_name"; status=c000007b

The same is repeated with and without .dll and starting with lib (library_name.dll, liblibrary_name and liblibrary_name.dll).

I was unable to get any more logs on what exactly goes wrong besides that error code.

On the C# side (ran by Unity in Proton) it results in a DllNotFoundException.

Wine version (winebuild, winegcc): 5.9 (staging) - gcc 9.0.1

Proton version: 5.0 - wine-5.0-603-g068dee4

Answer 1

There may be a solution for your binary only .so using RPC.

But, since many/most linux libraries have source code, what is the specific library [that doesn't]? Is there an alternate library that is similar that does? There may be source code for your binary library if you look hard enough for it.

However, I'm going to assume that you do have to use the binary linux library.

But, this may involve more work than you'd be willing to do. So, how badly do you want this? ;-)

Is this for production or just personal use? I'm assuming just for personal use because you're trying to get the WinX game to work under wine.

How big is the .so in terms of the number of public API functions [that your wine program needs to call]? How many different structs are involved?

You may need to create glue/interface routines for each API call and struct.

The basic method is to use [some sort of] RPC [remote procedure call] mechanism (e.g.) https://en.wikipedia.org/wiki/Remote_procedure_call

You create a linux server that incorporates the desired [linux only] library. You write wrapper routines that convert RPC calls into the library's API calls.

In the game, your plugin would issue RPC calls that get sent to the server, the server does the processing, and sends back the results.

There are some tools that can help you with that (e.g. rpcgen)

Normally, RPC is done over a network/socket connection. For your purposes, this could be over localhost or using an AF_UNIX connection. Maybe that's fast enough for your purpose.

Otherwise, you may have to establish a shared memory buffer pool and add that as an adjunct to the RPC calls.

---

UPDATE:

Thanks, but can you provide more information on why it's not doable using Winelib? In the guide it says I can link a Linux library to mine I think.

It may be possible to write some glue functions. Doing so may be problematic. But, you may be able to bypass [all] this, based on what I found in the Discord SDK. See below.

---

The ABI differences: In WinX, the first four arguments are passed in regs and, in linux, the first six arguments are passed in regs. The registers are different. This can be handled with a "thunk" routine that establishes a new stack frame and moves the values to the correct registers for linux. See: https://en.wikipedia.org/wiki/X86_calling_conventions

But, AFAICT, on my machine at least, wine ultimately goes to /usr/bin/wine32 which is a 32 bit executable. But, your linux lib is 64 bits. That issue can't be solved easily. See: Is it possible to use both 64 bit and 32 bit instructions in the same executable in 64 bit Linux? Personally, if I had to do that, I'd go the RPC route.

---

The library exports 2 functions, I'm only interested in 1, the rest of the functionality is via function pointers (I'm not sure these would work over wine but not there yet). The lib is actually Discord's Game SDK so there's no replacement for it, however this is just a hobby thing.

I just downloaded discord_game_sdk.zip from the website and unzipped it. It has source code example programs. And, the requisite .h files.

There is a lib/x86 subdirectory that has: discord_game_sdk.dll and discord_game_sdk.dll.lib which are 32 bit PE format files. And, there is a lib/x86_64 subdir as well [which has a .so]

So, instead of the linux .so, can you link to those PE format files directly instead? (e.g.) They would probably be loadable with LoadLibrary

---

I could use sockets if there is no direct way (however wouldn't creating a unix socket have the same issues? I can definitely use localhost though).

Then, consider this a fallback position.

PF_INET socket calls to localhost are quite fast, and [with some "trickery"] you can establish shared memory space between your plugin [under wine] and a server program [which you create] that has the linux .so linked into it.