Should macOS driverkit system extensions be arm64 or arm64e for Apple Silicon / M1?

Question

I compile a macOS driverkit system extension as a Universal library so that it contains both x86_64 and arm64. One Apple Silicon computer A the driver starts when I attach the USB device. On Apple Silicon computer B I can see kernel: exec_mach_imgact: disallowing arm64 platform driverkit binary "com.example.driver", should be arm64e being printed in the Console.app when the USB device is attached. I've looked at the source code of where this is happening but I cannot figure out what the problem is. If I compile it for arm64e then it get exec_mach_imgact: not running binary "com.example.driver" built against preview arm64e on computer A, bit then it starts on computer B.

None of the computers have -arm64e_preview_abi set in the boot-args.

If I create a new Xcode (12.4) project on each machine and build Release then computer A and otool -fvv com.example.driver gives

Fat headers
fat_magic FAT_MAGIC
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    capabilities 0x0
    offset 16384
    size 73856
    align 2^14 (16384)
architecture arm64
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64_ALL
    capabilities 0x0
    offset 98304
    size 73856
    align 2^14 (16384)

On computer B the same command gives

Fat headers
fat_magic FAT_MAGIC
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    capabilities 0x0
    offset 16384
    size 73280
    align 2^14 (16384)
architecture arm64
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64_ALL
    capabilities 0x0
    offset 98304
    size 73296
    align 2^14 (16384)

How can I make the driver start on both machines?

Answer 1

Dexts should indeed be arm64 and x86_64 (but as pmdj explains, system binaries are still arm64e.)

As hinted by the name of (and need for) the -arm64e_preview_abi, arm64e is currently only exposed as a developer preview, to allow for testing.

However, you shouldn't get the disallowing arm64 error: did you set other interesting boot-args on computer B? (in particular, amfi= may be relevant)

Answer 2

Updated answer

We're now quite a few years down the road of DriverKit and Apple Silicon and I've shipped a few dexts since this question first showed up. Looks like I got my answer spectacularly wrong and didn't fix it when I had new information. Oops!

arm64 is definitely the correct architecture for developing and deploying dexts for macOS 11-13 and iPadOS 16.

Apple uses arm64e for its own "platform" dexts, but that doesn't have any bearing on third-party dexts.

There seems to be a bug in macOS which adds to the confusion: in certain setups, the system appears to expect dexts to be built as arm64e binaries. I've not tracked down the precise trigger, but my main suspect is turning off System Integrity Protection (SIP). It may also be related to the amfi_get_out_of_my_way=0x1 flag. In any case, turning on SIP (the default state) and not using any custom kernel boot-args, makes correctly signed and entitled arm64 ABI dexts work just fine.

Original answer

(I've left this because the first 4 comments below refer to it.)

My experience so far indicates that arm64e is the correct and only correct Apple Silicon architecture to use for dexts.

For one, there's the "disallowing arm64 platform" error, and also Apple's own DriverKit based drivers are built for arm64e:

% otool -fvv /System/Library/DriverExtensions/com.apple.AppleUserHIDDrivers.dext/com.apple.AppleUserHIDDrivers
Fat headers
fat_magic FAT_MAGIC
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    capabilities 0x0
    offset 16384
    size 96208
    align 2^14 (16384)
architecture arm64e
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64E
    capabilities CPU_SUBTYPE_ARM64E_PTRAUTH_VERSION 0
    offset 114688
    size 95312
    align 2^14 (16384)

That leaves the question of why your arm64e build isn't working. The "built against preview arm64e" error suggests the problem isn't with the computer but the binary. Are you using identical binaries on the 2 systems? Perhaps one has SIP disabled, so it's more permissive of badly built binaries?

Have you tried a "hello world" style dext, in a freshly created project on the latest version of Xcode? Check that runs natively on both machines. Once that's working, compare Xcode's compiler and linker command lines with those from your build script - or if you're also using Xcode, compare your target's build settings with the "clean" one.

Answer 3

I think the problem is "amfi_get_out_of_my_way=0x1",

My DEXT was once executed on MacOS12.4 Mac mini M1 arm64, and its boot-args was as follows: boot-args="-arm64e_preview_abi"

But now my boot-args are set to: boot-args="-arm64e_preview_abi amfi_get_out_of_my_way=0x1"

I try to compile my DEXT with Xcode1430 on MacOS1331 Mac mini M1 arm64, Then I get the same result as tuple_cat.

kernel: exec_mach_imgact: disallowing arm64 platform driverkit binary "com.example.driver"

I tried to compile my DEXT with Xcode1430 on MacOS1331, Mac mini x86_64 works fine. But Mac mini M1 arm64 doesn't work.