How to specify specific SSE level to differnet source files when compiling Apple universal binary?

Question

In our product, we have a DSP part that is:

• in x86, it is compiled with a series of different SIMD versions (at runtime it uses the supported version determined by cpuid);
• in arm64, it is compiled against armv8 neon.

The CMake code controlling the build looks like this:

if(CMAKE_SYSTEM_PROCESSOR STREQUAL "arm64")
    add_library(dsp_neon STATIC neon_impl.cpp)
else()
    add_library(dsp_sse3 STATIC sse3_impl.cpp)
    add_library(dsp_sse41 STATIC sse41_impl.cpp)
    add_library(dsp_avx1 STATIC avx1_impl.cpp)
    add_library(dsp_avx2 STATIC avx2_impl.cpp)
    target_compile_options(dsp_sse3 PRIVATE -msse3)
    target_compile_options(dsp_sse41 PRIVATE -msse4.1)
    target_compile_options(dsp_avx1 PRIVATE -mavx)
    target_compile_options(dsp_avx2 PRIVATE -mavx2 -mfma)
endif()

Now we are trying to create Mac universal binary by specifying CMAKE_OSX_ARCHITECTURES=arm64;x86_64, and the problem comes: it seems I should add all source files into compile, and let compiler to generate corresponding machine code. However it claims massive conflicts when compiling these platform-specific sources, because ARM target don't support Intel's SIMD header files (surely it won't).

I want to make clear that:

• What is the underlying process of compiling into universal binary? Is it actually compiled twice - one time in ARM and one time in x86? If so, should I use platform macros to mask out platform-specific codes?
• Is it still valid to specify -mxxx when compiling universal binaries?