How to get a Swift Package to be build with the Release Build Configuration of the current Scheme in Xcode?

Question

I have moved some of my code from my main application to a Swift Package. But I noticed a slow down.

I have set the Build Configuration to Release in the scheme configuration, but it does not seem to be used when building the package.

So the same code in the main application will run much faster than in the package because it's optimised.

What could I be doing wrong? What configuration should I check?

I am using the Xcode 14.0.1 and I put a very small example project in github