Does XCTest methods generated dynamically by testInvocations work with xcodebuild's -only-testing?

Question

I have an app which needs to dynamically generate test methods in its XCTestCase subclass. I'm using +(NSArray<NSInvocation*>*)testInvocations to dynamically generate test methods at runtime with the following (dummy) names: example_test, permissions_location_test and permissions_many_test. Below is a simplified, ready-to-be-pasted-into-Xcode code.

@import XCTest;
@import ObjectiveC.runtime;

@interface ParametrizedTests : XCTestCase
@end

@implementation ParametrizedTests
+ (NSArray<NSInvocation *> *)testInvocations {
  NSLog(@"testInvocations() called");

  /* Prepare dummy input */
  __block NSMutableArray<NSString *> *dartTestFiles = [[NSMutableArray alloc] init];
  [dartTestFiles addObject:@"example_test"];
  [dartTestFiles addObject:@"permissions_location_test"];
  [dartTestFiles addObject:@"permissions_many_test"];

  NSMutableArray<NSInvocation *> *invocations = [[NSMutableArray alloc] init];

  NSLog(@"Before the loop, %lu elements in the array", (unsigned long)dartTestFiles.count);

  for (int i = 0; i < dartTestFiles.count; i++) {
    /* Step 1 */

    NSString *name = dartTestFiles[i];

    void (^anonymousFunc)(ParametrizedTests *) = ^(ParametrizedTests *instance) {
      NSLog(@"anonymousFunc called!");
    };

    IMP implementation = imp_implementationWithBlock(anonymousFunc);
    NSString *selectorStr = [NSString stringWithFormat:@"%@", name];
    SEL selector = NSSelectorFromString(selectorStr);
    class_addMethod(self, selector, implementation, "v@:");

    /* Step 2 */

    NSMethodSignature *signature = [self instanceMethodSignatureForSelector:selector];
    NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];
    invocation.selector = selector;

    NSLog(@"RunnerUITests.testInvocations(): selectorStr = %@", selectorStr);

    [invocations addObject:invocation];
  }

  NSLog(@"After the loop");

  return invocations;
}

@end

I can run all these tests at once using:

xcodebuild test \
  -scheme Landmarks \
  -destination 'platform=iOS Simulator,name=iPhone 15'

Excerpt from above command's stdout:

Test Suite 'Selected tests' passed at 2023-11-16 13:44:59.148.
     Executed 3 tests, with 0 failures (0 unexpected) in 0.246 (0.248) seconds

Problem

The problem I'm facing now is that I cannot select just one test to run using xcodebuild's -only-testing flag. For example:

xcodebuild test \
  -scheme Landmarks \
  -destination 'platform=iOS Simulator,name=iPhone 15' \
  -only-testing 'LandmarksUITests/ParametrizedTests/example_test'

does not work - no tests are executed:

Test Suite 'ParametrizedTests' passed at 2023-11-16 13:45:58.472.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds

I also tried doing:

xcodebuild test \
  -scheme Landmarks \
  -destination 'platform=iOS Simulator,name=iPhone 15' \
  -only-testing 'LandmarksUITests/ParametrizedTests/testInvocations'

but the result is the same.

So the question is: how can I select a subset of tests (that were generated dynamically at runtime using testInvocations) with the -only-testing option?. Is is even possible?

Answer 1

After some digging, I found out that XCTest calls tests differently when -only-testing MyTarget/MyClass/myTest is passed. More specifically, the XCTestCase.defaultTestSuite is not called when a single test is specified in -only-testing.

XCTest framework checks if it can run the test passed in -only-testing by sending a message to NSObject.instancesRespondToSelector:aSelector: (XCTestCase of course inherits from NSObject) and checking what it returns. This seemed like a good hook point to call defaultTestSuite manually, which in turn calls testInvocations, which generates and swizzles-in test methods to the ParametrizedTests class.

My LandmarksTests class was missing override of that selector. After I added it to my ParametrizedTests class:

+ (BOOL)instancesRespondToSelector:(SEL)aSelector {
  [self defaultTestSuite]; // calls testInvocations
  BOOL result = [super instancesRespondToSelector:aSelector];
  return true;
}

it started working fine!

Here's the final file that can be copy-pasted into a file ParametrizedTests.m that is inside the LandmarksUITests UI Test Target and it works fine!

@import XCTest;
@import ObjectiveC.runtime;

@interface ParametrizedTests : XCTestCase
@end

@implementation ParametrizedTests

+ (BOOL)instancesRespondToSelector:(SEL)aSelector {
  [self defaultTestSuite]; // calls testInvocations
  BOOL result = [super instancesRespondToSelector:aSelector];
  return true;
}

+ (NSArray<NSInvocation *> *)testInvocations {
  NSLog(@"testInvocations() called");

  /* Prepare dummy input */
  __block NSMutableArray<NSString *> *dartTestFiles = [[NSMutableArray alloc] init];
  [dartTestFiles addObject:@"example_test"];
  [dartTestFiles addObject:@"permissions_location_test"];
  [dartTestFiles addObject:@"permissions_many_test"];

  NSMutableArray<NSInvocation *> *invocations = [[NSMutableArray alloc] init];

  NSLog(@"Before the loop, %lu elements in the array", (unsigned long)dartTestFiles.count);

  for (int i = 0; i < dartTestFiles.count; i++) {
    /* Step 1 */

    NSString *name = dartTestFiles[i];

    void (^anonymousFunc)(ParametrizedTests *) = ^(ParametrizedTests *instance) {
      NSLog(@"anonymousFunc called!");
    };

    IMP implementation = imp_implementationWithBlock(anonymousFunc);
    NSString *selectorStr = [NSString stringWithFormat:@"%@", name];
    SEL selector = NSSelectorFromString(selectorStr);
    class_addMethod(self, selector, implementation, "v@:");

    /* Step 2 */

    NSMethodSignature *signature = [self instanceMethodSignatureForSelector:selector];
    NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];
    invocation.selector = selector;

    NSLog(@"RunnerUITests.testInvocations(): selectorStr = %@", selectorStr);

    [invocations addObject:invocation];
  }

  NSLog(@"After the loop");

  return invocations;
}

@end

See also:

• how Quick framework overrides this method