Is there a way to run a test using XCTest multiple times just with different values?

Question

I'm currently working on a project trying to determine how long different sorting algorithms take to sort different sized arrays. To measure the time, I've decided to use XCTest in Swift Playgrounds since it can automate the process of running the algorithm multiple times and averaging it out. But I have an issue with this method because I have to test a large variety array sizes from 15 elements up to 1500 or so, at 5 element intervals (ie. 15 elements, 20 elements, 25 elements...).

The only way I've been able to do this with one test is multiple functions with the different size and measuring the performance. Here is a sample of what that looks like:

class insertionSortPerformanceTest: XCTestCase {
    func testMeasure10() {
        measure {
            _ = insertionSort(arrLen: 10)
        }
    }
    func testMeasure15() {
        measure {
            _ = insertionSort(arrLen: 15)
        }
    }
    func testMeasure20() {
        measure {
            _ = insertionSort(arrLen: 20)
        }
    }
}

insertionSort() works by generating an array of length arrLen and populating it with random numbers.

Is there a way to automate this process somehow?

Also, is there a way to take the output in the console and save it as a string so I can parse it for the relevant information later?

Answer 1

It's not exactly what you described but with the advent of Swift Macros, you can do very similar things in pure Swift. My colleague and I have developed a neat Swift Macro for parameterized XCTest. With this, you can simply use:

@Parametrize(input: [5, 100, 1500])
func testExample(input length: Int) {
  ...
}

The macro will create a copy of your method for a given parameter set. So, of course, it's not exactly your scenario but, data-driven testing is closer than before.

Check it out here: https://github.com/PGSSoft/XCTestParametrizedMacro

Answer 2

XCTest doesn't have built-in support for parameterized tests in the way that some other testing frameworks do, but you can simulate this feature with loops within your test functions.

Here is an example that iterates through an array of array lengths and calls insertionSort(arrLen:) for each one:

class insertionSortPerformanceTest: XCTestCase {
    func testInsertionSortPerformance() {
        let arrayLengths = Array(stride(from: 15, through: 1500, by: 5))
        
        for arrLen in arrayLengths {
            measureMetrics([.wallClockTime], automaticallyStartMeasuring: false) {
                startMeasuring()
                _ = insertionSort(arrLen: arrLen)
                stopMeasuring()
            }
        }
    }
}

The use of automaticallyStartMeasuring: false allows you to manually control the measuring process. You start measuring with startMeasuring() and stop it with stopMeasuring().

Capture Console Output Capturing the console output is a bit trickier because XCTest itself doesn't provide a direct way to save test logs to a string. However, there are ways to capture stdout and stderr in Swift that you can use to collect these logs.

Here's an example function to capture the standard output temporarily:

func captureStandardOutput(closure: () -> ()) -> String {
    let originalStdout = dup(STDOUT_FILENO)
    let pipe = Pipe()
    dup2(pipe.fileHandleForWriting.fileDescriptor, STDOUT_FILENO)

    closure()

    let data = pipe.fileHandleForReading.readDataToEndOfFile()
    if let output = String(data: data, encoding: .utf8) {
        dup2(originalStdout, STDOUT_FILENO)
        close(originalStdout)
        return output
    }
    return ""
}

You can use this function like this:

class insertionSortPerformanceTest: XCTestCase {
    func testInsertionSortPerformance() {
        // Your code
        let output = captureStandardOutput {
            // Your test logic here
        }
        print("Captured output: \(output)")
    }
}

Just be cautious when using this approach, as it involves changing the file descriptors for stdout, which can be risky in a multi-threaded environment. Always restore the original stdout file descriptor after capturing the output to avoid affecting other parts of your application.

You can then parse the output string to extract the information you want to collect.