SwiftCheck

QuickCheck for Swift

Code Quality Rank: L1

Programming language: Swift

License: MIT License

Tags: Testing    

Latest version: v0.12.0

README

SwiftCheck

QuickCheck for Swift.

For those already familiar with the Haskell library, check out the source. For everybody else, see the [Tutorial Playground](Tutorial.playground) for a beginner-level introduction to the major concepts and use-cases of this library.

Introduction

SwiftCheck is a testing library that automatically generates random data for testing of program properties. A property is a particular facet of an algorithm or data structure that must be invariant under a given set of input data, basically an XCTAssert on steroids. Where before all we could do was define methods prefixed by test and assert, SwiftCheck allows program properties and tests to be treated like data.

To define a program property the forAll quantifier is used with a type signature like (A, B, C, ... Z) -> Testable where A : Arbitrary, B : Arbitrary ... Z : Arbitrary. SwiftCheck implements the Arbitrary protocol for most Swift Standard Library types and implements the Testable protocol for Bool and several other related types. For example, if we wanted to test the property that every Integer is equal to itself, we would express it as such:

func testAll() {
    // 'property' notation allows us to name our tests.  This becomes important
    // when they fail and SwiftCheck reports it in the console.
    property("Integer Equality is Reflexive") <- forAll { (i : Int) in
        return i == i
    }
}

For a less contrived example, here is a program property that tests whether Array identity holds under double reversal:

property("The reverse of the reverse of an array is that array") <- forAll { (xs : [Int]) in
    // This property is using a number of SwiftCheck's more interesting 
    // features.  `^&&^` is the conjunction operator for properties that turns
    // both properties into a larger property that only holds when both sub-properties
    // hold.  `<?>` is the labelling operator allowing us to name each sub-part
    // in output generated by SwiftCheck.  For example, this property reports:
    //
    // *** Passed 100 tests
    // (100% , Right identity, Left identity)
    return
        (xs.reversed().reversed() == xs) <?> "Left identity"
        ^&&^
        (xs == xs.reversed().reversed()) <?> "Right identity"
}

Because SwiftCheck doesn't require tests to return Bool, just Testable, we can produce tests for complex properties with ease:

property("Shrunken lists of integers always contain [] or [0]") <- forAll { (l : [Int]) in
    // Here we use the Implication Operator `==>` to define a precondition for
    // this test.  If the precondition fails the test is discarded.  If it holds
    // the test proceeds.
    return (!l.isEmpty && l != [0]) ==> {
        let ls = self.shrinkArbitrary(l)
        return (ls.filter({ $0 == [] || $0 == [0] }).count >= 1)
    }
}

Properties can even depend on other properties:

property("Gen.one(of:) multiple generators picks only given generators") <- forAll { (n1 : Int, n2 : Int) in
    let g1 = Gen.pure(n1)
    let g2 = Gen.pure(n2)
    // Here we give `forAll` an explicit generator.  Before SwiftCheck was using
    // the types of variables involved in the property to create an implicit
    // Generator behind the scenes.
    return forAll(Gen.one(of: [g1, g2])) { $0 == n1 || $0 == n2 }
}

All you have to figure out is what to test. SwiftCheck will handle the rest.

Shrinking

What makes QuickCheck unique is the notion of shrinking test cases. When fuzz testing with arbitrary data, rather than simply halt on a failing test, SwiftCheck will begin whittling the data that causes the test to fail down to a minimal counterexample.

For example, the following function uses the Sieve of Eratosthenes to generate a list of primes less than some n:

/// The Sieve of Eratosthenes:
///
/// To find all the prime numbers less than or equal to a given integer n:
///    - let l = [2...n]
///    - let p = 2
///    - for i in [(2 * p) through n by p] {
///          mark l[i]
///      }
///    - Remaining indices of unmarked numbers are primes
func sieve(_ n : Int) -> [Int] {
    if n <= 1 {
        return []
    }

    var marked : [Bool] = (0...n).map { _ in false }
    marked[0] = true
    marked[1] = true

    for p in 2..<n {
        for i in stride(from: 2 * p, to: n, by: p) {
            marked[i] = true
        }
    }

    var primes : [Int] = []
    for (t, i) in zip(marked, 0...n) {
        if !t {
            primes.append(i)
        }
    }
    return primes
}

/// Short and sweet check if a number is prime by enumerating from 2...⌈√(x)⌉ and checking 
/// for a nonzero modulus.
func isPrime(n : Int) -> Bool {
    if n == 0 || n == 1 {
        return false
    } else if n == 2 {
        return true
    }

    let max = Int(ceil(sqrt(Double(n))))
    for i in 2...max {
        if n % i == 0 {
            return false
        }
    }
    return true
}

We would like to test whether our sieve works properly, so we run it through SwiftCheck with the following property:

import SwiftCheck

property("All Prime") <- forAll { (n : Int) in
    return sieve(n).filter(isPrime) == sieve(n)
}

Which produces the following in our testing log:

Test Case '-[SwiftCheckTests.PrimeSpec testAll]' started.
*** Failed! Falsifiable (after 10 tests):
4

Indicating that our sieve has failed on the input number 4. A quick look back at the comments describing the sieve reveals the mistake immediately:

- for i in stride(from: 2 * p, to: n, by: p) {
+ for i in stride(from: 2 * p, through: n, by: p) {

Running SwiftCheck again reports a successful sieve of all 100 random cases:

*** Passed 100 tests

Custom Types

SwiftCheck implements random generation for most of the types in the Swift Standard Library. Any custom types that wish to take part in testing must conform to the included Arbitrary protocol. For the majority of types, this means providing a custom means of generating random data and shrinking down to an empty array.

For example:

import SwiftCheck

public struct ArbitraryFoo {
    let x : Int
    let y : Int

    public var description : String {
        return "Arbitrary Foo!"
    }
}

extension ArbitraryFoo : Arbitrary {
    public static var arbitrary : Gen<ArbitraryFoo> {
        return Gen<(Int, Int)>.zip(Int.arbitrary, Int.arbitrary).map(ArbitraryFoo.init)
    }
}

class SimpleSpec : XCTestCase {
    func testAll() {
        property("ArbitraryFoo Properties are Reflexive") <- forAll { (i : ArbitraryFoo) in
            return i.x == i.x && i.y == i.y
        }
    }
}

There's also a Gen.compose method which allows you to procedurally compose values from multiple generators to construct instances of a type:

public static var arbitrary : Gen<MyClass> {
    return Gen<MyClass>.compose { c in
        return MyClass(
            // Use the nullary method to get an `arbitrary` value.
            a: c.generate(),

            // or pass a custom generator
            b: c.generate(Bool.suchThat { $0 == false }),

            // .. and so on, for as many values and types as you need.
            c: c.generate(), ...
        )
    }
}

Gen.compose can also be used with types that can only be customized with setters:

public struct ArbitraryMutableFoo : Arbitrary {
    var a: Int8
    var b: Int16

    public init() {
        a = 0
        b = 0
    }

    public static var arbitrary: Gen<ArbitraryMutableFoo> {
        return Gen.compose { c in
            var foo = ArbitraryMutableFoo()
            foo.a = c.generate()
            foo.b = c.generate()
            return foo
        }
    }
}

For everything else, SwiftCheck defines a number of combinators to make working with custom generators as simple as possible:

let onlyEven = Int.arbitrary.suchThat { $0 % 2 == 0 }

let vowels = Gen.fromElements(of: [ "A", "E", "I", "O", "U" ])

let randomHexValue = Gen<UInt>.choose((0, 15))

let uppers = Gen<Character>.fromElements(in: "A"..."Z")
let lowers = Gen<Character>.fromElements(in: "a"..."z")
let numbers = Gen<Character>.fromElements(in: "0"..."9")

/// This generator will generate `.none` 1/4 of the time and an arbitrary
/// `.some` 3/4 of the time
let weightedOptionals = Gen<Int?>.frequency([
    (1, Gen<Int?>.pure(nil)),
    (3, Int.arbitrary.map(Optional.some))
])

For instances of many complex or "real world" generators, see [ComplexSpec.swift](Tests/SwiftCheckTests/ComplexSpec.swift).

System Requirements

SwiftCheck supports OS X 10.9+ and iOS 7.0+.

Setup

SwiftCheck can be included one of two ways:

Using The Swift Package Manager

• Add SwiftCheck to your Package.swift file's dependencies section:

.package(url: "https://github.com/typelift/SwiftCheck.git", from: "0.8.1")

Using Carthage

• Add SwiftCheck to your Cartfile
• Run carthage update
• Drag the relevant copy of SwiftCheck into your project.
• Expand the Link Binary With Libraries phase
• Click the + and add SwiftCheck
• Click the + at the top left corner to add a Copy Files build phase
• Set the directory to Frameworks
• Click the + and add SwiftCheck

Using CocoaPods

• Add our Pod to your podfile.
• Run $ pod install in your project directory.

Framework

• Drag SwiftCheck.xcodeproj into your project tree as a subproject
• Under your project's Build Phases, expand Target Dependencies
• Click the + and add SwiftCheck
• Expand the Link Binary With Libraries phase
• Click the + and add SwiftCheck
• Click the + at the top left corner to add a Copy Files build phase
• Set the directory to Frameworks
• Click the + and add SwiftCheck

License

SwiftCheck is released under the MIT license.