news

[Published in Open Source For You (OSFY) magazine, January 2015 edition.]

In this article we shall cover testing of Haskell programs.

HUnit is a unit testing framework available for Haskell. It is similar to JUnit, which is used for the Java programming language. You can install HUnit on Fedora using the following command:

$ sudo yum install ghc-HUnit-devel

Consider a simple example that follows:

import Test.HUnit

test1 = TestCase $ assertEqual "Test equality" 3 (2 + 1)

The TestCase is a constructor defined in the Test data type. The definition is as follows:

-- Test Definition
-- ===============

-- | The basic structure used to create an annotated tree of test cases.
data Test
    -- | A single, independent test case composed.
    = TestCase Assertion
    -- | A set of @Test@s sharing the same level in the hierarchy. 
    | TestList [Test]
    -- | A name or description for a subtree of the @Test@s.
    | TestLabel String Test

On executing the above code with GHCi, you get the following output:

$ ghci test.hs 

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( one.hs, interpreted )
Ok, modules loaded: Main.

ghci> runTestTT test1

Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package HUnit-1.2.5.2 ... linking ... done.
Cases: 1  Tried: 1  Errors: 0  Failures: 0
Counts {cases = 1, tried = 1, errors = 0, failures = 0}

You can build a test suite of tests with TestList as shown below:

import Test.HUnit

test1 = TestList [ "Test addition" ~: 3 ~=? (2 + 1)
                 , "Test subtraction" ~: 3 ~=? (4 - 1)
                 ]

The ‘~=?’ operation is shorthand to assert equality. Its definition is as follows:

-- | Shorthand for a test case that asserts equality (with the expected 
--   value on the left-hand side, and the actual value on the right-hand
--   side).
(~=?) :: (Eq a, Show a) => a     -- ^ The expected value 
                        -> a     -- ^ The actual value
                        -> Test

The ‘~:’ operation is shorthand to attach a label to a test. Its definition is shown below:

(~:) :: (Testable t) => String -> t -> Test
label ~: t = TestLabel label (test t)

By compiling and executing the above code with GHCi, you get the following result:

ghci> runTestTT test1

Cases: 2  Tried: 2  Errors: 0  Failures: 0
Counts {cases = 2, tried = 2, errors = 0, failures = 0}

A failure is reported when there is a mismatch between the expected result and the observed value. For example:

import Test.HUnit

test1 = TestList [ "Test addition" ~: 3 ~=? (2 + 1)
                 , "Test subtraction" ~: 3 ~=? (4 - 2)
                 ]

Running the above code reports the failure in the output:

ghci> runTestTT test1

### Failure in: 1:Test subtraction
expected: 3
 but got: 2
Cases: 2  Tried: 2  Errors: 0  Failures: 1
Counts {cases = 2, tried = 2, errors = 0, failures = 1}

If our test case definition is incorrect, the compiler will throw an error during compilation time itself! For example:

import Data.Char
import Test.HUnit

test1 = TestList [ "Test addition" ~: 3 ~=? (2 + 1)
                 , "Test case" ~: "EARTH" ~=? (map "earth")
                 ]

On compiling the above code, you get the following output:

ghci> :l error.hs

[1 of 1] Compiling Main             ( error.hs, interpreted )

error.hs:5:48:
    Couldn't match expected type `[Char]'
                with actual type `[a1] -> [b0]'
    In the return type of a call of `map'
    Probable cause: `map' is applied to too few arguments
    In the second argument of `(~=?)', namely `(map "earth")'
    In the second argument of `(~:)', namely
      `"EARTH" ~=? (map "earth")'

error.hs:5:52:
    Couldn't match expected type `a1 -> b0' with actual type `[Char]'
    In the first argument of `map', namely `"earth"'
    In the second argument of `(~=?)', namely `(map "earth")'
    In the second argument of `(~:)', namely
      `"EARTH" ~=? (map "earth")'
Failed, modules loaded: none.

The correct version of the code and its output are shown below:

import Data.Char
import Test.HUnit

test1 = TestList [ "Test addition" ~: 3 ~=? (2 + 1)
                 , "Test case" ~: "EARTH" ~=? (map toUpper "earth")
                 ]

The expected test output is as follows:

ghci> runTestTT test1

Cases: 2  Tried: 2  Errors: 0  Failures: 0
Counts {cases = 2, tried = 2, errors = 0, failures = 0}

In Haskell, one needs to check for an empty list when using the head function, or else it will throw an error. An example test is shown below:

import Test.HUnit

test1 = TestCase $ assertEqual "Head of emptylist" 1 (head [])

Executing the above code gives:

ghci> runTestTT test1

### Error:
Prelude.head: empty list
Cases: 1  Tried: 1  Errors: 1  Failures: 0
Counts {cases = 1, tried = 1, errors = 1, failures = 0}

Other than the assertEqual function, there are conditional assertion functions like assertBool, assertString and assertFailure that you can use. The assertBool function takes a string that is displayed if the assertion fails and a condition to assert. A couple of examples are shown below:

import Test.HUnit

test1 = TestCase $ assertBool "Does not happen" True

Executing the above code in GHCi, gives you the following output:

ghci> runTestTT test1

Cases: 1  Tried: 1  Errors: 0  Failures: 0
Counts {cases = 1, tried = 1, errors = 0, failures = 0}

Consider the case when there is a failure:

import Test.HUnit

test1 = TestCase $ assertBool "Failure!" (False && False)

The corresponding output is shown below:

ghci> runTestTT test1

### Failure:
Failure!
Cases: 1  Tried: 1  Errors: 0  Failures: 1
Counts {cases = 1, tried = 1, errors = 0, failures = 1}

assertFailure and assertString functions take a string as input, and return an Assertion. These are used as part of other test functions. The assertFailure function is used in the definition of the assertBool function as shown below:

-- | Asserts that the specified condition holds.
assertBool :: String    -- ^ The message that is displayed if the assertion fails
           -> Bool      -- ^ The condition
           -> Assertion
assertBool msg b = unless b (assertFailure msg)

You can also use assertString for handling a specific case. For example:

import Test.HUnit

test1 = TestCase $ assertString "Failure!"

Executing the above code in GHCi results in the following failure message:

ghci> runTestTT test1

### Failure:
Failure!
Cases: 1  Tried: 1  Errors: 0  Failures: 1
Counts {cases = 1, tried = 1, errors = 0, failures = 1}

Hspec is another testing framework for Haskell, similar to Ruby’s RSpec. You can install it on any GNU/Linux distribution using the Cabal tool:

$ cabal update && cabal install hspec hspec-contrib

A simple example is shown below:

import Test.Hspec

main :: IO ()
main = hspec $ do
  describe "Testing equality" $ do
    it "returns 3 for the sum of 2 and 1" $ do
      3 `shouldBe` (2 + 1)

Executing the above code with GHCi produces the following verbose output:

$ ghci spec.hs 

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( spec.hs, interpreted )
Ok, modules loaded: Main.

ghci> main
Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package HUnit-1.2.5.2 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package random-1.0.1.1 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package stm-2.4.2 ... linking ... done.
Loading package bytestring-0.10.0.2 ... linking ... done.
Loading package unix-2.6.0.1 ... linking ... done.
Loading package containers-0.5.0.0 ... linking ... done.
Loading package pretty-1.1.1.0 ... linking ... done.
Loading package template-haskell ... linking ... done.
Loading package QuickCheck-2.6 ... linking ... done.
Loading package ansi-terminal-0.6.2.1 ... linking ... done.
Loading package async-2.0.1.6 ... linking ... done.
Loading package hspec-expectations-0.6.1 ... linking ... done.
Loading package quickcheck-io-0.1.1 ... linking ... done.
Loading package setenv-0.1.1.1 ... linking ... done.
Loading package primitive-0.5.0.1 ... linking ... done.
Loading package tf-random-0.5 ... linking ... done.
Loading package hspec-core-2.0.2 ... linking ... done.
Loading package hspec-discover-2.0.2 ... linking ... done.
Loading package hspec-2.0.2 ... linking ... done.

Testing equality
  returns 3 for the sum of 2 and 1

Finished in 0.0002 seconds
1 example, 0 failures

The keywords describe and it are used to specify the tests. The context keyword can also be used as an alias for describe. A particular test can be marked ‘pending’ using the pending and pendingWith keywords. The pendingWith function takes a string message as an argument, as illustrated below:

import Test.Hspec

main = hspec $ do
  describe "Testing equality" $ do
    it "returns 3 for the sum of 2 and 1" $ do
      3 `shouldBe` (2 + 1)

  describe "Testing subtraction" $ do
    it "returns 3 when subtracting 1 from 4" $ do
      pendingWith "need to add test"

The corresponding output for the above code snippet is provided below:

ghci> main

Testing equality
  returns 3 for the sum of 2 and 1
Testing subtraction
  returns 3 when subtracting 1 from 4
     # PENDING: need to add test

Finished in 0.0006 seconds
2 examples, 0 failures, 1 pending

You can use the after and before keywords to specify setup and tear down functions before running a test. For example:

import Test.Hspec

getInt :: IO Int
getInt = do
  putStrLn "Enter number:"
  number <- readLn
  return number
 
afterPrint :: ActionWith Int
afterPrint 3 = print 3
 
main = hspec $ before getInt $ after afterPrint $ do
  describe "should be 3" $ do
    it "should successfully return 3" $ \n -> do
      n `shouldBe` 3

Executing the above code yields the following output:

ghci> main

should be 3
Enter number:
3
3
  should successfully return 3

Finished in 0.6330 seconds
1 example, 0 failures

The different options used with Hspec can be listed with the –help option, as follows:

$ runhaskell file.hs --help
Usage: spec.hs [OPTION]...

OPTIONS
                --help              display this help and exit
  -m PATTERN    --match=PATTERN     only run examples that match given PATTERN
                --color             colorize the output
                --no-color          do not colorize the output
  -f FORMATTER  --format=FORMATTER  use a custom formatter; this can be one of:
                                       specdoc
                                       progress
                                       failed-examples
                                       silent
  -o FILE       --out=FILE          write output to a file instead of STDOUT
                --depth=N           maximum depth of generated test values for
                                    SmallCheck properties
  -a N          --qc-max-success=N  maximum number of successful tests before a
                                    QuickCheck property succeeds
                --qc-max-size=N     size to use for the biggest test cases
                --qc-max-discard=N  maximum number of discarded tests per
                                    successful test before giving up
                --seed=N            used seed for QuickCheck properties
                --print-cpu-time    include used CPU time in summary
                --dry-run           pretend that everything passed; don't verify
                                    anything
                --fail-fast         abort on first failure
  -r            --rerun             rerun all examples that failed in the
                                    previously test run (only works in GHCi)

Other than the shouldBe expectation, you can also use assertions like shouldReturn, shouldSatisfy, and shouldThrow. Examples of each are given below:

import Test.Hspec
import Control.Exception (evaluate)

main = hspec $ do
  describe "shouldReturn" $ do
    it "should successfully return 3" $ do
      return 3 `shouldReturn` 3

  describe "shouldSatisfy" $ do
    it "should satisfy the condition that 10 is greater than 5" $ do
      10 `shouldSatisfy` (> 5)

  describe "shouldThrow" $ do
    it "should throw an exception when taking head of an empty list" $ do
      evaluate (1 `div` 0) `shouldThrow` anyException                                     

Executing the above code provides the following output:

ghci> main

shouldReturn
  should successfully return an Integer
shouldSatisfy
  should satisfy the condition that 10 is greater than 5
shouldThrow
  should throw an exception when taking head of an empty list

Finished in 0.0015 seconds
3 examples, 0 failures

The evaluate function can be used to check for exceptions. Its type signature is as follows:

ghci> :t evaluate
evaluate :: a -> IO a

You can also re-use HUnit tests and integrate them with Hspec. An example is shown below:

import Test.HUnit
import Test.Hspec
import Test.Hspec.Contrib.HUnit (fromHUnitTest)

test1 = TestList [ TestLabel "Test subtraction" foo ]

foo :: Test
foo = TestCase $ do
  3 @?= (4 - 1)

main = hspec $ do
  describe "Testing equality" $ do
    it "returns 3 for the sum of 2 and 1" $ do
      3 `shouldBe` (2 + 1)

  describe "Testing subtraction" $ do
    fromHUnitTest test1

Testing the code in GHCi produces the following output:

ghci> main
Loading package hspec-contrib-0.2.0 ... linking ... done.

Testing equality
  returns 3 for the sum of 2 and 1
Testing subtraction
  Test subtraction

Finished in 0.0004 seconds
2 examples, 0 failures

The Hspec tests can also be executed in parallel. For example, computing the Fibonacci for a set of numbers, and asserting their expected values can be run in parallel as shown below:

import Test.Hspec

fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)

main = hspec $ parallel $ do
  describe "Testing Fibonacci" $ do
    it "must return 6765 for fib 20" $ do
      fib 20 `shouldBe` 6765

    it "must return 75025 for fib 25" $ do
      fib 25 `shouldBe` 75025

    it "must return 832040 for fib 30" $ do
      fib 30 `shouldBe` 832040

    it "must return 9227465 for fib 35" $ do
      fib 35 `shouldBe` 9227465

You can compile and execute the above code as shown below:

$ ghc -threaded parallel.hs 
Linking parallel ...

$ ./parallel +RTS -N -RTS

Testing Fibonacci
  must return 6765 for fib 20
  must return 75025 for fib 25
  must return 832040 for fib 30
  must return 9227465 for fib 35

Finished in 0.9338 seconds
4 examples, 0 failures