Mutation Analysis

Limits to Fuzzing

Mutation testing is different from regular code coverage because code coverage only shows which parts of a program are tested. It does not tell us if the program gives the correct results. For example, we could have 100% code coverage and still have bugs that our tests do not catch. This happens because tests may check whether the program runs without errors instead of checking whether the results are correct. Code coverage is still useful, but it does not tell us everything we need to know. Mutation testing helps us find weaknesses in our tests by making small changes to the code and checking if our tests catch them. This helps developers find hidden bugs and feel more confident in their code.

Injecting Artificial Faults

Mutation analysis is a way to check how well our tests find bugs in a program. It works by making small changes to the code and checking if the tests catch them. For example, we could change a + to a -. If the tests do not catch the change, it means they may need to be improved. The mutation score tells us how many changes the tests were able to catch.

Mutation testing helps us understand if our tests can find real bugs. It can be used with regular tests and other testing tools. It is similar to fuzzing because both help find problems in software, but mutation testing focuses on checking how good our tests are.

Adding errors to code by hand can take a lot of time, and developers might miss some important problems. Mutation analysis makes this easier by creating small changes to the code automatically. Each changed version is called a mutant. If the tests catch the change, the mutant is called killed. If the tests do not catch it, the mutant survives. This helps developers see where their tests need improvement.

Here is MuProgramAnalyzer that runs tests for an entire program used in the FB:

class TestGCD(unittest.TestCase):
>>>     def test_simple(self):
>>>         assert cfg.gcd(1, 0) == 1
>>>
>>>     def test_mirror(self):
>>>         assert cfg.gcd(0, 1) == 1
>>> for mutant in MuProgramAnalyzer('gcd', gcd_src):
>>>     mutant[test_module].runTest('TestGCD')
>>> mutant.pm.score()
======================================================================
FAIL: test_simple (__main__.TestGCD)
----------------------------------------------------------------------

1. Creating the Tests

class TestGCD(unittest.TestCase):
    def test_simple(self):
        assert cfg.gcd(1, 0) == 1

    def test_mirror(self):
        assert cfg.gcd(0, 1) == 1

• class TestGCD(unittest.TestCase): creates a group of tests for the GCD function.
• def test_simple(self): creates the first test.
• assert cfg.gcd(1, 0) == 1 runs the GCD function with 1 and 0 and checks that the answer is 1.
• def test_mirror(self): creates a second test.
• assert cfg.gcd(0, 1) == 1 checks that the GCD function returns 1 when the numbers are reversed.

These tests check whether the function gives the correct answers.

2. Running Mutation Analysis

for mutant in MuProgramAnalyzer('gcd', gcd_src):
    mutant[test_module].runTest('TestGCD')

• MuProgramAnalyzer('gcd', gcd_src) takes the name of the program (gcd) and its source code (gcd_src) and creates changed versions of the program.
• for mutant in ... goes through each changed version, called a mutant.
• mutant[test_module] accesses the test module for that mutant.
• .runTest('TestGCD') runs the tests in TestGCD against the changed program.

This lets us check whether the tests can catch mistakes caused by changes to the code.

3. Calculating the Score

mutant.pm.score()

This calculates the mutation score, which tells us how many mutants were caught by the tests compared with the total number counted in the score.

4. Understanding the Error Message

FAIL: test_simple
AssertionError

Ran 1 test
FAILED (failures=1)

• FAIL: test_simple means the first test failed.
• AssertionError means the result was not equal to the expected answer, 1.
• Ran 1 test means one test ran during this execution.
• FAILED (failures=1) means that test did not pass.

The test failed because the modified program did not give the expected result. This shows how mutation analysis checks whether tests can detect problems in changed code.