Mutation Testing: Who Tests the Tests?

100% coverage, three different answers

Coverage tells us what code ran; it does not tell us what the tests checked.

The idea

By hand: two test runs on the one line you doubt — mutated, then restored. With a tool: the same loop, automated at breadth.

What gets mutated

Each mutant applies exactly one such change.

Worked example

def is_adult(age):
    return age >= 18

def test_is_adult():
    assert is_adult(30) is True
    assert is_adult(5) is False

The fix the survivors ask for

def test_is_adult_boundary():
    assert is_adult(18) is True
    assert is_adult(17) is False

With it added, every mutant in the table is killed.

Mutation score — read the definition

• A score means something only alongside its operator set and mutant count
• A useful signal of test strength, but a poor target
• The real information is in the individual survivors

Why it works

• Competent programmer hypothesis — real code is usually nearly right, so real bugs tend to be small deviations from a correct program
• Coupling effect — tests that catch simple faults tend to catch complex ones too
• Idea proposed by Richard Lipton around 1970–71; classic published treatment: DeMillo, Lipton & Sayward (1978)
• Just et al. (2014), 357 real faults: mutant detection correlates with real-fault detection, even when controlling for coverage — but while 73% of the real faults were coupled to the generated mutants, 27% were not

The catches

• Cost — the suite runs once per mutant: roughly mutants × suite runtime
• Equivalent mutants — changes that don't change behaviour:

  i = 0
  while i < 10: i += 1    # vs.  while i != 10: i += 1

  Starting at 0 and stepping by 1, both stop at 10 — no test can kill it. "To detect if a program and one of its mutants programs are equivalent is undecidable" — Jia & Harman
• Noise — mutants in logging, default values, boilerplate

Investigating survivors

In practice

• Start by hand — two test runs on the line you doubt: one mutated, one restored
• Mutate only changed code — run it in code review, not over the whole repository
• Start with critical modules — pricing, authentication, parsing
• Skip the noise — exclude logging and generated code
• Run full analysis nightly or on demand, not on every commit
• Read survivors; don't chase the score

Tools

Your turn (2 min)

def discount(total):
    if total > 100:
        return total * 0.9
    return total

def test_discount():
    assert discount(200) == 180
    assert discount(50) == 50

Find a mutant that survives — then the test that would kill it.

Takeaways

Questions?

Full paper: mutation-testing-paper.pdf — references available there.