You have consistently written unit tests and you have a line coverage of, let us say, 80% and all of your tests pass. Pretty good, isn’t it? But then you change your code and still all of your tests pass although you have changed code which is covered by your unit tests. In this post, we will take a look at mutation testing which will test the quality of your unit tests.
1. Introduction
We value our code quality very much. We execute static code analysis, we write unit tests, integration tests, etc. We can set minimum threshold values for certain metrics: we do not allow critical, major analysis issues, all tests must pass, etc. These are all good and valuable items to check in the pipeline to ensure a certain quality level. But what is the value of the metric itself? When your unit tests do not assert anything or assert the wrong items, the metric of passed unit tests does not give any indication about the quality of your unit tests. Of course, our static code analysis will raise an issue when we do not assert anything in our unit test but it will not alert us for wrong assertions. Besides that, we can take a look at the Line Coverage of our unit tests. This tells us something about how much of our code is covered by our unit tests. But even that can be deceiving. What if we do not assert the correct items? What if we change our code and the test still passes? In other words, we need something which will give us an indication about the quality of our tests. That is where mutation testing is for. With mutation testing, faults (or mutants) are introduced in your code and then your tests are run again. If your test fails, the mutant is killed. If your test passes, the mutant is lived. We now have a new metric Mutation Coverage which tells us how to interpret the Line Coverage metric in a more correct way.
2. PIT Mutation Testing
How to get this new Mutation Coverage metric? For Java based applications, we can make use of the PIT Mutation Testing Maven plugin. Traditionally, we use the JaCoCo Maven plugin, but JaCoCo can be replaced with the PIT Mutation Testing plugin because it provides us both metrics we want: the Line Coverage metric and the Mutation Coverage metric. A standard set of mutators is being used, for a complete list, see the PIT website.
3. In Practice
The proof of the pudding is in the eating. Let’s create a basic Spring Boot application and some unit tests in order to achieve a 100% line coverage. Next, we will run the PIT Mutation Testing Maven plugin and verify whether our unit tests survive the mutants or not. The source code can be found at GitHub.
3.1 Basic Spring Boot Application
We are going to create two URL’s which will perform a basic operation on a parameter and then return a result. We make use of Spring Web MVC and also add the Spring Boot test dependency. Our pom is the following:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
We add a MutationController. The first method compareToFifty compares whether the input value is greater than 50 or smaller than or equal to 50 and returns a corresponding text message. The second method increment increments the input value with one and returns the incremented value.
@RestController
public class MutationController {
@GetMapping("/compareToFifty/{value}")
public String compareToFifty(@PathVariable int value) {
String message = "Could not determine comparison";
if (value > 50) {
message = "Greater than 50";
} else {
message = "Smaller than or equal to 50";
}
return message;
}
@GetMapping("/increment/{value}")
public int increment(@PathVariable int value) {
value++;
return value;
}
}
Run the application:
$ mvn spring-boot:run
Verify whether the URL’s are accessible:
$ curl http://localhost:8080/compareToFifty/40 Smaller than or equal to 50 $ curl http://localhost:8080/compareToFifty/60 Greater than 50 $ curl http://localhost:8080/increment/5 6
We add three unit tests for the above:
- A test
smallerThanOrEqualToFiftyMessagein order to verify the response when the value is smaller than or equal to 50. We deliberately test with value 49 which is a poor boundary test because we should use 50 as a value to test. - A test
greaterThanFiftyMessagein order to verify the response when the value is greater than 50. - A test
incrementin order to verify whether the value is incremented. We deliberately only test whether a successful response is received and we do not test the return value.
@SpringBootTest
@AutoConfigureMockMvc
public class MutationTest {
@Autowired
private MockMvc mockMvc;
@Test
public void smallerThanOrEqualToFiftyMessage() throws Exception {
this.mockMvc.perform(get("/compareToFifty/49")).andDo(print()).andExpect(status().isOk())
.andExpect(content().string("Smaller than or equal to 50"));
}
@Test
public void greaterThanFiftyMessage() throws Exception {
this.mockMvc.perform(get("/compareToFifty/51")).andDo(print()).andExpect(status().isOk())
.andExpect(content().string("Greater than 50"));
}
@Test
public void increment() throws Exception {
this.mockMvc.perform(get("/increment/5")).andDo(print()).andExpect(status().isOk());
}
}
3.2 Line Coverage
First, we verify what our line coverage is when using the JaCoCo Maven Plugin. Add the plugin to the pom:
<build>
<plugins>
...
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.5</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>prepare-package</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Run the build:
$ mvn clean install
When finished, navigate to directory target\site\jacoco\ and open the index.html file which shows the results in your browser.
The report shows us that the MutationController has a 100% line coverage. The total line coverage also equals 100% because the line coverage of MyMutationTestingPlanetApplication is not applicable (The PIT Mutation report will not list the not applicable results for MyMutationTestingPlanetApplication).
The configuration with JaCoCo is present in branch feature/jacoco.
3.3 Mutation Coverage
Remove the JaCoCo plugin and add the PIT Mutation Test Plugin.
<build>
<plugins>
....
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.5.0</version>
<dependencies>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
<version>0.12</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
We also needed to add the dependency pitest-junit5-plugin because we are using junit 5. If you do not add this dependency, your unit tests will not be found. This information was not explicitly mentioned in the PIT documentation. It is, however, documented in the Maven quickstart section when option testPlugin is described: Support for other test frameworks such as junit5 can be added via plugins.
Run the following Maven goal:
$ mvn org.pitest:pitest-maven:mutationCoverage
When finished, navigate to directory target\pit-reports\ and open the index.html file which shows the results in your browser.
Again, we notice a 100% line coverage for our MutationController, but now it also indicates a 40% Mutation Coverage. Click down to the MutationController details, it shows us the following:
The report shows us exactly which mutants survived the test. The tests for compareToFifty survived the boundary test because the greater than sign (>) has been replaced by the mutant with greater than and equal to (>=). We used 49 as input value in our test which is obviously not a good boundary test. The unit test for the increment method does not assert the returned value. Changes made to the code for returning the value will not cause our unit test to fail.
3.4 Fix the Unit Tests
We fix the unit tests in branch feature/solutions.
The solution for the smallerThanOrEqualToFiftyMessage test is to test the value 50 instead of 49.
@Test
public void smallerThanOrEqualToFiftyMessage() throws Exception {
this.mockMvc.perform(get("/compareToFifty/50")).andDo(print()).andExpect(status().isOk())
.andExpect(content().string("Smaller than or equal to 50"));
}
The solution for the increment test is to check the returned value besides the successful response.
@Test
public void increment() throws Exception {
this.mockMvc.perform(get("/increment/5")).andDo(print()).andExpect(status().isOk())
.andExpect(content().string("6"));
}
If you run the mutation coverage Maven goal again, you will notice that your report has not been changed. There will still be a 40% mutation coverage. PIT analyzes the byte code and is not automatically going to compile your test classes again. Therefore, the report looks the same because the byte code has not been changed. So, first run your tests again and then generate the Mutation Coverage report.
The report shows us besides the 100% line coverage, also a 100% mutation coverage.
4. Conclusion
The PIT Mutation Maven plugin is a great tool for testing the quality of your tests. It is easy to use and can be added instantly to your CI/CD pipeline in order to generate useful results about the quality of your unit tests.
mydeveloperplanet.com 