When one develops, comes the fateful moment of testing. One would be tempted to say to himself "I am a developer not a tester⊠writing tests is not my job'. Unfortunately it does not always happen like that in real life (it's not always easy to be a developer đ). Sometimes you have to spend a lot of time trying to write the most exhaustive tests in order to have the highest possible code coverage and avoid surprises during execution. It is necessary to test at the same time that the code does what it is supposed to do, but also that it does not crash miserably because one forgot to test a particular case. This work is very tedious and it is clear that it is difficult to do things right.
Fortunately, Microsoft researchers are trying to make life a little easier. Indeed thanks to Pex, a tool developed by Microsoft Research, we have almost no more to worry about tests (well, maybe I exaggerate a little;)). To make it simple Pex parses our compiled code (which makes it compatible with all .NET languages), and tries to create tests with the highest code coverage for each method. Its purpose is not to validate the logic (since it is not yet able to guess what the developer is trying to do), but rather to verify that all cases are tested. This means that Pex seeks to identify the useful arguments to go through all branches of a method without multiplying the tests passing through the same branch.
To achieve this, Pex does not generate random arguments (Fuzzing) but understands the code and tries to find the correct arguments (thanks to the Z3 engine)
Let's start with a rather simple example:
C#
public static void AnswerToTheUltimateQuestionOfLifeTheUniverseAndEverything(int i)
{
if (i > 41)
return;
throw new Exception("Try again...");
}
Pex finds the right answer đ. Let's see an example of what Pex offers with a slightly more complicated mathematical example:
C#
public static void TestMath(object o, int[] values)
{
// Preconditions
if (values == null)
throw new ArgumentNullException("values");
if (values.Length < 3)
throw new ArgumentException("values must have at least 3 elements", "values");
if (values[0] == values[1] / 2)
{
o.ToString(); // NullReferenceException
}
else if (values[0] + values[1] == (values[1] * 71 + values[2] + 42) / 2)
{
throw new Exception("Simple equation");
}
// Pex makes it possible to check that 2 numbers are coprime integers!
// The proof using the Bachet-BĂ©zout theorem (https://en.wikipedia.org/wiki/Coprime_integers)
if (41 * values[0] + 42 * values[1] == GCD(41, 42)) // 41x+42y=1
{
throw new Exception("41 and 42 are coprimes");
}
}
private static int GCD(int a, int b)
{
return b == 0 ? a : GCD(b, a % b);
}
As you can see, the exceptions thrown on the arguments (ArgumentNullException, ArgumentException, ArgumentOutOfRangeException, etc.) are considered valid (green) while the others are considered as errors (red). We can indicate to Pex that an exception is expected:
Here's the result:
In addition to solving equations, Pex also understands Regex:
C#
public static void TestString(string s)
{
if (Regex.IsMatch(s, @"\w+\d{2}"))
throw new Exception("Match");
}
However there are many cases that Pex can not solve:
C#
public void TestXml(XDocument doc)
{
if (doc == null)
throw new ArgumentNullException("doc");
if (doc.Descendants("Tests").Any())
throw new Exception(); // Pex can't create the XDocument itself
}
Finding interesting values for the tests is good, but Pex also allows you to save the generated unit tests in order to replay them later. For this Pex generates several files:
- One file per class:
{class name}Test.cs - One file per method of the class:
{class name}Test.{method name}.g.cs
The first file contains test stubs of the type:
C#
[PexMethod, PexAllowedException(typeof(ArgumentException)), PexAllowedException(typeof(ArgumentNullException)), PexAllowedException(typeof(Exception))]
public void TestMath(object o, int[] values)
{
Program.TestMath(o, values);
// TODO: add assertions to method ProgramTest.TestMath(StringBuilder, Int32[])
}
The second file contains the tests with the values found by Pex during the analysis:
C#
[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
[PexRaisedException(typeof(NullReferenceException))]
public void TestMathThrowsNullReferenceException543()
{
int[] ints = new int[3];
this.TestMath((StringBuilder)null, ints);
}
[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
public void TestMath900()
{
int[] ints = new int[3];
ints[0] = 427;
ints[1] = 8;
ints[2] = 245;
this.TestMath((StringBuilder)null, ints);
}
The advantage of creating stubs is to be able to add logical tests (which the method actually has to do), tests that will be checked for each test generated by Pex.
For all those who say, I use NUnit (not Microsoft), so Pex (Microsoft) will not work with my NUnit projects, it's wrong. Pex can generate code for other test frameworks. It supports MSTest, xUnit.Net, NUnit, MbUnit.
As can be seen Pex integrates perfectly with Visual Studio 2008 and 2010. And yes, no need to leave its preferred development environment. Just right click on what you want to test and Pex starts. What more ? (if not support the newer versions of VS 2012 and 2013)
For Visual Studio 2012, the team has created Microsoft Code Digger plugin, but it only allows you to generate interesting arguments for a method, no test generation. Always very easy to use: right click â Generate Inputs / Outputs Table
Last but not least, Pex can be used on the command line (see the documentation) which allows for example to integrate it to the nightly build.
This is a summary presentation of this tool. To understand the possibilities, I advise you to consult the PDF document of the team in charge of the development of Pex: http://research.microsoft.com/en-us/projects/pex/pexmanual.pdf
PS: For those who are not convinced, quickly try the online version: http://www.pexforfun.com/
Do you have a question or a suggestion about this post? Contact me!
