MSTest v2: Customize test execution

This post is part of the serie 'MSTest v2'. Be sure to check out the rest of the blog posts of the serie!

MSTest v2 is extensible. In the previous posts, we saw how to extend DataTest and create new assert methods. In this post, we'll see how to customize the way the tests are executed.

Customizing execution at test method level

By default, the runner will execute the methods decorated by [TestMethod]. If you look closer at the TestMethod attribute, you'll see that the class contains the logic of the test execution. Indeed, the method Execute (GitHub) call the test method and return the result of the execution.

public class TestMethodAttribute : Attribute
{
    public virtual TestResult[] Execute(ITestMethod testMethod)
    {
        return new TestResult[] { testMethod.Invoke(null) };
    }
}

You can see the method is virtual. So, you can extend this class to customize the way the method is called. For instance, if you need to execute the method in an STA thread, you can create your own attribute and override the Execute method to use an STA thread.

public class STATestMethodAttribute : TestMethodAttribute
{
    public override TestResult[] Execute(ITestMethod testMethod)
    {
        if (Thread.CurrentThread.GetApartmentState() == ApartmentState.STA)
            return Invoke(testMethod);

        TestResult[] result = null;
        var thread = new Thread(() => result = Invoke(testMethod));
        thread.SetApartmentState(ApartmentState.STA);
        thread.Start();
        thread.Join();
        return result;
    }

    private TestResult[] Invoke(ITestMethod testMethod)
    {
        return new[] { testMethod.Invoke(null) };
    }
}

You can use this attribute instead of TestMethod:

[TestClass]
public class TestClass1
{
    [STATestMethod]
    public void Test_STA()
    {
        Assert.AreEqual(ApartmentState.STA, Thread.CurrentThread.GetApartmentState());
    }
}

Customizing execution at test class level

Some of you may ask, why do you need the TestClass attribute as each test is already decorated by TestMethod. It seems redundant. Indeed, the test runner may discover tests by only using the TestMethod attribute. In fact, it allows to customize the way test are discovered in the class. The TestClass attribute allows to customize the way tests are discovered. For instance, if you want all the test of a class to be executed in an STA thread, you can create a class that inherits from TestClass.

public class STATestClassAttribute : TestClassAttribute
{
    public override TestMethodAttribute GetTestMethodAttribute(TestMethodAttribute testMethodAttribute)
    {
        if (testMethodAttribute is STATestMethodAttribute)
            return testMethodAttribute;

        return new STATestMethodAttribute(base.GetTestMethodAttribute(testMethodAttribute));
    }
}

public class STATestMethodAttribute : TestMethodAttribute
{
    private readonly TestMethodAttribute _testMethodAttribute;

    public STATestMethodAttribute()
    {
    }

    public STATestMethodAttribute(TestMethodAttribute testMethodAttribute)
    {
        _testMethodAttribute = testMethodAttribute;
    }

    public override TestResult[] Execute(ITestMethod testMethod)
    {
        // code omitted for brevity (same as above)
    }

    private TestResult[] Invoke(ITestMethod testMethod)
    {
        if (_testMethodAttribute != null)
            return _testMethodAttribute.Execute(testMethod);

        return new[] { testMethod.Invoke(null) };
    }
}

You can use this attribute instead of TestClass:

[STATestClass]
public class TestClass1
{
    [TestMethod]
    public void Test1()
    {
        Assert.AreEqual(ApartmentState.STA, Thread.CurrentThread.GetApartmentState());
    }

    [STATestMethod]
    public void Test2()
    {
        Assert.AreEqual(ApartmentState.STA, Thread.CurrentThread.GetApartmentState());
    }

    [DataTestMethod]
    [DataRow(1)]
    [DataRow(2)]
    public void Test3(int i)
    {
        Assert.AreEqual(ApartmentState.STA, Thread.CurrentThread.GetApartmentState());
    }
}

All methods of the class are executed in an STA thread.

Conclusion

MSTest v2 is easily extensible. In this post, I've show how to use an STA thread to execute tests. But there are many scenarios, such as changing the culture of the current thread or executing tests many times to ensure the stability of a test. Indeed, you can create a Culture attribute to set the culture, or a RepeatTestMethod attribute to execute the test N times or for a specific period of time. This extension point really helps to factorize similar behaviors!

MSTest v2: Create new asserts

This post is part of the serie 'MSTest v2'. Be sure to check out the rest of the blog posts of the serie!

MSTest v2 is extensible. In a previous post, we saw how to extend data tests. In this post, we'll see how to create new asserts.

The MSTest framework contains lots of assert methods. They are all located in 3 classes: Assert, StringAssert, CollectionAssert. Sometimes, you want more asserts. In this case you can create your own assert methods.

An assert method is a method that validate a value statifies a specific criteria. If the criteria is not statisfied, an exception is thrown. In MSTest, the exception is of type AssertFailedException, but you can throw any kind of exception.

Let 's create a custom assert to ensure a string is equal to another one. If the 2 strings are differents the message contains a formatted diff that shows blank characters and the first different character:

internal static class MyAssert
{
    public static void StringEquals(string expected, string actual)
    {
        if (expected == actual)
            return;

        throw new AssertFailedException(GetMessage(expected, actual));
    }

    private static string GetMessage(string expected, string actual)
    {
        var expectedFormat = ReplaceInvisibleCharacters(expected);
        var actualFormat = ReplaceInvisibleCharacters(actual);

        // Get the index of the first different character
        var index = expectedFormat.Zip(actualFormat, (c1, c2) => c1 == c2).TakeWhile(b => b).Count();
        var caret = new string(' ', index) + "^";

        return $@"Strings are differents.
Expect: <{expectedFormat}>
Actual: <{actualFormat}>
         {caret}";
    }

    private static string ReplaceInvisibleCharacters(string value)
    {
        return value
            .Replace(' ', '·')
            .Replace('\t', '→')
            .Replace("\r", "\\r")
            .Replace("\n", "\\n");
    }
}

In case the values are differents, the output of the test run look like:

[TestMethod]
public void Test1()
{
    MyAssert.StringEquals("abc def", "abc  def");
}
Message: Strings are differents.
Expect: <abc·def>
Actual: <abc··def>
             ^

Ok, that great. But MSTest v2 provides a way to integrate your assert into the provided asserts. If you look at the intellisense, you may have notice the That property. The property has no member:

So, why the hell is this property? The idea is to provide a property that returns an instance of Assert. This allows you to create extension methods 😃 (of course, the same property exists in StringAssert and CollectionAssert)

So, you can rewrite the assert method as an extension method:

internal static class MyAssert
{
    public static void StringEquals(this Assert assert, string expected, string actual)
    {
        if (expected == actual)
            return;

        throw new AssertFailedException(GetMessage(expected, actual));
    }

    // ...
}

Now, you can use the extension method in the test:

[TestMethod]
public void Test1()
{
    Assert.That.StringEquals("abc def", "abc  def");
}

This syntax is more readable as the line is a valid English sentence.

Conclusion

Writing custom assert method allows to create more useful message, or to create business specific asserts. MSTest v2 provides a nice way to create custom assert by using a extension method. The That property is very smart 😃

Tip: Download Sysinternals tools from the console

Here's a nice tip to download all the Sysinternals tools in 3 command lines. You know all those awesome tools such as Sysmon, Process Explorer, Process Monitor, bginfo, and so on.

net use x: \\live.sysinternals.com
xcopy /s x:\ c:\sysinternals\
net use x: /d

If you just want to use a tool from time to time, you don't need to download them all. Instead, you can open the Run window (Win+R) and enter the full uri of the tool. Of course, you must know the name of the tool! For instance, if you want to run Process Monitor (procmon.exe), you can use the following command:

\\live.sysinternals.com\tools\procmon.exe

Hope this helps!

MSTest v2: Test lifecycle attributes

This post is part of the serie 'MSTest v2'. Be sure to check out the rest of the blog posts of the serie!

In a test project, some tests may have pre-conditions. You may also do some cleanup. For instance, you need to set a global configuration, or to delete some files after a test ran. This may be more useful for integration tests than for unit tests.

MSTest v2 provides a way to declare methods to be called by the test runner before or after running a test. Here's how to use declare them:

Initialize/Cleanup by assembly

The method decorated by [AssemblyInitialize] is called once before running the tests of the assembly. The method decorated by [AssemblyCleanup] is called after all tests of the assembly are executed. These methods can be located in any class as long as the class is decorated by [TestClass].

[TestClass]
public class Initialize
{
    [AssemblyInitialize]
    public static void AssemblyInitialize(TestContext context)
    {
        Console.WriteLine("AssemblyInitialize");
    }

    [AssemblyCleanup]
    public static void AssemblyCleanup()
    {
        Console.WriteLine("AssemblyCleanup");
    }
}

Initialize/Cleanup by class

The method decorated by [ClassInitialize] is called once before running the tests of the class. In some case, you can write the code in the constructor of the class. The method decorated by [ClassCleanup] is called after all tests from all classes are executed.

[TestClass]
public class TestClass1
{
    [ClassInitialize]
    public static void ClassInitialize(TestContext context)
    {
        Console.WriteLine("ClassInitialize");
    }

    [ClassCleanup]
    public static void ClassCleanup()
    {
        Console.WriteLine("ClassCleanup");
    }

    [TestMethod]
    public void Test1()
    {
    }
}

Initialize/Cleanup by test

The method decorated by [TestInitialize] is called before running each test of the class. The method decorated by [TestCleanup] is called after running each test of the class.

[TestClass]
public class TestClass1
{
    [TestInitialize]
    public void TestInitialize()
    {
        Console.WriteLine("TestInitialize");
    }

    [TestCleanup]
    public void TestCleanup()
    {
        Console.WriteLine("TestCleanup");
    }

    [TestMethod]
    public void Test1()
    {
    }
}

Execution log

Here's the execution log. I think this is much more clearer than long sentenses!

AssemblyInitialize (once by assembly)
  Class1Initialize (once by class)
    TestInitialize (before each test of the class)
      Test1
    TestCleanup    (after each test of the class)
    TestInitialize
      Test2
    TestCleanup
    ...
  Class2Initialize
      ...
  Class1Cleanup    (once by class)
  Class2Cleanup
AssemblyCleanup    (once by assembly)

Validation made easy with decorators

Before reading this post, you should read the previous post about Aspect Oriented Programming in TypeScript. It will help you understand the TypeScript decorators.

The idea of this post is to write a class and use decorators to set the validation rules on attributes:

class Customer {
    @required
    public firstName: string;
    @required
    public lastName: string;
}

var customer = new Customer();
customer.firstName = 'Gérald';
validate(customer); // 'lastName' is required

The solution is to use decorators and metadata to attach the validation rules to the associate properties. Then, you can query these metadata to run the rules. TypeScript supports the Metadata Reflection API. This API defines a few methods. Here's are the ones we'll use:

namespace Reflect {
    // Set metadata

    // Reflect.defineMetadata("custom:annotation", value, Customer);
    function defineMetadata(metadataKey: any, metadataValue: any, target: Object): void;
    // Reflect.defineMetadata("custom:annotation", value, Customer.prototype, "firstName");
    function defineMetadata(metadataKey: any, metadataValue: any, target: Object, propertyKey: string | symbol): void;

    // Get metadata

    // Reflect.getMetadata("custom:annotation", Customer);
    function getMetadata(metadataKey: any, target: Object): any;
    // Reflect.getMetadata("custom:annotation", Customer.prototype, "firstName");
    function getMetadata(metadataKey: any, target: Object, propertyKey: string | symbol): any;
}

The following code attach a metadata named "validation" to the property and the class. The metadata for the property is a list of ValidationRule. For the class we attach the list of properties to validate.

First, you need to install the metadata polyfill:

npm install reflect-metadata

Then, let's create a function to add the metadata to the property and class:

import "reflect-metadata";

function addValidationRule(target: any, propertyKey: string, rule: IValidationRule) {
    let rules: IValidationRule[] = Reflect.getMetadata("validation", target, propertyKey) || [];
    rules.push(rule);

    let properties: string[] = Reflect.getMetadata("validation", target) || [];
    if (properties.indexOf(propertyKey) < 0) {
        properties.push(propertyKey);
    }

    Reflect.defineMetadata("validation", properties, target);
    Reflect.defineMetadata("validation", rules, target, propertyKey);
}

Let's implement the required validation rule:

interface IValidationRule {
    evaluate(target: any, value: any, key: string): string | null;
}

class RequiredValidationRule implements IValidationRule {
    static instance = new RequiredValidationRule();

    evaluate(target: any, value: any, key: string): string | null {
        if (value) {
            return null;
        }

        return `${key} is required`;
    }
}

function required(target: any, propertyKey: string) {
    addValidationRule(target, propertyKey, RequiredValidationRule.instance);
}

Finally, you can validate an object using the metadata:

function validate(target: any) {
    // Get the list of properties to validate
    const keys = Reflect.getMetadata("validation", target) as string[];
    let errorMessages: string[] = [];
    if (Array.isArray(keys)) {
        for (const key of keys) {
            const rules = Reflect.getMetadata("validation", target, key) as IValidationRule[];
            if (!Array.isArray(rules)) {
                continue;
            }

            for (const rule of rules) {
                const error = rule.evaluate(target, target[key], key);
                if (error) {
                    errorMessages.push(error);
                }
            }
        }
    }

    return errorMessages;
}

function isValid(target: any) {
    const validationResult = validate(target);
    return validationResult.length === 0;
}

Conclusion

In the previous post, I showed how to use decorators to change the default behavior of a method or class. Metadata allows to use decorators for new scenario. If you look at it, it really similar to C#.